# Deep Reinforcement Learning Method for 3D-CT
## 2 Background  
RL is inspired by behavioral psychology and neuroscience. RL agents learn strategies by interacting with the environment \(\mathbf{e}\). In each state \(\mathbf{s}\), agents make a single decision from a set of discrete actions and select an action \(\mathbf{a}\). Each effective action selection generates a related scalar reward, defined as the reward signal \(\mathbf{r}\). The main goal of the agent is to learn a strategy \(\pi\) that not only maximizes immediate rewards but also maximizes future rewards in the future. Due to similarities in the interactions between agents and the environment, reinforcement learning can be considered as a universal learning framework that may solve the problems of general artificial intelligence.  
**Q-Learning**: The selection of strategies was obtained through the action-value function Q(\(s\),\(a\)), which is defined as the sum of future discounted expected rewards \(E[r_{t+1}+\gamma r_{t+2}+\dots+\gamma^{n-1}r_{t+n}|s,a]\); this function can be recursively expanded through the Bellman equation [26] to iteratively solve  
\[Q_{i+1}(s,a)=E[r+\gamma\underset{a^{\prime}}{\max}Q_{i}(s^{\prime},a^{\prime})] \tag{1}\]  
Among \(\gamma\in(0,1)\) is a discount factor, which is used to weight the future reward accordingly. It represents the uncertainty in the agent's environment; \(a^{\prime}\) and \(s^{\prime}\) represent the next action to be taken and the next state after taking that action, respectively. The optimal strategy \(\pi^{*}\) of the agent is obtained based on the optimal action function:  
\[Q^{*}(s,a)=\underset{\pi}{\max}E\{R_{t}|s_{t}=s,a_{t}=a,\pi\},\ \pi^{*}(s)= \underset{a\in A}{\text{argmax}}Q^{*}(s,a) \tag{2}\]  
**Deep Q-Learning**: [PERSON] et al. [27] proposed the Deep Q-Network (DQN). It introduces a target network \(Q^{-}(S_{t+1},a;\theta^{\prime})\) based on a Q network \(Q(S_{t+1},a;\theta)\). The target network has the same structure and the same initial weight as the Q network. After \(c\) iterations of training, the parameters of the Q network are assigned to the target network \(Q^{-}\), thereby reducing overestimation of the Q value. The loss function of the DQN is defined as:  
\[L(\theta)=E_{s,d,r,s^{\prime}}[(r+\gamma\underset{a^{\prime}}{\max}Q_{target}(s ^{\prime},a^{\prime};\theta^{\prime})-Q_{net}(s,a;\theta))^{2}] \tag{3}\]  
where \(\gamma\) is the discount factor determining the agent's horizon, \(\theta\) are the parameters of the Q-network, and \(\theta^{\prime}\) are the parameters of the target network.  
In recent years, researchers have gradually applied DRL algorithms to the field of medical image analysis. [PERSON]'s team [12; 24; 28] applied the DRL method to the anatomical landmark detection problem in medical images, which mainly solved the difficulty of multi-scale and incomplete data. [PERSON]'s team [21; 25; 29; 30] used DRL methods for anatomical landmark detection and automatic view planning problems, and they brought the multi-agent approach to their research work. Some researchers [10; 31] have applied the DRL method to other medical image analysis problems, such as the location of human organs (such as the breast and pancreas). Based on the existing research work in the research community, we tried to use the DRL method to solve the problem of cancer localization in nasopharyngeal carcinoma imaging. However, cancer localization and anatomical landmark detection tasks are different, such as: (1) there is no multi-target localization problem, so the direct use of the multi-agent method is not appropriate; and (2) the size variation of the cancer area is greater than that of a single anatomical landmark. In order to explore the problem of cancer localization suitable for the nasopharyngeal carcinoma 3D-CT scene, we tried to improve the existing methods in this paper.