# Deep Reinforcement Learning Method for 3D-CT
## 3 Method  
The framework of reinforcement learning consists of many basic elements, which are defined in this work as follows:  
**State**: The environment e is a three-dimensional CT scan, where each state is defined as a region of interest (ROI) centered around the agent. In order to improve the robustness of the experiment, the latest 4 frame states are stored in the frame history. DQN is difficult to converge in the early stage of training because agents usually need to be based on trial and error in a large exploration space. In this paper, we introduce prior knowledge to guide the agent to reasonably reduce the search space and promote the convergence rate of the model. Specifically, instead of randomly setting the starting point, we set the starting point based on the average relative location of the nasopharyngeal carcinoma lesion. The average relative position of the target lesion in CT scan volumes was calculated based on the training set (i.e., 50 CT scans), and it was used as the starting position of the agent for searching in training and inference. The starting search point for each agent at the beginning of each episode was calculated on the training set. For N CT scans \(I_{1}\), \(I_{2}\), \(I_{3}\ldots I_{N}\), we defined \(\overset{\rightarrow}{r}(d)\) as:  
\[\overset{\rightarrow}{r}(d)=\frac{1}{N}\sum_{k=1}^{N}\frac{gt_{I_{k}}^{d}}{ size_{I_{k}}^{d}} \tag{4}\]  
Among them, \(size_{I_{k}}^{d}\) represents the size (i.e., the voxel number) of the \(k\)-th CT scans in dimension \(d\), and \(gt_{I_{k}}^{d}\) represents the coordinates of the ground truth feature point in dimension \(d\) of the \(k\)-th CT scans. The starting position \(p_{0}\) of the agent is:  
\[p_{0}=\overset{\rightarrow}{r}\odot size[I] \tag{5}\]  
In Formula (5), the \(\odot\) means multiplication of the corresponding elements for each dimension. Compared to randomly generating search starting points, this method can reduce errors while improving training robustness. **Actions**: The agent interacts with the environment through actions \((a\in A)\), and the action space A is composed of six actions in the Cartesian coordinate system, namely \(\{\pm\ x,\ \pm y,\ \pm z\}\), where '+' represents to move one unit distance along the positive direction of the axis at the current scale, and '\(-\)' represents moving one unit distance along the negative direction of the axis at the current scale. We adopted a multi-scale search strategy using three levels of scales: 3 mm, 2 mm, and 1 mm. Initially, the agent searches at the maximum scale. As the action progresses, the search step size and ROI size are reduced as the agent approaches the target until the agent reaches the termination state at the minimum scale. **Reward**: The reward function \(R_{D}\) drives the agent's action, and in this work, we define it as the difference in the Euclidean distances between the two states and the target point:  
\[R_{D}=D(p_{i-1},p_{t})-D(p_{i},p_{t}) \tag{6}\]  
\[R= \begin{cases}+1&R_{D}>0,\text{ {the agent approaches the target}}\\ -1&R_{D}<0,\text{{the agent is moving away from the target}}\end{cases}  \tag{7}\]  
\(D\) represents the Euclidean distance function between two points, \(p_{i}\) represents the location of the agent at step \(i\), and \(p_{t}\) represents the ground truth position of the target. According to the reward function, if the agent moves away from the target, the system gives a reward signal of \(-1\) as a punishment. Otherwise, the system gives a positive reward signal. **Terminal Condition**: In this experiment, the position of the target feature points was set as the physical center point of the ground truth in the lesion area. In the training phase, the agent terminates the search when it oscillates at the minimum scale or when it reaches the maximum number of steps (200 steps in our experiment). If the agent oscillates at a larger scale, it will proceed to the next smaller scale level to continue the search.