# Deep Reinforcement Learning Method for 3D-CT
## 5 Discussion and Conclusions  
We propose a single-agent reinforcement learning method based on small sample data for the localization of nasopharyngeal carcinoma lesion areas in 3D-CT scanning. Due to the approximate location of each patient's cancerous area, we found that the baseline model's agent explored some useless space when searching, which affected the accuracy and speed of positioning to some extent. Therefore, prior knowledge is added to our method, which reasonably reduces the searching space required by the agent and speeds up the convergence of the model. Specifically, the average relative position of the target lesion in the CT scan volumes is calculated based on the training set, and it is used as the starting position of the agent for searching in training and inference. At the same time, we use multi-scale processing technology to facilitate the localization of small targets. In the experimental section, we compare our model with the previous single-agent method and the baseline model. The experimental results show that the proposed model has good robustness and accuracy. On the premise of using a smaller data set, the accuracy of the DQN model is improved by more than 1 mm on average compared with the current study. This indicates that the model can effectively locate the nasopharyngeal carcinoma lesion area in 3D-CT scanning. At the same time, we also conducted ablation experiments, and the experimental results show that the fusion of prior knowledge can indeed improve the accuracy of the model, which indicates the effectiveness of our work. **Future work**: On the basis of existing DQN models, we may improve the DRL model performance with Double DQN or Dueling DQN and systematically evaluate the advantages and disadvantages of different models. In addition, the dataset in this experiment is limited in size, and more data should be obtained later in order to improve the accuracy of prior knowledge results and further improve the robustness of the model. Finally, the proposed method can be combined with the medical image segmentation method to form a two-stage nasopharyngeal carcinoma focal segmentation model so as to achieve more accurate extraction of focal areas. Conceptualization, H.W. and H.L. ; Formal analysis, G.H. and Y.K. ; Funding acquisition, G.H. ; Methodology, G.H., H.W. and H.L. ; Project administration, G.H. and H.L. ; Resources, H.W. and H.L. ; Supervision, H.W. and H.L. ; Validation, G.H. and Y.K. ; Visualization, G.H. and Y.K. ; Writing--original draft, G.H. and Y.K. ; Writing--review and editing, G.H., Y.K., H.W. and H.L. All authors have read and agreed to the published version of the manuscript. This work was supported in part by the National Natural Science Foundation of China (61901533), the Shenzhen Fundamental Research Program of China (JCYJ20190807154601663), and the High-level Talents Research Project of NCWU (202101002). This retrospective study was conducted in accordance with the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. Ethics approval was obtained from the Institutional Review Board of SYSUCC (approval number: B2019-222-01). Furthermore, according to Article 32 of the \"Measures for the Ethical Review of Life Science and Medical Research Involving Humans\"(national legislation of China [34]), research using anonymized information data can be exempted from the ethical review. \begin{table}
\begin{tabular}{c c c c} \hline \(\gamma\) & \(\delta\) & **Error Distance (mm)** & **Average Loss** \\ \hline
0.9 & \(2\times 10^{-4}\) & 4.36 \(\pm\) 1.22 & 0.269 \\ \hline
**0.9** & \(\mathbf{1\times 10^{-4}}\) & \(\mathbf{3.52\pm 1.18}\) & \(\mathbf{0.134}\) \\ \hline
0.9 & \(5\times 10^{-5}\) & 5.06 \(\pm\) 1.94 & 0.332 \\ \hline
0.99 & \(1\times 10^{-4}\) & 26.45 \(\pm\) 2.44 & 16.08 \\ \hline
0.8 & \(1\times 10^{-4}\) & 4.19 \(\pm\) 0.92 & 0.147 \\ \hline \end{tabular}
\end{table} 
Table 4: Model performance under different hyperparameters.