# Deep Reinforcement Learning Method for 3D-CT
## 1 Introduction
Experiments have shown that, given a series of annotated images, the agent can learn the best path to converge to the marked position. In response to the issue of artifacts in 3D fetal MRI images, [PERSON] et al. [13] deployed 15 agents based on DRL, simultaneously detecting 15 feature markers, and set additional rewards based on the distance between the agent and the fetal body nodes to improve detection accuracy. [PERSON] et al. [14] studied how to locate lung lesions on chest radiographs. In one of these methods, reinforcement learning is used to learn a cyclic attention model with annotated feedback (RAMAF) to observe short sequences of image patches. [PERSON] et al. [15] proposed a deep reinforcement learning method for tracking and locating blood vessels in 3D images. Instead of an exhaustive search, they use artificial agents to interact with their surroundings and collect rewards from the interactions. [PERSON] et al. [3] trained DQN on 70 T1-weighted two-dimensional image slices to predict the location of brain tumors. This study shows that supervised deep learning is seriously overfitting on the training set, with low accuracy, while reinforcement learning may make meaningful predictions based on very small datasets. Based on this work, [PERSON] et al. [16] trained a Deep Q network on 30 2D image slices from the BraTS brain tumor database, which showed good accuracy (70% accuracy) compared to the supervised deep learning network (11% accuracy) on the testing set. [PERSON] et al. [17] proposed a DRL method for CT organ localization for exhaustive or regional suggestion search strategies that require large amounts of annotated data. Another research direction is the detection of lesion areas and landmarks. [PERSON] et al. [18] proposed a multi landmark detection method to improve time efficiency and enhance robustness for missing data. [PERSON] et al. [19] proposed using DRL models for lung cancer detection and tested multiple deep reinforcement learning models, such as DQN. By combining DRL methods for the early detection and diagnosis of lung cancer, treatment effectiveness can be significantly improved. [PERSON] et al. [20] studied a reinforcement learning model based on deep artificial neural networks for the early detection of lung nodules in chest CT scans. [PERSON] et al. [21] proposed a fully automatic recognition method for standardized visual planes in 3D image acquisition and conducted experimental evaluations in isotropic brain and heart MRI images. [PERSON] and [PERSON] et al. [22] followed the formula of [PERSON] et al. [23] and learned the RL agent for landmark localization in 3D medical images. In addition, in order to speed up learning and achieve more robust localization, multiple local strategies are learned on different sub-action spaces instead of a single complex strategy on the original action space. [PERSON] et al. [24] redefined the detection problem as a behavioral learning task for artificial intelligence agents, using the capabilities of DRL and scale space theory to learn the optimal search strategy for finding anatomical structures based on multi-scale image information for 3D landmark detection. They conducted experimental evaluations on 3D-CT scans, and the results showed that the method had good accuracy and robustness. [PERSON] et al. [25] proposed a novel communicative multi-agent reinforcement learning (C-MARL) system to automatically detect landmarks in medical scans. The architecture of their proposed C-MARL model takes, as an input, a tensor of size number agents \(\times\) 4 \(\times\) 45 \(\times\) 45 \(\times\) 45 and consists of four 3D convolutional and three 3D max pooling layers, followed by four FC layers. In their architecture, each agent has its own FC layer, but the convolutional layers are shared between all the agents. They evaluated the proposed approach using two brain imaging datasets from adult magnetic resonance imaging (MRI) and fetal ultrasound scans.