# Automatic Needle Route Proposal in Preoperative Neck CT for Injection Laryngoplasty
## 2 Methodology
### Vocal Fold Localization  
We obtain the vocal fold locations using the RL agent in our previous work [7]. In standard deep learning approaches, either the coordinates of the target location [9] or a  
Figure 1: Automatic needle route planning for TIL. Based on the vocal fold locations and the segmented critical regions, the search space is defined by covering all possible trajectories from the anterior neck skin to the vocal folds. The optimal needle routes (one for each vocal fold) are then determined based on a minimal cost search. The first three figures (from left) are presented in axial view, whereas the last figure presents a volumetric render of the result. The blue and orange lines indicate the needle routes for the right and left folds, respectively.  
heatmap around it [10] is usually regressed. Coordinates are difficult to regress because the outputs do not have direct spatial correspondence. Spatial input-output correspondence is utilized in heatmap regression. However, this often suffers from sample bias [11] because the heatmap voxels are usually negligible compared to the huge background voxels.  
RL adopts a pixel-to-pixel navigation framework coupled with deep feature learning, where an agent tries to gradually navigate to the target location through a sequential decision process. Recently, RL-based localization has recently shown significant outcomes in several studies [11; 12; 13] compared with standard deep learning methods. The application of RL for our vocal fold localization is also extensively studied in our previous work [7].  
In the RL-based approach, the agent is initialized at a random position inside the CT volume and takes a number of steps to reach the vocal fold. At any position, it observes an ROI (region-of-interest) of \(45\times 45\times 45\) voxels around and decides an optimal move (i.e., left, right, up, down, superior, inferior to move one voxel in any axis). Relating to the standard terminologies of RL, the observed ROI is defined as the _state_ and the six moves are the _actions_. We denote state by \(s\) and action by \(a\). The optimal action is determined by the policy network \(\pi\) (i.e., a convolutional neural network) that takes the ROI as input and produces optimal action probabilities \(\pi(s,a)\) as output. Eventually after conducting a number of such moves, the agent converges to the target vocal fold location.  
During policy training, the agent takes episodes of steps in different CT volumes. For each step, an action is rewarded \(+1\) if it takes the agent closer to the target or \(-1\), otherwise. Thus, a set of transitions is gathered where each transition \((s,a,r)\) includes state \(s\), action \(a\), and the corresponding reward \(r\). The goal of training is to maximize the expected cumulative reward.  
The policy network used in the current work is exactly the same as the network used in our previous study [7]. We have reused the same network without changing any parameters and tested with the current test data. The network has three convolutional blocks followed by two fully connected layers. Each convolutional block consists of two convolutional layers (kernel size: \(3\times 3\)) with a trailing max pooling layer (pool size: \(2\times 2\); stride: \(2\)). The final fully connected layer has six units to output the optimal probabilities of six actions.  
Note that the vocal fold localization is not the focus of the current work. Moreover, we have only reused the same architecture from our previous study [7]. Therefore, we do not include the elaborated training process here. Interested readers may refer to [7] for more details.