## 2 Methodology  
In our deep RL-based method, we automatically localize the RVF and LVF in preoperative neck CT. Neck CT can be represented by a 3D volume \(V\) with the three Cartesian axes \(X\), \(Y\), and \(Z\) indicating towards _right_-to-_left_, _anterior_-to-_posterior_, and _superior_-to-_inferior_, respectively (see Figure 2). Inside this volume, we denote the right and left fold positions by \(\mathbf{p}_{r}\) and \(\mathbf{p}_{l}\). Specifically, these positions are identified as the posterior vocal fold landmarks.  
Figure 1: Right and left vocal folds (RVF and LVF) shown in axial slices of three neck CT volumes.  
Figure 2: Cartesian axes directions in neck CT volume. On the left, volume rendering of a neck CT using RadiAnt Dicom Viewer ([[http://www.radiantviewer.com](http://www.radiantviewer.com)]([http://www.radiantviewer.com](http://www.radiantviewer.com)) (accessed on 15 December 2022)) is shown with axes annotation.  
The basic agent interaction in the proposed RL environment is similar to the standard formulation commonly found in all existing RL-based studies [8; 10; 11]. Usually, an agent localizes a target landmark \(\mathbf{p}_{\text{goal}}\) by taking an episode of sequential steps starting from a random initial voxel inside the environment (i.e., input CT volume). At any step \(t\), it observes the current state \(\mathbf{s}_{t}\) as the local patch centered at the current voxel \(\mathbf{p}_{t}\) and takes an action \(a_{t}\in\{x^{+},x^{-},y^{+},y^{-},z^{+},z^{-}\}\) to move one voxel forward or backward along any of the axes. Thus, it updates to a neighboring voxel \(\mathbf{p}_{t+1}\) and transitions to a new state \(\mathbf{s}_{t+1}\). A policy \(\pi(\mathbf{s}_{t},a_{t})\) gives the optimal probability for choosing an action \(a_{t}\) in a given state \(\mathbf{s}_{t}\). Figure 3 illustrates the agent-interaction with the volume. Similar to the existing studies [8; 10; 11], the local patch size of the state was set to 64 \(\times\) 64 \(\times\) 64 voxels.  
During training, a scalar reward \(r_{t}\) is provided after each step, in which a positive reward is assigned for a step towards the target and a negative reward is assigned otherwise. Specifically, the reward for transition from position \(\mathbf{p}_{t}\) to \(\mathbf{p}_{t+1}\) by action \(a_{t}\) can be indicated as follows:  
\[r_{t}=\begin{cases}-1&\text{if }|\mathbf{p}_{\text{goal}}-\mathbf{p}_{t+1}|>|\mathbf{p}_{ \text{goal}}-\mathbf{p}_{t}|\\ +1&\text{if }|\mathbf{p}_{\text{goal}}-\mathbf{p}_{t+1}|\leq\max(|\mathbf{p}_{\text{goal}}- \mathbf{p}_{t}|,\tau)\end{cases} \tag{1}\]  
where \(\tau\) represents a small distance within which the agent constantly receives positive rewards, indicating the convergence state. For training, the agent-trajectories after multiple episodes of transitions is recorded, where each transition can be denoted by \((\mathbf{s},a,\mathbf{\tilde{s}},\tau)\) where \(\mathbf{s}=\mathbf{s}_{t},a=a_{t},\mathbf{\tilde{s}}=\mathbf{s}_{t+1}\), and \(r=r_{t}\). The goal of training is to optimize the policy so that the expected cumulative reward over such trajectories is maximized.  
In a multi-landmark situation, individual agent-policies are usually trained on different trajectories rather than for different landmarks [8; 10; 11]. However, this is sub-optimal and inefficient because RVF and LVF in our problem have high similarity in features and high proximity in location. In the following, we describe the mirror environment that allows for efficient and improved training of a single agent-policy to localize both the RVF and LVF.