## 2 Methodology
### Mirror Environment  
The proposed mirror environment is solely based on the laryngeal symmetry where each of the two vocal folds is almost a mirrored version of the other. We first model the left vocal fold \(\mathbf{p}_{t}\) in volume \(V\) as the right fold \(\mathbf{p}^{\prime}_{r}\) in volume \(V^{\prime}\), a laterally (along the \(X\)-axis) flipped version of the original volume \(V\). Then, we propose a single RL agent to localize both the right vocal fold (as \(\mathbf{p}_{r}\)) and left vocal fold (as \(\mathbf{p}^{\prime}_{r}\)), essentially arguing \(\mathbf{p}_{r}\) and \(\mathbf{p}^{\prime}_{r}\) to be the same target landmark with similar features. Figure 3: Proposed mirror environment. A single RL agent localizes the right fold in the original volume and left fold in the flipped volume. With this new model, agent interaction with the environment during localization remains the same. Except, the environment now can either be the original volume \(V\) or the mirrored volume \(V^{\prime}\), depending on the target being the right fold \(\mathbf{p}_{\text{goal}}=\mathbf{p}_{r}\) or the (mirrored) left fold \(\mathbf{p}_{\text{goal}}=\mathbf{p}^{\prime}_{r}\), respectively. Consequently, the reward in (1) is also calculated as \(r_{t}\) or \(r^{\prime}_{t}\) based on the target. Similar to previous RL frameworks [8; 10; 11], we represent our policy function by a 3D CNN that outputs optimal action-probabilities \(\pi(\mathbf{s},a)\) for an input state \(\mathbf{s}\). The CNN consists of three convolutional blocks, each having two convolutional layers followed by a max-pooling layer (see Figure 3). The last convolutional block is connected to two fully connected layers to produce the action probabilities. Algorithm 1 summarizes the policy training process with the mirror environment. ```
Input: Training CT volumes \(\mathbf{V}_{\text{train}}\), episodes per epoch \(E\) Output: Optimized policy \(\pi\)
1\(\pi\leftarrow\) random policy repeat
2 Trajectories \(\mathbb{T}=\emptyset\), mirror trajectories \(\mathbb{T}^{\prime}=\emptyset\)for\(\text{episode}=1\) to \(E\)do
3 Sample a volume \(V\in\mathbf{V}_{\text{train}}\) and a target \(\mathbf{p}_{\text{goal}}\in\{\mathbf{p}_{r},\mathbf{p}_{l}\}\)if\(\mathbf{p}_{\text{goal}}=\mathbf{p}_{l}\)then
4\(V=V^{\prime}\leftarrow\) laterally flipped \(V\)\(\mathbf{p}_{\text{goal}}=\mathbf{p}^{\prime}_{r}\leftarrow\) laterally flipped \(\mathbf{p}_{l}\)
5 Gather trajectories \((\mathbf{s}^{\prime},a^{\prime},\mathbf{s}^{\prime},r^{\prime})\)into \(\mathbb{T}^{\prime}\) applying \(\pi\)
6 else
7 Gather trajectories \((\mathbf{s},a,\mathbf{\tilde{s}},r)\) into \(\mathbb{T}\) applying \(\pi\)
8
9 Optimize \(\pi\) for the PPO objective in (4) on \(\mathbb{T}\) and \(\mathbb{T}^{\prime}\) until convergence
```  
**Algorithm 1**RL policy training with mirror environment  
At each epoch, we conduct a number of localization episodes. Each episode is initiated by sampling a training volume \(V\) and a target landmark \(\mathbf{p}_{\text{goal}}\) between the right and left folds. Based on the sampled target, the environment is set with either the original volume \(V\) or mirrored volume \(V^{\prime}\). Then, subsequent steps are performed by applying the current policy. Running multiple episodes, we record the trajectories for the original and mirror volumes as \(\mathbb{T}\) and \(\mathbb{T}^{\prime}\), respectively. Now, the policy is updated so that the expected cumulative reward on both \(\mathbb{T}\) and \(\mathbb{T}^{\prime}\) is maximized. We follow the widely used proximal policy optimization (PPO) framework [12] where _advantage_ (i.e., the improvement of the current policy over the previous) is maximized to reduce variance during training. Instead of using the reward directly, the improvement is estimated using a value function.