## 6 Actor-Critic Methods
### Asynchronous Advantage Actor-Critic (A3C) Method
#### 6.2.1. A3C Algorithm for Learning to Navigate in Complex Environments  
[PERSON] et al. (2017) have extended A3C by incorporating auxiliary tasks to improve navigation in complex environments. The architecture augments the standard A3C framework with two auxiliary outputs: depth prediction and loop closure detection. These tasks enhance data efficiency and representation learning by providing additional gradient updates during training. The first auxiliary task, depth prediction, estimates a depth map from RGB inputs. The loss function for depth prediction is implemented as a classification task, dividing depth values into discrete bins:  
\[L_{\text{depth}} = \frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{C}y_{ij}\log(y_{ij}), \tag{103}\]  
where \(y_{ij}\) represents the true probability of the \(j\)-th depth bin for the \(i\)-th pixel, \(\hat{y}_{ij}\) is the predicted probability, \(N\) is the number of pixels, and \(C\) is the number of depth bins. The second auxiliary task, loop closure detection, predicts whether the agent is revisiting a previously visited location. The training targets are generated based on position similarity between the current position and earlier positions in the trajectory. The binary classification loss is given by the following:  
\[L_{\text{loop}} = -\frac{1}{T}\sum_{i=1}^{T}\Big{(}l_{t}\log(\hat{l}_{t})+(1-l_{t}) \log(1-\hat{l}_{t})\Big{)}, \tag{104}\]where \(l_{t}\) is the ground truth loop closure label for time step \(t\), \(\hat{l}_{t}\) is the predicted probability, and \(T\) is the total number of steps. The overall loss function combines the A3C objective with these auxiliary losses. The total loss is as follows:  
\[L_{\text{total}} = L_{\text{A3C}}+\beta_{\text{depth}}L_{\text{depth}}+\beta_{ \text{loop}}L_{\text{loop}}, \tag{105}\]  
where \(\beta_{\text{depth}}\) and \(\beta_{\text{loop}}\) are coefficients balancing the contributions of the auxiliary tasks. To integrate auxiliary tasks with the A3C framework, the architecture uses a shared convolutional encoder to process RGB inputs. The shared features are passed to both the policy/value networks and the auxiliary task heads. Specifically, the depth prediction head predicts depth values from the convolutional features, while the loop closure head predicts loop closure probabilities using the hidden states of the LSTM layer. The policy \(\pi(a_{t}|s_{t};\theta)\) and value \(V(s_{t};\theta_{V})\) networks follow the A3C architecture and are updated with standard A3C gradients. The depth prediction output is computed from the convolutional features \(f_{t}\) as follows:  
\[\hat{y}_{ij} = \text{softmax}(W_{\text{depth}}f_{t}+b_{\text{depth}}), \tag{106}\]  
where \(W_{\text{depth}}\) and \(b_{\text{depth}}\) are the weights and biases of the depth prediction layer. The loop closure output is computed from the LSTM hidden states \(h_{t}\) as follows:  
\[\hat{l}_{t} = \text{sigmoid}(W_{\text{loop}}h_{t}+b_{\text{loop}}), \tag{107}\]  
where \(W_{\text{loop}}\) and \(b_{\text{loop}}\) are the weights and biases of the loop closure layer. These auxiliary tasks provide additional supervision signals, accelerating learning and improving the agent's ability to represent spatial and geometric features critical for navigation. The block diagram shown in Figure 6 presents how the proposed architecture extends A3C by integrating auxiliary tasks for depth prediction and loop closure detection. The shared encoder processes RGB inputs to extract features \(f_{t}\), which are passed to the policy and value networks as well as auxiliary task-specific heads. Depth prediction and loop closure detection contribute to the total loss \(L_{\text{total}}\), accelerating training and improving navigation performance. [PERSON] et al. [35] have extended A3C to address navigation in unknown rough terrain environments by incorporating an elevation map, depth images, and the robot's 3D orientation as inputs into the policy and value networks.