## 6 Actor-Critic Methods
### Asynchronous Advantage Actor-Critic (A3C) Method
#### 6.2.3 A3C Algorithm for Teaching a Machine to Read Maps  
The study by [PERSON] et al. [36] introduces a modular architecture that decomposes the navigation task into intermediate subtasks, each handled by a specialized module. These modules are trained either independently or in conjunction with the main A3C policy. The subtasks include visible local map estimation, recurrent localization, map interpretation, and reactive action execution.  
The visible local map network processes the 3D RGB visual input and a discretized 3-hot encoded orientation angle \(\hat{\text{a}}\). The map excerpt is gated using the visible field estimation:  
\[\text{Visible Local Map} = \text{Map Excerpt}\cdot\text{Visible Field}, \tag{114}\]  
Figure 7: Block diagram of the deep reinforcement learning framework for robot navigation in unknown rough terrain [35].  
where \(\cdot\) denotes element-wise multiplication. The visible local map output is used to construct an egocentric local map.  
The recurrent localization cell integrates the visible local map excerpts into an egocentric local map. It estimates the egomotion as follows:  
\[s_{t} = \text{softmax}(f(s_{t-1},\hat{\alpha},a_{t-1},r_{t-1})+\text{LM}_{ \text{est},t-1}*\hat{V}), \tag{115}\]  
where \(f(\cdot)\) is a two-layer feedforward network, \(*\) represents 2D convolution, \(\text{LM}_{\text{est},t-1}\) is the previous local map estimation, and \(\hat{V}\) is the current visible local map input. The estimated local map is updated as follows:  
\[\text{LM}_{\text{est},t} = [\text{LM}_{\text{est},t-1}*s_{t}+\hat{V}]^{+0.5}_{-0.5}, \tag{116}\]  
where \([\cdot]^{+0.5}_{-0.5}\) denotes clipping to the range \([-0.5,+0.5]\). The updated map is combined with a feedback map to enhance localization:  
\[\text{LM}_{\text{est+mfb},t} = [\text{LM}_{\text{est},t}+\lambda\cdot\text{LM}_{\text{mfb},t-1}* s_{t}]^{+0.5}_{-0.5}. \tag{117}\]  
The location probability distribution is calculated by correlating the estimated local map with the global map:  
\[\{p_{\text{loc},i}\}_{i=1}^{N} = \text{softmax}(M*\text{LM}_{\text{est+mfb},t}), \tag{118}\]  
where \(M\) is the global map and \(N\) is the number of discrete location cells.  
The map interpretation network processes the global map to identify rewarding locations and outputs a short-term target direction (STTD). The STTD at each location is derived using a shortest path algorithm, producing a direction distribution \(\{d_{\text{sttd},i}\}_{i=1}^{N}\).  
The reactive agent combines the STTD distribution, the normalized entropy of the location probability \(H_{\text{loc}}\), and the estimated target distance to select actions. It maximizes the total reward, which combines extrinsic and intrinsic rewards:  
\[I_{\text{explor},t} = H_{\text{loc},t-1}-H_{\text{loc},t}, \tag{119}\] \[I_{\text{exploit},t} = \mathbf{e}_{t}^{\top}\cdot\mathbf{d}_{\text{sttd},t-1}, \tag{120}\]  
where \(\mathbf{e}_{t}\) is the egomotion vector and \(\mathbf{d}_{\text{sttd},t-1}\) is the STTD vector. The agent's policy is updated using A3C's policy gradient:  
\[\Delta\theta = \
abla_{\theta}\log\pi(a_{t}|s_{t};\theta)A(s_{t},a_{t};\theta_{ v})+\zeta\
abla_{\theta}H(\pi(s_{t};\theta)), \tag{121}\]  
where \(A(s_{t},a_{t};\theta_{v})\) is the advantage function, and \(H(\pi(s_{t};\theta))\) is the entropy of the policy.  
The proposed architecture improves navigation by integrating modular components, leveraging A3C's stability, and introducing additional supervision signals through intrinsic rewards and auxiliary tasks. Algorithm 7 presents the step-by-step pseudocode for implementing the proposed architecture.