## 6 Actor-Critic Methods
### Asynchronous Advantage Actor-Critic (A3C) Method
#### 6.2.6. A3C-Based End-to-End Navigation Strategy  
The study by [PERSON] et al. [39] introduced an Intrinsic Curiosity Module (ICM) to address the challenges of sparse rewards in navigation tasks. The ICM generates intrinsic rewards by leveraging the prediction error of a forward model, thereby encouraging exploration in complex environments. Additionally, the network is enhanced with two LSTM layers to capture temporal dependencies, enabling the agent to better utilize previous observations, actions, and rewards.  
The policy \(\pi(a_{t}|s_{t};\theta_{p})\) and value \(V(s_{t};\theta_{v})\) are represented by neural networks with parameters \(\theta_{p}\) and \(\theta_{v}\), respectively. The A3C policy gradient and value loss functions are extended to incorporate intrinsic rewards. The policy loss is as follows:  
\[f_{\pi}(\theta_{p})=\log\pi(a_{t}|s_{t};\theta_{p})\cdot(R_{t}-V(s_{t};\theta_ {v})), \tag{141}\]  
where the total reward \(R_{t}\) combines extrinsic and intrinsic rewards:  
\[R_{t}=\sum_{i=0}^{k-1}\gamma^{i}r_{t+i}+\gamma^{k}V(s_{t+k};\theta_{v}). \tag{142}\]  
The value loss minimizes the discrepancy between the predicted and actual returns:  
\[f_{v}(\theta_{v})=\frac{1}{2}(R_{t}-V(s_{t};\theta_{v}))^{2}. \tag{143}\]  
To calculate intrinsic rewards, the ICM includes an inverse model and a forward model. The inverse model predicts the action \(a_{t}\) given consecutive states \(s_{t}\) and \(s_{t+1}\):  
\[\hat{a}_{t}=g(s_{t},s_{t+1};\theta_{t}), \tag{144}\]  
where \(\theta_{t}\) are the parameters of the inverse model. The loss for the inverse model is as follows:  
\[L_{I}=\sum_{i=1}^{n}-P(a_{t}^{i})\log q(\hat{a}_{t}^{i}), \tag{145}\]  
with \(P(a_{t}^{i})\) as the true action probability and \(q(\hat{a}_{t}^{i})\) as the predicted action probability. The forward model predicts the next state feature \(\phi(s_{t+1})\) based on the current state feature \(\phi(s_{t})\) and action \(a_{t}\):  
\[\hat{\phi}(s_{t+1})=f(\phi(s_{t}),a_{t};\theta_{F}), \tag{146}\]  
where \(\theta_{F}\) are the parameters of the forward model. The forward model loss is as follows:  
\[L_{F}=\frac{1}{2}\|\hat{\phi}(s_{t+1})-\phi(s_{t+1})\|_{2}^{2}. \tag{147}\]  
The intrinsic reward \(r_{t}^{i}\) is derived from the forward model's prediction error:  
\[r_{t}^{i}=\frac{\eta}{2}\|\hat{\phi}(s_{t+1})-\phi(s_{t+1})\|_{2}^{2}, \tag{148}\]  
where \(\eta>0\) scales the intrinsic reward. The total loss function combines policy, value, inverse, and forward losses:  
\[\mathcal{L}=-\lambda\mathbb{E}_{\pi(a_{t}|s_{t};\theta_{p})}\left[\sum_{t}r_ {t}\right]+(1-\beta)L_{I}+\beta L_{F}, \tag{149}\]  
where \(\lambda\) adjusts the weight of the policy gradient, and \(\beta\) balances the forward and inverse model losses.  
This method significantly enhances A3C by addressing sparse rewards and enabling robust navigation in complex environments. The use of intrinsic rewards, alongside the temporal memory provided by LSTMs, allows the agent to explore effectively and generalize better from simulation to real-world scenarios. The block diagram of Figure 9 illustrates the integration of the Intrinsic Curiosity Module (ICM) with A3C. The input states \(s_{t}\) and \(s_{t+1}\) are processed to extract features \(\phi(s_{t})\) and \(\phi(s_{t+1})\), which feed into the inverse and forward models. The forward model generates intrinsic rewards \(r_{t}^{j}\), while the policy and value networks compute the action policy \(\pi(a_{t}|s_{t};\theta_{p})\) and value estimate \(V(s_{t};\theta_{v})\). The output is the navigation action \(a_{t}\), optimized to balance exploration and exploitation in sparse-reward environments.