## 6 Actor-Critic Methods
### Asynchronous Advantage Actor-Critic (A3C) Method
#### 6.2.4 A3C Algorithm for Target-Driven Visual Navigation in Indoor Scenes  
[PERSON] et al. [37] extended A3C by introducing a target-driven visual navigation framework based on a deep Siamese actor-critic network. The key innovation is the use of a dual input structure, where both the current state \(s_{t}\) and the target representation \(g\) are processed to jointly estimate policy \(\pi\) and value \(V\). This allows generalization to unseen targets without re-training.  
The network takes as input two images: \(s_{t}\), representing the agent's current observation, and \(g\), representing the target. Both inputs are processed through Siamese layers that share weights, producing embeddings \(\phi(s_{t})\) and \(\phi(g)\) in a shared latent space:  
\[\phi(s_{t})=f_{\text{Siamese}}(s_{t}),\quad\phi(g)=f_{\text{Siamese}}(g). \tag{122}\]  
The embeddings are concatenated to form a joint representation:  
\[\phi_{\text{joint}}=\text{concat}(\phi(s_{t}),\phi(g)). \tag{123}\]  
This joint representation is passed through scene-specific layers to capture environment-specific features. The scene-specific representation is used to compute the policy \(\pi(a_{t}|s_{t},g;\theta)\) and the value \(V(s_{t},g;\theta_{v})\). The policy network predicts the probability distribution over actions:  
\[\pi(a_{t}|s_{t},g;\theta)=\text{softmax}(W_{\pi}\phi_{\text{joint}}+b_{\pi}). \tag{124}\]  
The value network estimates the expected future reward:  
\[V(s_{t},g;\theta_{v})=W_{V}\phi_{\text{joint}}+b_{V}. \tag{125}\]  
The policy is updated using the A3C policy gradient:  
\[\Delta\theta=\
abla_{\theta}\log\pi(a_{t}|s_{t},g;\theta)A(s_{t},g;\theta_{v })+\zeta\
abla_{\theta}H(\pi(s_{t},g;\theta)), \tag{126}\]where \(A(s_{t},g;\theta_{v})\) is the advantage function:  
\[A(s_{t},g;\theta_{v})=r_{t}+\gamma V(s_{t+1},g;\theta_{v})-V(s_{t},g;\theta_{v}), \tag{127}\]  
and \(H(\pi(s_{t},g;\theta))\) is the entropy of the policy to encourage exploration.  
The value network is updated by minimizing the squared error:  
\[L_{V}=\frac{1}{2}\big{(}r_{t}+\gamma V(s_{t+1},g;\theta_{v})-V(s_{t},g;\theta_ {v})\big{)}^{2}. \tag{128}\]  
The network leverages asynchronous training, where multiple threads independently interact with the environment. Each thread collects gradients and updates the shared model parameters. Generic Siamese layers are updated across all targets and scenes, while scene-specific layers are fine-tuned for each environment, enabling both inter-target and inter-scene generalization.  
The reward function for navigation includes a goal-reaching reward \(R_{\text{goal}}\) and a time penalty \(R_{\text{time}}\):  
\[R_{t}=\begin{cases}10.0,&\text{if goal is reached},\\ -0.01,&\text{otherwise}.\end{cases} \tag{129}\]  
This framework improves upon A3C by enabling generalization to unseen targets through the Siamese structure and shared parameter updates, while maintaining data efficiency through asynchronous training. Figure 8 shows a block diagram of the proposed target-driven navigation method based on A3C. The Siamese network processes the current state \(s_{t}\) and target \(g\) to generate embeddings \(\phi(s_{t})\) and \(\phi(g)\), which are concatenated to form a joint representation \(\phi_{\text{joint}}\). Scene-specific layers extract environment-specific features used by the policy and value networks. The policy network outputs navigation actions \(a_{t}\), while the value network evaluates future rewards. Both networks are updated asynchronously using the A3C framework.  
Figure 8: Block diagram of the deep reinforcement learning framework proposed in [37].