## 6 Actor-Critic Methods
### Asynchronous Advantage Actor-Critic (A3C) Method
#### 6.2.5 A3C Algorithm for Autonomous Navigation in Indoor Environments  
The study by [PERSON] et al. [38] builds on A3C by using the GA3C framework, which improves efficiency and scalability in parallel training. Instead of maintaining separate network copies for each agent, GA3C centralizes predictions and updates using a global deep neural network (DNN) model. This eliminates the need for synchronization and allows for efficient GPU utilization. The algorithm employs two global queues: a prediction queue for obtaining observations and a learning queue for training updates. The inputs to the network are the last four laser scans \(\{L_{t-3},L_{t-2},L_{t-1},L_{t}\}\), each with 1081 values, and a one-hot encoded orientation vector \(\mathbf{o}_{t}\) representing the direction of the goal. The laser scan values are normalized to the range \([0,1]\), and the orientation is encoded as follows:  
\[\mathbf{o}_{t}=\text{one-hot}(g_{\text{dir}})\quad\text{where}\;g_{\text{dir}} \in[0,360]. \tag{130}\]  
The network begins with two 1D convolutional layers:  
Conv1: \[1\times 9\times 16,\quad\text{stride: 4},\] (131) Conv2: \[1\times 5\times 32,\quad\text{stride: 2}.\] (132)  
The outputs are flattened and passed to a dense layer with 256 neurons to generate the shared latent representation \(\phi(s_{t})\). The policy network computes the probability distribution over actions \(\pi(a_{t}|s_{t};\theta)\):  
\[\pi(a_{t}|s_{t};\theta)=\text{softmax}(W_{\pi}\phi(s_{t})+b_{\pi}), \tag{133}\]  
where \(a_{t}\) represents discrete actions corresponding to linear and angular velocities, such as forward motion and turning. The value network estimates the state value \(V(s_{t};\theta_{v})\):  
\[V(s_{t};\theta_{v})=W_{V}\phi(s_{t})+b_{V}. \tag{134}\]  
The policy and value networks are updated using the A3C loss. The policy gradient is computed as follows:  
\[\Delta\theta=\
abla_{\theta}\log\pi(a_{t}|s_{t};\theta)A(s_{t},a_{t};\theta_{v })+\xi\
abla_{\theta}H(\pi(s_{t};\theta)), \tag{135}\]  
where \(A(s_{t},a_{t};\theta_{v})\) is the advantage function:  
\[A(s_{t},a_{t};\theta_{v})=r_{t}+\gamma V(s_{t+1};\theta_{v})-V(s_{t};\theta_{v }). \tag{136}\]  
The value loss is minimized to improve the value network:  
\[L_{V}=\frac{1}{2}\big{(}r_{t}+\gamma V(s_{t+1};\theta_{v})-V(s_{t};\theta_{v}) \big{)}^{2}. \tag{137}\]  
The reward function integrates goal-reaching rewards, collision penalties, and intermediate rewards. For goal-reaching, the reward is as follows:  
\[R_{\text{goal}}=20\quad\text{if the goal is reached.} \tag{138}\]  
For collisions:  
\[R_{\text{collision}}=-20\quad\text{if a collision occurs.} \tag{139}\]Intermediate rewards encourage progress toward the goal:  
\[R_{t}= \begin{cases}+1,&\text{if the distance to the goal decreases,}\\ -1,&\text{if the distance to the goal increases.}\end{cases}  \tag{140}\]  
The GA3C framework allows for high-speed parallel training with up to 32 simulation instances running concurrently, ensuring robust and efficient learning. A step-by-step pseudocode of the algorithm is presented in Algorithm 8. ```
Input: Laser scans \(\left\{L_{t-3},L_{t-2},L_{t-1},L_{t}\right\}\), orientation vector \(\mathbf{o}_{t}\), goal \(g\), reward function \(R\). Output: Optimal navigation actions \(a_{t}\).