## 6 Actor-Critic Methods
### Asynchronous Advantage Actor-Critic (A3C) Method
#### 6.2.7 A3C Algorithm for Autonomous Navigation with Landmark Generators  
[PERSON] et al. [40] proposed a method that enhances the A3C algorithm by introducing a hierarchical navigation framework that integrates global and local planners connected by a dynamic sub-goal generator. The global planner uses traditional path planning algorithms like A* or Dijkstra to create a global path, while the local planner employs A3C to navigate to dynamically generated sub-goals, optimizing navigation efficiency in dynamic environments.  
The local planner is modeled as a partially observable Markov decision process (POMDP) defined as a 5-tuple \((T,S,O,A,R,\gamma)\), where \(T\) is the current time, \(S\) represents the environment states, \(O\) denotes the observed information, \(A\) is the action space, \(R\) is the reward function, and \(\gamma\) is the reward discount factor. To address the challenge of sparse rewards, a reward mechanism is designed as follows:  
\[R_{t}=r_{ts}+r_{td}+r_{tc}+r_{lst}, \tag{150}\]  
Figure 9: Algorithm flow for the method of [39].  
where  
\[r_{ts} = \begin{cases}50,&\text{if the robot reaches the sub-goal},\\ 0,&\text{otherwise}.\end{cases} \tag{151}\] \[r_{td} = \begin{cases}-0.5,&\text{if distance to obstacles}<d_{\text{safe}},\\ 0,&\text{otherwise}.\end{cases}\] (152) \[r_{tc} = \begin{cases}-20,&\text{if collision occurs},\\ 0,&\text{otherwise}.\end{cases}\] (153) \[r_{tst} = \begin{cases}-0.05,&\text{if movement distance}<r_{\text{min}},\\ 0,&\text{otherwise}.\end{cases} \tag{154}\]  
The policy \(\pi(a|s;\theta)\) and value \(V(s;\theta_{v})\) networks are enhanced with a shared feature extraction backbone and a memory module to retain long-term dependencies. The policy gradient for updating the actor network is as follows:  
\[\Delta\theta=\
abla_{\theta}\log\pi(a_{t}|s_{t};\theta)A(s_{t},a_{t};\theta_{v }), \tag{155}\]  
where the advantage function \(A(s_{t},a_{t};\theta_{v})\) is computed as follows:  
\[A(s_{t},a_{t};\theta_{v})=R_{t}+\gamma V(s_{t+1};\theta_{v})-V(s_{t};\theta_{v }). \tag{156}\]  
The critic network minimizes the temporal difference (TD) loss:  
\[L_{V}=\frac{1}{2}\big{(}R_{t}+\gamma V(s_{t+1};\theta_{v})-V(s_{t};\theta_{v} )\big{)}^{2}. \tag{157}\]  
The integration of the sub-goal generator with A3C further reduces the computational burden. The sub-goal generator dynamically identifies intermediate targets based on two types of frontiers, \(A\) and \(B\), defined in the robot's LiDAR data. Sub-goals are generated using the following rules:  
\[\text{For type }A\text{:}\] \[\text{sub-goal at }d_{\text{max}}\text{ along the frontier's midpoint}. \tag{158}\] \[\text{For type }B\text{:}\] \[\text{adjust based on safe distance }d_{\text{safe}}\text{ and obstacle edges}. \tag{159}\]  
The hierarchical design of global and local planners ensures robust navigation in dynamic environments, leveraging the memory-enhanced A3C to handle long-term dependencies and optimize navigation efficiency. The block diagram of the proposed hierarchical navigation method, shown in Figure 10, exhibits how the system integrates a global planner with a sub-goal generator to dynamically identify intermediate targets for the local planner. The feature extraction backbone processes the current state \(s_{t}\), and the reward function computes \(R_{t}\) based on proximity, collision, and progress metrics. The policy \(\pi(a|s;\theta)\) and value network \(V(s;\theta_{v})\) are optimized using the advantage function \(A(s_{t},a_{t})\), with the final output being the navigation action \(a_{t}\).