## 6 Actor-Critic Methods
### Soft Actor-Critic (SAC)
#### 6.4.3 Dynamic Obstacle Avoidance and Path Planning Integration  
[PERSON] et al. (2018) developed an SAC-based framework for decentralized dynamic obstacle avoidance and path planning integration. The enhancements to the SAC algorithm include a tailored state representation, a hybrid reward function, and a combination of reinforcement learning and classical path planning for improved navigation. The state \(s_{t}\) is formulated as follows:  
\[s_{t}=\{s_{t}^{\text{lidar}},s_{t}^{\text{goal}},s_{t}^{\text{ speed}}\}, \tag{231}\]  
where \(s_{t}^{\text{lidar}}=[o_{t-2}^{l},o_{t-1}^{l},o_{t}^{l}]\) represents LiDAR measurements from three consecutive time steps to capture the temporal dynamics of obstacles. The goal state is \(s_{t}^{\text{goal}}=[x,y]\), the relative position of the robot to the target, and the speed state \(s_{t}^{\text{speed}}=[v,\omega]\), consisting of forward velocity \(v\) and rotational velocity \(\omega\). The action \(a_{t}=[v,\omega]\) is defined in a continuous space:  
\[v\in[0,0.55],\quad\omega\in[-0.60,0.60]. \tag{232}\]The reward function \(R(s_{t},a_{t})\) combines terms for goal achievement, collision avoidance, and control efficiency:  
\[R(s_{t},a_{t})=R_{g}+R_{c}+R_{\omega}, \tag{233}\]  
where  
\[R_{g} = \begin{cases}10,&\text{if dist}_{\text{curr}}<0.5,\\ \text{dist}_{\text{prev}}-\text{dist}_{\text{curr}},&\text{otherwise},\end{cases} \tag{234}\] \[R_{c} = \begin{cases}-10,&\text{if collision},\\ 0,&\text{otherwise},\end{cases}\] (235) \[R_{\omega} = \begin{cases}-0.1|\omega|,&\text{if }|\omega|>0.6,\\ 0,&\text{otherwise}.\end{cases} \tag{236}\]  
The SAC objective is augmented to optimize this hybrid reward while maintaining exploration through entropy regularization:  
\[J(\theta)=\mathbb{E}\left[\sum_{t=0}^{T}(r(s_{t},a_{t})+\alpha H(\pi(a_{t}|s_{ t})))\right], \tag{237}\]  
where \(H(\pi(a_{t}|s_{t}))\) is the entropy of the policy, encouraging diverse action exploration.  
Figure 14: Block diagram of the sim-to-real transfer method of [47].  
To address navigation inefficiencies and conflicts between obstacle avoidance and target-reaching, a classical path planning component is integrated. The global path planner generates a trajectory using an A* algorithm, which provides a \"look-ahead point\" \(p_{\text{lookahead}}\) for the SAC agent:  
\[p_{\text{lookahead}}=p_{\text{path}}(d_{\text{lookahead}}), \tag{238}\]  
where \(d_{\text{lookahead}}\) is the distance parameter guiding the point selection along the path. The relative position of the look-ahead point is incorporated into the RL agent's input:  
\[s_{t}^{\text{goal}}=[p_{\text{lookahead}}-p_{\text{robot}}]. \tag{239}\]  
The actor and critic networks are designed with convolutional layers to process LiDAR data and fully connected layers for speed and goal components. The critic evaluates the Q-value as follows:  
\[Q(s_{t},a_{t})=r(s_{t},a_{t})+\gamma\mathbb{E}_{s_{t+1}\sim\pi}[V(s_{t+1})], \tag{240}\]  
and the value function \(V(s_{t})\) is as follows:  
\[V(s_{t})=\mathbb{E}_{a_{t}\sim\pi}[Q(s_{t},a_{t})-\alpha\log\pi(a_{t}|s_{t})]. \tag{241}\]  
By combining SAC with classical path planning and integrating dynamic obstacle prediction into the state representation, the method achieves robust and efficient navigation in complex environments. Algorithm 12 presents a step-by-step pseudocode for the method.