## 6 Actor-Critic Methods
### Soft Actor-Critic (SAC)
#### 6.4.1 SAC Algorithm in Dynamic Path Planning for Mobile Robots  
[PERSON] et al. [46] improved the original SAC algorithm to address the challenges of mobile robot path planning in environments with static and dynamic obstacles. They introduced an enhanced reward function, state dynamic normalization, and a prioritized replay buffer to improve robustness and decision-making efficiency.  
The SAC framework is retained, where the policy \(\pi_{\theta}\) maps the current state \(s_{t}\) to a stochastic action \(a_{t}\):  
\[a_{t}\sim\pi_{\theta}(a_{t}|s_{t}), \tag{213}\]  
with the actor network optimized to maximize the entropy-regularized objective:  
\[J_{\pi}(\theta)=\mathbb{E}_{s_{t}\sim\mathcal{D},a_{t}\sim\pi_{ \theta}}[\alpha H(\pi_{\theta}(\cdot|s_{t}))-Q^{\pi}(s_{t},a_{t})], \tag{214}\]  
where \(H(\pi_{\theta})\) represents the entropy term, balancing exploration and exploitation, and \(\alpha\) is the entropy temperature coefficient.  
The critic networks are updated by minimizing the temporal difference (TD) error for the value function \(Q^{\pi}\):  
\[L_{Q}(\phi)=\mathbb{E}_{(s_{t},a_{t},r_{t},s_{t+1})\sim\mathcal{D}}\Big{[}\big{(} Q_{\phi}(s_{t},a_{t})-\hat{Q}(s_{t},a_{t})\big{)}^{2}\Big{]}, \tag{215}\]  
where the target \(\hat{Q}(s_{t},a_{t})\) includes the entropy term:  
\[\hat{Q}(s_{t},a_{t})=r_{t}+\gamma\mathbb{E}_{a_{t+1}\sim\pi_{\theta}}\big{[}Q^ {\pi}(s_{t+1},a_{t+1})-\alpha\log\pi_{\theta}(a_{t+1}|s_{t+1})\big{]}. \tag{216}\]  
To enhance the reward structure, a hybrid reward function \(r_{t}\) is defined to guide robot behavior:  
\[r_{t}=r_{t}^{\text{angle}}+r_{t}^{\text{target}}+r_{t}^{\text{collision}}+r_{t }^{\text{termination}}, \tag{217}\]  
where  
\[r_{t}^{\text{angle}} = \cos(\mathbf{I}_{\text{vector}},\mathbf{v}_{\text{robot}}), \tag{218}\] \[r_{t}^{\text{target}} = -0.4\frac{d_{t}}{500}+1.0+\left(\frac{d_{t}^{\text{prev}}-0.95d_{ t}}{24}\right),\] (219) \[r_{t}^{\text{collision}} = \begin{cases}-1,&\text{if $d_{\text{obs}}<5$m},\\ 0.5,&\text{if exiting danger zone},\end{cases}\] (220) \[r_{t}^{\text{termination}} = \begin{cases}100,&\text{if goal reached},\\ -6,&\text{if collision occurs},\\ -5,&\text{if border exceeded}.\end{cases} \tag{221}\]  
State dynamic normalization ensures stable training by normalizing the state vector to follow a normal distribution:  
\[\hat{s}_{t}=\frac{s_{t}-\mu}{\sigma}, \tag{222}\]  
where \(\mu\) and \(\sigma\) are the mean and standard deviation of the state distribution. The prioritized replay buffer selects samples with higher importance for training, weighted by the following:  
\[P(i)=\frac{p_{i}^{\beta}}{\sum_{k}p_{k}^{\beta}}, \tag{223}\]  
where \(p_{i}\) is the priority of sample \(i\) and \(\beta\) controls the influence of priority on sampling.  
This combination of techniques enables SAC to efficiently navigate in complex environments, outperforming the original SAC and PPO algorithms in terms of stability and cumulative rewards. A block diagram of the proposed SAC-based method for mobile robot navigation is shown in Figure 13. The input comprises state information \(s_{t}\), including LiDAR, velocity, and target observations. Normalized states are processed by the actor and critic networks. The prioritized replay buffer improves sampling, and the entropy adjustment refines exploration. The final policy update combines outputs to produce the optimized policy \(\pi_{\theta}\), which ensures efficient navigation actions.