## 6 Actor-Critic Methods
### Soft Actor-Critic (SAC)
#### 6.4.2. Sim-to-Real Transfer with Incremental Environment Complexity  
The study by [PERSON] et al. [47] proposed a method that builds on SAC by introducing incremental environment complexity and an improved reward shaping mechanism to achieve robust depth-based mapless navigation for mobile robots. The main enhancements address the sim-to-real transfer problem by combining domain randomization with structured environment transitions. The SAC framework is used, where the policy \(\pi_{\theta}(a_{t}|s_{t})\) maps states \(s_{t}\) to actions \(a_{t}\), optimizing an entropy-regularized objective:  
\[J(\theta)=\mathbb{E}_{\pi_{\theta}}\Bigg{[}\sum_{t=1}^{T}(r(s_{t},a_{t})+ \alpha H(\pi_{\theta}(\cdot|s_{t})))\Bigg{]}, \tag{224}\]  
where \(H(\pi_{\theta})\) is the entropy of the policy:  
\[H(\pi_{\theta})=-\sum_{a\in A}\pi_{\theta}(a|s)\log\pi_{\theta}(a|s), \tag{225}\]  
and \(\alpha\) is a temperature parameter balancing exploration and exploitation.  
The Q-value and value functions are defined as follows:  
\[Q_{w}(s_{t},a_{t}) = r(s_{t},a_{t})+\gamma\mathbb{E}_{s_{t+1}\sim\rho_{\pi}}\big{[}V_ {\psi}(s_{t+1})\big{]}, \tag{226}\] \[V_{\psi}(s_{t}) = \mathbb{E}_{a_{t}\sim\pi_{\theta}}\big{[}Q_{w}(s_{t},a_{t})- \alpha\log\pi_{\theta}(a_{t}|s_{t})\big{]}. \tag{227}\]  
The reward function is shaped to guide the robot's behavior effectively during training. The reward \(r(s_{t},a_{t})\) is defined as follows:  
Figure 13: Block diagram of the dynamic path planning algorithm for mobile robots using SAC algorithm [46].  
\[r(s_{t},a_{t})=\begin{cases}C\cdot dR_{t}\cdot V_{r},&\text{if }dR_{t}>0,\\ r_{\text{recede}},&\text{if }dR_{t}\leq 0,\\ r_{\text{reached}},&\text{if }d_{t}<d_{\text{min}},\\ r_{\text{collision}},&\text{if collision detected},\end{cases} \tag{228}\]  
where \(dR_{t}=d_{t-1}-d_{t}\) is the change in distance to the target, and \(V_{r}\) is a velocity reduction factor:  
\[V_{r}=(1-\max(v_{t},0.1))\cdot\frac{1}{\max(d_{t},0.1)}. \tag{229}\]  
The constants \(r_{\text{reached}}\), \(r_{\text{collision}}\), and \(r_{\text{recede}}\) are set to encourage task completion while penalizing collisions and ineffective movements.  
Incremental complexity is introduced by progressively training the policy in three simulated environments (empty room, static obstacles, and mixed static and dynamic obstacles). Transitions between environments are governed by success rates \(S_{\text{rate}}\) over recent episodes. If \(S_{\text{rate}}>\alpha\), the agent progresses to a more complex environment; if \(S_{\text{rate}}<\beta\), it returns to a simpler environment. The policy update is performed by minimizing the KL divergence:  
\[\pi^{\text{new}}=\arg\min_{\pi^{t}}\text{D}_{\text{KL}}\big{(}\pi^{t}(\cdot|s_ {t})\|\exp(Q_{\pi^{\text{old}}}(s_{t},\cdot)-\log Z_{\pi^{\text{old}}}(s_{t}) )\big{)}. \tag{230}\]  
Here, \(Z_{\pi^{\text{old}}}(s_{t})\) is a partition function that normalizes the distribution.  
This approach ensures smooth transitions between simulation and real-world scenarios by gradually increasing task complexity and refining the policy through domain randomization and reward shaping.  
A block diagram of the SAC-based method is presented in Figure 14. The input includes state \(s_{t}\) with LiDAR, velocity, and target observations. State normalization refines inputs for the actor and critic networks. The actor generates a policy \(\pi_{\theta}(a_{t}|s_{t})\), while the critic evaluates value functions. The hybrid reward function and entropy adjustment guide the training process. The policy update refines \(\pi_{\theta}\), producing optimized navigation actions.