## 6 Actor-Critic Methods
### Soft Actor-Critic (SAC)
#### 6.4.5 Velocity Range-Based SAC Reward Shaping for Mapless Navigation  
A novel SAC reward shaping method was proposed by [PERSON] and [PERSON] [50] in which a \"Velocity Range-Based Reward Shaping\" technique is devised to improve map-less navigation for autonomous mobile robots. This technique adjusts the reward function to prioritize efficient and safe movements by incorporating velocity-based evaluations into the reinforcement learning framework. The robot's state \(s_{t}\) is defined as follows:  
\[s_{t}=f(x_{t},p_{t}), \tag{252}\]  
where \(x_{t}\) represents LiDAR-based distance data, and \(p_{t}\) is the relative target coordinate.  
The action \(a_{t}=[v,\omega]\) consists of the linear velocity \(v\) and angular velocity \(\omega\), with velocity bounded by the following:  
\[v\in[-v_{\text{max}},v_{\text{max}}],\quad\omega\in[-\omega_{\text{max}}, \omega_{\text{max}}]. \tag{253}\]  
The reward function \(r(s_{t},a_{t})\) incorporates velocity-based evaluation and consists of the following:  
\[r(s_{t},a_{t})=r_{\text{goal}}+r_{\text{step}}+r_{\text{velocity}}, \tag{254}\]  
where  
\[r_{\text{goal}} = \begin{cases}r_{\text{arrive}},&\text{if }d_{t}<d_{\text{goal}},\\ r_{\text{collide}},&\text{if }\min_{x}<c_{\text{collision}},\end{cases} \tag{255}\] \[r_{\text{step}} = w_{1}d_{t}+w_{2}\theta_{t},\] (256) \[r_{\text{velocity}} = e_{t}, \tag{257}\]  
with \(d_{t}\) being the distance to the target, \(\theta_{t}\) the orientation error, \(e_{t}\) the velocity-based evaluation score, and \(w_{1},w_{2}\) the importance weights.  
The velocity evaluation score \(e_{t}\) is determined by dividing the velocity range \([-v_{\max},v_{\max}]\) into \(n\) segments and assigning evaluation scores \(e_{1},e_{2},\ldots,e_{n}\) to each range:  
\[v_{1}<v_{2}<\cdots<v_{n},\quad e_{1}<e_{2}<\cdots<e_{n}. \tag{258}\]  
The policy \(\pi_{\theta}(a_{t}|s_{t})\) is optimized using the SAC objective, modified to include the velocity-based reward:  
\[J(\pi)=\mathbb{E}\Bigg{[}\sum_{t=0}^{\infty}\gamma^{t}(r(s_{t},a_{t})+\alpha H (\pi(a_{t}|s_{t})))\Bigg{]}, \tag{259}\]  
where \(\alpha\) is the entropy temperature, and \(H(\pi(a_{t}|s_{t}))\) is the entropy of the policy:  
\[H(\pi(a_{t}|s_{t}))=-\int\pi(a_{t}|s_{t})\log\pi(a_{t}|s_{t})\,da_{t}. \tag{260}\]  
The value function \(V(s_{t})\) and Q-function \(Q(s_{t},a_{t})\) are trained using the following updates:  
\[V(s_{t}) = \mathbb{E}_{a_{t}\sim\pi}[Q(s_{t},a_{t})-\alpha\log\pi(a_{t}|s_{t })], \tag{261}\] \[L_{Q} = \mathbb{E}\Big{[}(Q(s_{t},a_{t})-(r(s_{t},a_{t})+\gamma V(s_{t+1} )))^{2}\Big{]}. \tag{262}\]  
By incorporating velocity-based evaluation into the reward function and leveraging the entropy-regularized objective, the method achieves stable and efficient map-less navigation in dynamic environments. A block diagram of the process flow of the method is presented in Figure 15. The input includes state information \(s_{t}\), derived from LiDAR data and the target position. State normalization prepares inputs for the actor and critic networks. The actor generates the policy \(\pi_{\theta}(a_{t}|s_{t})\), while the critic evaluates the Q-function. A hybrid reward function and entropy adjustment guide the training process. The replay buffer stores transitions for policy updates. The final policy update combines outputs, producing optimized navigation actions.  
Figure 15: Regenerated block diagram of the velocity range-based reward shaping technique for map-less navigation using LiDAR sensors [50].