## 3 FGRL for Mapless Navigation
### RL-Based Mapless Navigation
#### 3.1.4 Reward Function  
The objective of the UGV navigation task is to successfully arrive at the target location without colliding with any potential obstacles. To this end, the RL agent aims to obtain the most cumulative rewards, and the reward function is defined as follows:  
\[r=r_{\text{collision}}+r_{\text{arrive}}+r_{\text{distance}}+r_{\text{speed}}. \tag{4}\]  
During the training period, if a collision event occurs, the learning agent will receive a large negative penalty \(r_{\text{collision}}\):  
\[r_{\text{collision}}=\left\{\begin{array}{ll}-200,&d_{min}\leq d_{lim}\\ -e^{\beta((d_{min}/d_{s})-0.5)},&d_{lim}<d_{min}\leq 2d_{lim}\\ 0,&\text{otherwise},\end{array}\right. \tag{5}\]  
where \(d_{min}\) represents the shortest value of LiDAR readings, and \(d_{s}\) represents the normal detection distance of LiDAR readings without obstacles. \(d_{lim}\) defines the limited distance at which collisions can occur, and \(\delta\) denotes the collision adjustment coefficient used to modify the collision reward function curve within the range \([d_{lim},2d_{lim}]\), which is a negative value. According to the above rules, if the distance between the robot and an obstacle is less than the limited value, it will trigger a collision event and receive a large negative reward \(-200\). If the distance between the robot and an obstacle is less than twice the limited distance but larger than this limit, the negative penalty increases as the LiDAR readings become smaller. In this study, we set \(d_{s}=3.5\) m, \(d_{lim}=0.5\) m, and \(\delta=-15\) to allow the reward function to better cover the reward range \([-200,0]\), effectively reflecting the reward adjustment as the UGV approaches obstacles.  
Since the ultimate goal of UGVs is to reach the target location, a positive reward \(r_{\text{arrive}}\) is obtained if it can successfully arrive at its destination without collisions, with the condition that the distance between the robot's center and the target location is less than \(0.5\) m. Here, we define a sparse reward to assign a large positive reward:  
\[r_{\text{arrive}}=\left\{\begin{array}{ll}200,&\text{if success}\\ 0,&\text{otherwise}.\end{array}\right. \tag{6}\]  
To solve the problem of sparse rewards during the training process, we also design a continuous reward function, where the agent can obtain feedback \(r_{\text{distance}}\) at each time step to provide a heuristic for the target location, as follows:  
\[r_{\text{distance}}=\mu d_{\text{step}}, \tag{7}\]  
where \(d_{\text{step}}\) indicates the difference between the estimated distance to the target location at current time step and the one at the previous time step. Here, \(\mu\) denotes the adjustment coefficient of the distance reward. The larger the value of \(\mu\), the higher the reward for each step, which encourages the UGV to move toward the target. However, it is important to note that the value of \(\mu\) is influenced by the final target arrival reward \(r_{\text{arrive}}\). If \(\mu\) is set too high, it can reduce the attractive force of the target, causing the UGV to approach it more slowly as it gets closer, rather than reaching it directly. Therefore, selecting an appropriate value for \(\mu\) is crucial for balancing the trade-off between approaching the target and reaching the target in the reward function.  
During the training process, the moving speed of UGVs may be too fast due to the incentive of distance rewards. For safety concerns, we further consider a penalty \(r_{\text{speed}}=r_{v}+r_{\omega}\) to regulate the linear and angular velocity, as follows:  
\[r_{v}=\left\{\begin{array}{ll}-5,&v\geq v_{max}\\ 0,&\text{otherwise},\end{array}\right. \tag{8}\]  
\[r_{\omega}=\left\{\begin{array}{ll}-5,&|\omega|\geq\omega_{max}\\ 0,&\text{otherwise}.\end{array}\right. \tag{9}\]  
The safety term works when the linear velocity \(v\) and the angular velocity \(\omega\) exceed the maximum permitted values \(v_{max}\) and \(\omega_{max}\), respectively.