## 3 Methodology
### Reinforcement Learning Setup
#### 3.2.3 Reward Design  
We design the reward function from three aspects to avoid obstacles in the process of reaching the goal and make the action smooth. The composition of reward \(r\) is:  
\[r=r_{g}+r_{c}+r_{s}+r_{a}, \tag{3}\]  
where \(r_{g}\) is reaching goal reward, \(r_{c}\) is the collision avoidance reward, \(r_{s}\) is action smoothing reward, and \(r_{a}\) is action urge reward.  
The arrival reward guides the agent to approach the goal, which consists of the arrival reward and the distance target approaching reward:  
\[r_{g}=\begin{cases}r_{arrived}&if\big{\|}p_{t}-p_{g}\big{\|}<d_{min}\\ \alpha\big{(}\big{\|}p_{t-1}-p_{g}\big{\|}-\big{\|}p_{t}-p_{g}\big{\|}\big{)} &\text{otherwise}\end{cases}, \tag{4}\]  
where \(r_{arrived}\) is the reward for reaching the goal, \(p_{t}\) is current position of the agent, \(p_{g}\) is the position of goal, \(d_{min}\) is the distance to judge the arrival, and \(\alpha\) is a constant. If the distance between the position of the agent and goal is within \(d_{min}\), it is determined that it has reached the goal, giving the arrival reward \(r_{arrived}\). If the agent has not arrived, according to the distance between the current position and the goal and the same distance at the previous moment, we calculate the distance difference close to the target point, and perform \(\alpha\) weighting as a reward for approaching the target point.  
Figure 3: For the sampling of laser data, the data volume changes from 1080 to 24, taking the minimum value every 11.25\({}^{\circ}\) and centering it.  
Collision avoidance reward guides the agent away from obstacles:  
\[r_{c}=\begin{cases}2r_{collision}&\text{if }\|p_{t}-O_{k}\|<R\\ r_{collision}&\text{if }R\leqslant\|p_{t}-O_{k}\|<2R\,,\\ r_{laser}&\text{otherwise}\end{cases} \tag{5}\]  
where \(r_{collision}\) is collision warning reward, \(O_{k}(0\leq k\leq N)\) is the position of obstacle, \(R\) is collision distances, and \(r_{laser}\) is laser distance reward. If the distance is less than \(R\), it is determined as a collision and a collision penalty of \(2r_{collision}\) is given. If the distance is less than \(2R\), a collision warning penalty \(r_{collision}\) is given. For other cases, the laser judges the openness in the direction of travel, and the penalty \(r_{laser}\) is given by the normalized laser.  
Action smoothing reward guides the agent to move smoother:  
\[r_{s}=\left\{\begin{array}{ll}r&\text{if }\|v_{t}-v_{t-1}\|>v_{min}\\ 0&\text{otherwise}\end{array}\right.+\left\{\begin{array}{ll}r_{\omega}& \text{if }\|\omega_{t}-\omega_{t-1}\|>\omega_{min}\\ 0&\text{otherwise}\end{array}\right.\,, \tag{6}\]  
where \(r_{v}\) is translational velocity smoothing reward, \(v_{t}\) is current translational velocity, \(v_{min}\) is translational velocity variation limitation, \(r_{\omega}\) is rotational velocity smoothing reward, \(\omega_{t}\) is current rotational velocity, and \(\omega_{min}\) is translational velocity variation limitation. When the change in velocity between the previous moment(\(v_{t-1}\) or \(\omega_{t-1}\)) and this moment(\(v_{t}\) or \(\omega_{t}\)) is greater than \(v_{min}\) or \(\omega_{min}\), the corresponding velocity penalty \(r_{v}\) or \(r_{\omega}\) is given.  
Action urge reward prevents the agent from stopping in place or moving too slowly:  
\[r_{a}=\begin{cases}r_{uv}&\text{if }v_{t}<v_{umin}\\ 0&\text{otherwise}\end{cases}\,, \tag{7}\]  
where \(r_{uv}\) is translational velocity urge reward and \(v_{umin}\) is translational velocity limit. If the translational velocity is less than the minimum velocity limit \(v_{umin}\), a penalty \(r_{uv}\) is given for moving too slowly.  
We set \(d_{min}=0.25\), \(r_{arriod}=100\), \(\alpha=420\), \(R=0.25\), \(r_{collision}=-100\), \(r_{v}=r_{\omega}=-2\), \(v_{min}=\omega_{min}=0.5\), \(r_{uv}=-2\), \(v_{umin}=0.2\) in the training.