# Self-Learning Robot Autonomous Navigation with Deep Reinforcement Learning Techniques
## 2 Materials and Methods
### Reward Functions
\begin{array}{ccc}0&,&d>d_{limit}\\ \frac{d-d_{limit}}{d_{limit}}&,&d\leq d_{limit}\end{array} \right. \tag{9}\]  
(2) Legal requirements  
Legal requirements limit the maximum speed that the robot can travel. The robot must control its linear speed \(v_{rob}\) so that it does not exceed this maximum value. The implementation of this reward function is very simple as well (10). There are two speed thresholds (\(v_{1}\) and \(v_{2}\)). If the speed does not exceed the lower threshold \(v_{1}\), the reward will be 0. The reward will be linearly interpolated between 0 and \(-1\) between the first and second thresholds. \[r_{legal}=\left\{ \begin{array}{ccc}0&,&v_{rob}<v_{1}\\ -\frac{v_{rob}-v_{1}}{v_{2}-v_{1}}&,&v_{1}\leq v_{rob}\leq v_{2}\\ -1&,&v_{rob}>v_{2}\end{array} \right. \tag{10}\]  
If the legal maximum velocity is \(v_{legal}\), \(v_{1}\) can be equal to \(v_{legal}\) and \(v_{2}\) can be set a little bit higher than \(v_{legal}\). Figure 3: Safety margin around the obstacle footprint. If the robot enters the area delimited by the safety margin, a negative reward will be forwarded to the agent during the learning process. 3. Comfort requirements Comfort can be defined as the pleasant feeling of the people traveling inside the robot. Feelings of discomfort can come from different sources. The three main sources that can decrease the feeling of comfort are listed below: * High acceleration values (longitudinal and lateral/angular) * High jerk values (longitudinal and lateral/angular) * Low-frequency acceleration over long periods of time (feeling of sickness) In this work, the first source of discomfort is implemented. The reward function (11) defines a window given two acceleration values, \(\hat{w}_{1}\) and \(\hat{w}_{2}\). Accelerations below \(\hat{w}_{1}\) are not penalized. Accelerations within \(\hat{w}_{1}\) and \(\hat{w}_{2}\) are penalized using linear interpolation. The maximum penalization value is clipped to \(-1\) if the acceleration is higher than \(\hat{w}_{2}\). \[r_{comfort}=\left\{ \begin{array}{ccc}0&,&\hat{w}_{rob}<\hat{w}_{1}\\ -&\frac{\hat{w}_{rob}-\hat{w}_{1}}{\hat{w}_{2}-\hat{w}_{1}}&,&\hat{w}_{1}\leq \hat{w}_{rob}\leq\hat{w}_{2}\\ -1&,&\hat{w}_{rob}>\hat{w}_{2}\end{array} \right.\] (11) Reward functions for longitudinal and angular accelerations are added together to build the final comfort function. (4) Task-oriented requirements It is possible to define multiple and specific tasks to be accomplished during autonomous navigation. In this work, the robot is intended to reach a specific destination pose. For that purpose, the robot is rewarded when facing the right direction towards the destination pose. The robot will be also rewarded when traveling at the maximum legal speed. For the first task, if the direction to the destination pose \(\theta_{target}\) (determined by visual sensors) is not the same as the robot's heading \(\theta_{rob}\), the robot is penalized according to (12). \[r_{destination}=-\frac{\left|\theta_{target}-\theta_{rob}\right|}{\pi}\] (12) If the destination cannot be detected by visual sensors, the destination reward is set to 0 and the robot moves forward within the drivable space. For the second task, the robot is penalized if the linear speed is not the maximum legal one (13). Since legal requirements are also implemented, if the robot exceeds the maximum legal speed, it is penalized according to (10). \[r_{speed}=\left\{ \begin{array}{ccc}-\frac{v_{target}-v_{rob}}{v_{target}}&,&v _{rob}<v_{target}\\ 0&,&v_{rob}\geq v_{target}\end{array} \right.\] (13) In the previous reward function, the target linear velocity \(v_{target}\) is equal to \(v_{legal}\). Finally, the reward due to the task-oriented requirements is obtained by adding the single functions (14): \[r_{task}=r_{destination}+r_{speed}\] (14) (5) Final reward function The final reward function can be simply built by adding the single reward functions (15). Each reward function is normalized between \(-1\) and \(0\).