# Self-Learning Robot Autonomous Navigation with Deep Reinforcement Learning Techniques
## 2 Materials and Methods
### Sense-Explore-Act-Learn Paradigm  
The methods based on reinforcement learning techniques attempt to bring a solution to the previously mentioned drawbacks. The drawbacks and the solution proposed by the reinforcement learning approach are summarized in Table 1.  
Due to the advantages summarized in the previous table, reinforcement learning techniques have been the subject of extensive study within the autonomous driving domain. In [21], Q-learning was used to find the shortest path from the starting pose to the destination pose. Reinforcement learning techniques with continuous actions and states were introduced in [22], with very promising results. A deep deterministic policy gradient (DDPG) was introduced in [23] to learn from a real scenario how to drive and keep the vehicle inside the road boundaries by providing just a single camera image of the road as the input. However, this work only implemented partial safety requirements (driving within the drivable space without obstacle avoidance) and did not consider any comfort or legal requirements. Additionally, the perception, motion planning, and motion control modules were embedded in the deep neural networks of the deep reinforcement learning algorithm. In [24], the autonomous driving algorithm relied not only on input images of the cameras as in [23] but also on post-processed data as velocities or relative distances. In [25], comfort requirements were included for lane change maneuvers as a simple reward function. In contrast to [23], it separated the perception module from the reinforcement learning task, and the agent consumed already post-processed signals from the perception module, such as robot velocities or relative distances, instead of images. The algorithm presented in this work keeps the perception, motion planning, and motion control modules separated from each other. Images are not directly provided to the agent. Instead, all the relevant post-processed information from the perception module is provided to the agent so that it can learn the best possible trajectory based only on variables that have an important impact on motion planning. Regarding the definition of reward functions, [26] included positively and negatively weighted rewards. The methodology proposed only included negatively weighted rewards to exclude policies that end up in endless loops of collecting permanently positive rewards instead of reaching the destination. In terms of requirements, ref. [26] included vehicle speed (legal requirements), collisions (safety requirements), and travel distance towards the destination (task-oriented requirements), but comfort requirements \begin{table}
\begin{tabular}{l l} \hline \hline \multicolumn{1}{c}{**State-of-the-Art Drawbacks**} & \multicolumn{1}{c}{**Reinforcement Learning Strengths**} \\ \hline Cost function design & Cost function is automatically learned \\ \hline Performance under dynamic environments & Agent learns from dynamic environments \\ \hline Uncertainties of embedded dynamic models & Agent free from embedded dynamic models \\ \hline High computational cost & Low computational cost avoiding iterative motion planning modules \\ \hline Integration of comfort requirements & Easy integration with simple reward functions \\ \hline Databases must be available and labeled & Data are generated on-the-go and automatically labeled based on a reward value \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Strengths of reinforcement learning-based algorithms for autonomous driving navigation. were not included. In [27], comfort, legal, and task-oriented requirements (lane keeping) were included, but safety requirements were not, which are needed for, for example, overtaking maneuvers in dynamic scenarios. In [28], all requirements were included. However, the comfort requirements were not expressed in terms of acceleration and jerk but in terms of limiting the steering wheel angle (thus only penalizing lateral acceleration but excluding lateral jerk and longitudinal acceleration and jerk).