# Self-Learning Robot Autonomous Navigation with Deep Reinforcement Learning Techniques
## 2 Materials and Methods
### Sense-Explore-Act-Learn Paradigm
In [28], a mix of sparse and continuous reward functions was used. The methodology proposed only used continuous reward functions to better guide the reinforcement learning algorithm to find the most valuable rewards. The methodology proposed is based on the deep deterministic policy gradient (DDPG) algorithm. The DDPG method, like the Q-learning method, is a model-free algorithm. That is, it does not use any embedded model of the robot inside the agent logic. Instead, it learns the optimal policy directly from the environment. The DDPG method is intended for environments with continuous states and actions. This is possible by approximating the actor model (which computes the action \(a_{t}\)) with deep neural network \(\mu\) of weights \(\theta\) as a function of state \(s\) at time \(t\) (1). \[a_{t}=\mu(s_{t}|\theta)=\mu_{\theta}(s_{t}) \tag{1}\]  
The DDPG algorithm is an actor-critic method and, therefore, the critic function is also approximated with another deep neural network \(Q\) of weight \(\phi\) as a function of state \(s\) and the action a at time \(t\) (2). \[Q_{\phi}(s_{t},a_{t})=Q_{\phi}(s_{t},\mu_{\theta}(s_{t})) \tag{2}\]  
Actor-critic methods combine the benefits of policy-based and value-based methods. They benefit from the good convergence properties of policy-based methods and they also benefit from the sample efficiency of value-based methods, which tend to find an optimal solution faster. The deterministic policy gradient theorem (3) updates the weight \(\theta\) of the actor model \(\mu_{\theta}\) in the direction of the maximum value of the critic model \(Q_{\phi}\):  
\[\
abla_{\theta}J(\mu_{\theta})=\mathbb{E}_{s\sim\mathcal{D}}\Big{[}\left. \
abla_{a}Q_{\phi}(s,a)\right|_{a=\mu_{\theta}(s)}\
abla_{\theta}\mu_{\theta }(s)\Big{]} \tag{3}\]  
On the other hand, the Bellman equation is used to iteratively update the weight \(\phi\) of the critic network \(Q_{\phi}\) (4) and eventually learn the optimal critic and actor networks. \[Q_{\phi}(s_{t},a_{t})=\mathbb{E}_{s\sim\mathcal{D}}\Big{[}r(s_{t},a_{t})+ \gamma Q_{\phi}(s_{t+1},\mu_{\theta}(s_{t+1}))\Big{]} \tag{4}\]  
Additionally, this actor-critic method uses the replay buffer and target networks to bring more stability to the learning process. Random observations are selected from the replay buffer to avoid training the neural networks with consecutive and correlated observations. The weights \(\phi^{\prime}\) and \(\theta^{\prime}\) of the target networks \({u^{\prime}}_{\theta^{\prime}}(s)\) and \({Q^{\prime}}_{\phi^{\prime}}(s,a)\) are updated after the weights of the actor and critic networks with a soft factor \(\tau\) (5). This lag brings stability to the learning. \[ \begin{split}\theta^{\prime}&=\tau\theta+(1-\tau) \theta^{\prime}\\ \phi^{\prime}&=\tau\phi+(1-\tau)\phi^{\prime}\end{split}  \tag{5}\]  
The neural network architecture of actor and critic models considers several hidden layers (Figure 1). The variables in brackets denote the number of neurons for that layer. Finally, the exploration is essential to find the optimal policy. For that purpose, a white noise signal of zero mean value is added to the actor-network output (6). The amplitude of the exploration is controlled with the variance \(\sigma\) of the normal distribution \(\mathcal{N}\). \[a_{t}=\mu_{\theta}(s_{t})+\mathcal{N}(0,\sigma) \tag{6}\]  
As the learning process progresses and the actor and critic networks learn the optimal values, less exploration is needed. Therefore, a decay of the exploration was programmed. This decay also helped to stabilize the learning process.