## 3 Enabling Technologies for Autonomous Navigation
### Deep Q-Networks (DQN)  
The reinforcement learning system aims to provide knowledge to a given agent through interactions in an environment. The qualities of this agent's actions are measured through the reward function. The state-action value function \(Q(s,a)\) quantifies the rewards obtained by this agent in relation to its states and actions, both in the short and in the long term. In this way, the agent learns \(\pi\) action policies that maximize the value function. The function \(Q(s,a)\) is a variation of the temporal learning algorithm proposed by [PERSON] and is given by:  
\[Q^{\pi}(s,a)=r+\gamma\ \max Q^{\pi}(s^{\prime},a^{\prime}), \tag{1}\]  
where \(r\) is the immediate obtained reward and \(\gamma\) a discount factor for future rewards; \(s,s^{\prime}\), \(a\), and \(a^{\prime}\) are the states and actions of the present and future, respectively. The DQN algorithm, a type of Reinforcement Learning (RL) technique, incorporates the parameters of a neural network into its algorithm, with synaptic weights (\(\theta\)) and bias. Normally, the network used is deep (more than three layers) and aims to estimate the optimal \(Q(s,a)\) function, displayed in Equation (1). The DQN does not need a tabular representation to store and map agent states and actions, like traditional Q-Learning. In an environment with many states and actions, for example, images captured by a camera, tabular representations are not feasible. The DQN state action mapping is provided by the neural network itself. In addition, unlike supervised learning, in DQN, the data are dynamic and updated at each step of the algorithm's integration. \[Q(s,a)\approx\hat{Q}(s,a,\theta) \tag{2}\]  
The DDQN algorithm, proposed by [26], is a variation of the traditional DQN. This algorithm helps to reduce stock overestimation problems that may arise as a result of the regression process. Thus, the DDQN uses one more neural network than the traditional DQN, called the \(Q_{target}\) network, which helps in the process of estimating future actions. The overestimation problem in the traditional DQN algorithm occurs because the algorithm always chooses the maximum value \(Q(s^{\prime},a^{\prime})\), as expressed in Equation (1). However, it does not always obtain the maximum value of \(Q(s^{\prime},a^{\prime})\), meaning that it obtains the best policy, since the DQN deals with estimates, and this maximization process contains uncertainties mainly at the beginning of the algorithm exploration. The consequence of overestimation is the development of low quality policy and instability in training. In DDQN, we have two independent neural networks: the \(Q^{\pi}\) and \(Q_{target}\) network. The modified TD Equation is described below:  
\[Q^{\pi}(s,a)=r+\gamma\ (Q_{target}(s^{\prime},\ argmax_{a^{\prime}}Q^{\pi}(s^{ \prime},a^{\prime})) \tag{3}\]  
As it can be seen from Equation (3), the function \(Q^{\pi}(s^{\prime},a^{\prime})\) is applied to a neural network responsible for evaluating the best action \(a\). The function \(Q_{target}\) is applied to another neural network to estimate the expected value using the best action estimated by \(Q^{\pi}(s^{\prime},a^{\prime})\). In the realm of reinforcement learning, the distinction between on-policy and off-policy methods is pivotal in the design of algorithms for training agents. Off-policy methods, such as the DDQN, offer significant advantages that align with our objectives of creating a robust and computationally efficient navigation system for autonomous robots. The DDQN, an extension of the standard DQN algorithm, inherits its predecessor's power but with an improved stability in learning. This stability stems from the DDQN'sability to decouple the selection and evaluation of the action, which mitigates the overestimation of Q-values that often plagues the original DQN. This characteristic is particularly beneficial in complex decision-making environments, where overestimation can lead to suboptimal policy development.