# A Deep Reinforcement Learning Approach for Active SLAM
## 1 Introduction
Further discussion about the monotonocity of optimality criteria under Lie algebra representations can be found in Appendix A, where the proof of their monotonocity under differential uncertainty representations is presented. Aside from traditional methods, such as particle filters [10] or model predictive control [11] where frontier-based or random tree strategies are generally used, the problem of selecting the most informative action in robotic exploration has been lately addressed with Deep Neural Networks (DNN), often equating to Convolutional Neural Networks (CNN). Notably, Deep Reinforcement Learning (DRL) is suitable for such problems in which the agent must learn by directly interacting with the environment under partial observability, and where its _a priori_ knowledge is nonexistent. Thus, some DRL algorithms and strategies originally developed for other decision making problems (e.g., video-games) have already been successfully applied to robotic navigation and exploration, such as Deep Q-Networks (DQN) [12] and their double [13], dueling [14] and Rainbow [15] successors, the Asynchronous Advantage Actor-Critic (A3C) architecture [16], the Soft Actor-Critic (SAC) [17] or the Prioritized Experience Replay (PER) buffer [18]. A first example of those methods appears in [19], where [PERSON] and [PERSON] used a large CNN to extract the feature maps of a front-view camera that would be later fed into a 2-layer DQN capable of selecting the best next action to execute. In this case, experiments were carried out in a complex simulator,but convergence of training was only achieved for topologically simple scenarios (e.g., a straight corridor). Moreover, the same simple scenarios were used in both training and testing stages, being the agent's generalization ability unclear. Work in [20] was among the firsts showing that navigation via DRL was also achievable using the A3C algorithm, notably improving speed convergence with respect to DQN. However, the acquired knowledge was never tested in environments different from the training one, hence the environment was again known in all cases. A critic generalization experiment, though, was carried out in [21], showing that trained agents do not perform as well in previously unseen maps, and could even demonstrate behaviors comparable to random agents in certain scenarios. [22] shows a similar approach in which obstacle avoidance problem is tackled using the D3 QN architecture and FastSLAM backend to map the environment and locate the robot on it. The aforementioned works used simple extrinsic rewards to encourage obstacle avoidance. However, once navigation was proved to be achievable, motivation and curiosity capabilities [23] were added to the agents in order to expand the problem to robotic exploration. While some approaches, frequently called _curiosity-driven_, encourage the agent to visit user-defined or novel states [24] or to maximize the coverage of a known map [25], others motivate those actions that minimize the environment's uncertainty (i.e., the agent's knowledge of the environment is maximized). The latter, that will be referred as _uncertainty-based_, have been commonly addressed by encouraging the visit of those states which are more difficult to predict [26; 27], although they have been recently tackled on the basis of true uncertainty metrics. In [28; 29] actions are selected so as to maximize the entropy reduction (information gain or Kullback-Leibler divergence) of a 2-dimensional occupancy grid map; either by learning in a supervised fashion with labelled cells, or by introducing a metric of it in the DRL's reward function, respectively. Another approach to solve the active localization problem that also acts directly on the reward function design was proposed in [30; 31], where the entropy reduction was achieved by including on it the maximum likelihood (ML) of being in any state (belief accuracy). Recently, [PERSON] et al. [32] trained both DQN and A2C agents using the underlying SLAM pose-graph through Graph Neural Networks (GNN).