# A Deep Reinforcement Learning Approach for Active SLAM
## 3 Experimental Setup  
### Environment Setup  
In order to evaluate the proposed approach, several experiments have been conducted in Gazebo robotics simulator ([[http://gazebosim.org/](http://gazebosim.org/)]([http://gazebosim.org/](http://gazebosim.org/))). A Turtlebot robot has been used, consisting of with two differential wheels and a modelled laser sensor that generates \(n\) rays equally distributed in the 180\({}^{\circ}\) front field of view of the robot, each of them with a minimum range of 0.1 m, a maximum range of 10 m, two-digit decimal precision and a Gaussian noise model with zero mean and small variance. Three different environments (Figure 1), adapted from [37], have been used during training and testing stages. The first one, where the agent is trained, consists of a simple maze with 90\({}^{\circ}\) turnarounds. Second and third environments have a more complex topology, including consecutive turnarounds and dead-ends, and have been used only during testing (i.e., agents do not improve their policy anymore, see Figure 2). A training/testing stage consists of a certain number of episodes, in which the robot moves until either a collision occurs (i.e., gets closer to any obstacle than a defined threshold of 0.2 m) or 500 steps (i.e., decisions) are consumed. A ROS framework has been used to connect the simulation environments with the SLAM algorithm and the decision making module, by using GymGazebo ([[https://github.com/erlerobot/gym-gazebo](https://github.com/erlerobot/gym-gazebo)]([https://github.com/erlerobot/gym-gazebo](https://github.com/erlerobot/gym-gazebo))) library, based on OpenAI Gym ([[https://gym.openai.com/](https://gym.openai.com/)]([https://gym.openai.com/](https://gym.openai.com/))).  
Every step, the DNN receives the \(n\)-dimensional sensed vector and estimates the Q-values for each possible action, namely going forward, turning right and turning left:  
\[a_{1} =\{v=0.3,\ \ \omega=0\}\] \[a_{2} =\{v=0.05,\omega=0.3\}\] \[a_{3} =\{v=0.05,\omega=-0.3\}\]  
where \(v\) is the linear velocity in m/s and \(\omega\) is the angular velocity in rad/s; and \(v\
eq 0\) in \(a_{2}\) and \(a_{3}\) to avoid continuous turnings on the same spot. By using a fully extrinsic approach, after each decision  
Figure 1: Gazebo simulation environments (top) and their corresponding maps generated by the SLAM algorithm (bottom).  
Figure 2: Training (**left**) and testing (**right**) processes.  
making, the environment would reward the agent with a large negative value if a collision occurred, or with a small positive value otherwise:  
\[\mathcal{R_{F}}=\left\{\begin{array}{cl}-&100&\text{if collision}\\ &1&\text{if }\omega=0\\ -&0.05&\text{if }\omega\
eq 0\end{array}\right. \tag{11}\]  
Note that the reward values have been designed empirically, and the penalty to nonzero angular velocities prevents continuous spins and uneven trajectories.  
The proposed _uncertainty-based_ approach requires, following Equation (8), its augmentation as follows,  
\[\mathcal{R}_{aug}=\left\{\begin{array}{cl}-&100\\ &1&+\tanh\left(\dfrac{\eta}{f(\Sigma)}\right)\\ &-&0.05&+\tanh\left(\dfrac{\eta}{f(\Sigma)}\right)\\ \end{array}\right.\text{if }\omega\
eq 0 \tag{12}\]  
where \(\eta\) is a task-dependent scale factor and \(f(\Sigma)\) is the D-optimality criterion. Since this criterion -likewise any other of the presented criteria- is right-unbounded, the \(\tanh(\cdot)\) function has been used to bound it in the interval \([0,1]\) and therefore to maintain a relationship between the free-collision trajectory and uncertainty terms.