# A Deep Reinforcement Learning Approach for Active SLAM
## 4 Discussion of Results
### Deep Reinforcement Learning  
Once the suitability of the approach has been conceptually proved, it has been extended to include the benefits of DL. First of all, the three agents (DQN, DDQN and D3 QN) have been trained in the simplest environment with an extrinsic reward, in order to compare their performance, study their generalization ability and prove that exploration is not achievable with such formulation. Figure 4 contains their training curves (i.e., the cumulative reward after each episode of the training) after roughly 30 h and 500 episodes (all experiments have been performed on a laptop with Intel Core i7-7500U CPU, Nvidia Quadro M520 (2 GB) GPU and a 32 GB RAM). Light-colored curves correspond to raw data while bold ones do to outlier-filtered moving averages. DQN agent, depicted in blue, demonstrates permanent instabilities due to the memory saturation with useless experiences, resulting in continuous cycles of forgetting and learning similar policies. DDQN agent (red) mitigates them, although converged maximum values are slightly lower (i.e., policies are less optimal and thus trajectories become rougher). Finally, D3 QN curve (green) has a significantly more stable behavior attributable to the PER, and also higher maximum converged values. Note that, in this case, convergence is much faster because of the non-random initialization of the network weights. Table 3 shows the performance of each agent in the three scenarios during testing (generalization), in terms of success ratio (SR, i.e., number of times over all trials that they sense more than 95% of the map at least once, where 95% have been used instead of 100% due constraints of the laser sensor), mean number of steps per episode and mean reward per episode. In addition, results have been averaged over five experiments with different simulation random seeds, and their standard deviation is presented in parenthesis. D3 QN results are the most remarkable in the first environment, as it  
Figure 4: Cumulative reward during training in environment 1 for DQN (blue), DDQN (red) and D3 QN (green) agents. Light-colored curves correspond to raw data while bold ones do to outlier-filtered moving averages. Figure 3: Logarithmic occurrence of entropy along episodes in the second scenario. Results are shown for traditional (**left**) and _uncertainty-based_ (**right**) RL’s reward functions. Darker colors denote lower occurrence. outperforms the reward that a human would obtain by manually controlling the robot (\(\approx\)350). In the second environment both DDQN and D3 QN show a good behavior. Despite DDQN have higher SR (and mean steps, thus), the higher mean reward obtained by D3 QN proves the generation of more optimal trajectories: smoother movements and less spins. The third environment presents quite a challenge not only for being unknown but also because of its complex topology. None of the agents trained with \(\mathcal{R}_{\mathcal{F}}\) is capable of going across the whole map, leaving some areas such as the top right corner and the middle dead-end unvisited (see Figure 1f). Besides that, D3 QN navigation performance is exceedingly superior, with mean rewards several times higher than the ones obtained by DQN and DDQN. The three trained agents performed well during generalization in the second environment, due to its similarity to the training one. After the training, the agents have learned to decide when to go forward and when to steer when an obstacle is close enough. As the complexity of the agent's architecture increases, the learned skills complexity grows too. For instance, it can be noticed how the D3 QN agent learned the exact moment to steer so as to maximize rewards (i.e., minimize the traveled distance). In this sense, the second scenario keep this topology and thus the agent is capable of transferring the learned skills. The third scenario, however, is more challenging. Several dead-end corridors appear and also a corridor with two ways.