# Occupancy Reward-Driven Exploration with Deep Reinforcement Learning for Mobile Robot System
## 6 Simulation Results and Comparison
### Simulation Results  
The training DDPG agent and the exploration are carried out online. It means that the robot drives in one episode until time runs out, up 150 steps. Figure 9 shows the environments in which the robot tries to build the maps for two experiments. The first environment is a simple one without obstacles inside the room. The environment of Figure 9b is a more sophisticated version of the first one with obstacles. During the training, the robot drives in one of the environments of Figure 9. Step by step, as it moves during the mapping, it upgrades the occupancy map of Figure 10. It should be noted that the location coordinates of walls and obstacles are not used in the computation. The environments in Figure 9 can be treated as the simulated rooms, which can be easily substituted by real-world environments.  
Figure 10 demonstrates the exploration results of training the DDPG agent. Two results from each experiment are presented in (a) map coverage percentage and (b) total reward value. The results were captured during the training according to the greatest values.  
Here, it is important to explain the reason for presenting two map results for one experiment. The results of map (a) and map (b) are different because creating a reward function based on only one goal, mapping, does not return a positive result. Several robot behaviors were found to be inappropriate, such as collisions with obstacles, being stuck in one place without any motion, and turning to one side episode after episode. These occurred because the linear velocity, angular velocity, and sensing ranges should be considered in the reward function as they are in the proposed occupancy reward function. This is why the greatest result of the map coverage is not equal to the greatest result of the reward function.  
Figure 9: The environments of 20 \(\times\) 15 m size for the two experiments: (**a**) simple environment, (**b**) environment with obstacles.  
Nonetheless, a full map coverage was obtained. This proves that the DDPG agent is able to provide 99% of the exploration in the simple map (Figure 10a). The greatest reward value (\(G=90\)) returns a positive result of the exploration in Figure 10b.  
In Experiment 2, the training of the DDPG agent was carried out in the environment with obstacles (Figure 10c,d). Two results with the greatest values were taken for the map coverage and reward function categories. It can be seen that the results are worse compared to the results of Experiment 1. Only 86% of the map was explored. The reward function value is 67, which is less than 90.  
Based on our practice with the DDPG agent in the mobile-robot exploration, several conclusions can be drawn:  
* The agent can solve the mapping problem, especially in a simple environment.
* The reward function can be described as the single objective function. The navigation and exploration of new areas using the reward function of DDPG agent are insufficient for the exploration.
* The mapping performance deteriorates when the number of obstacles in the environment is increased.
* Increasing the training time did not improve the mapping results. As Figure 11 shows, the overfitting of the neural network occurs in the training after 500 episodes.
* The mapping is a real-time procedure, and the training of the DDPG agent works online as well. The two together used as one system can be considered a time-consuming process.  
Figure 10: The mapping results of the DDPG agent in the custom environments. (**a**) Map coverage: 99%, Initial positions: x = 6, y = 4. (**b**) Total reward (\(G\)): 90, Initial positions: x = 6, y = 4. The black line is the robot trajectory. (**c**) Map coverage: 86%, Initial positions: x = 13, y = 5. (**d**) Total reward (\(G\)): 67, Initial positions: x = 13, y = 5. The black line is the robot trajectory.  
The experimental results of the proposed occupancy reward-driven exploration using the DDPG agent are recorded and demonstrated in video [63].