# Autonomous Navigation Framework for Intelligent Robots Based on a Semantic Environment Modeling
## 4 Experiments
### Experimental Results
#### 4.3.2 Action Planner  
The policy training described in Section 3.3.5 was performed using the simulation shown in Figure 8 generated by the mental simulation building algorithm. We used an experience database with a maximum size of 1 million tuples and learned on batches of 512. The learning rate used was \(10^{-6}\), and the critic network decay value was 0.95. Finally, the rewards were chosen as follows: \(r_{completion}=1000\), \(r_{time}=1\), \(r_{collision}=500\), and \(r_{closer}\) were rewarded sparsely when the robot travelled 25%, 50% and 75% of the path, giving a reward proportional to \(r_{completion}\). We trained the robot for 30,000 episodes (approximately 1.3 million steps). The first 5000 episodes used a purely random policy in order to generate a large variety of experiences. The rewards obtained on each episode can be seen in Figure 14. It can be seen that until episode 15,000, the robot was stagnated, then learned a better policy and started exploiting these actions, which lead to higher rewards. After 22,000 steps, the leaning plateaued again.  
To demonstrate the advantage of our hybrid approach when compared to pure reinforcement learning navigation methods, we performed the experiment shown in Figure 15. The robot started on the red arrow, and its goal was to navigate until the red circle by avoiding collision with any obstacle within 10 min. The simulated environment was generated by the mental simulation building algorithm. It was based on a narrow corridor sequence inside a university building. The total path size was 65.6 m. We performed two different experiments ten times each. The first one was using only the policy learned to navigate the robot directly to the goal. The second one used the PRM algorithm to generate a waypoint sequence as the one displayed in Figure 15b, which would be used as goals for the navigation policy. The results of this experiment are summarized in Table 8. It can be seen that the hybrid approach arrived at the goal 40% faster than the navigation done purely by using the learned policy. When the goal was too far away, the robot kept turning itself to the goal point,  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Place** & **Number** & **Graph Build** & **Path Search** & **Path** & **Success** \\
**Name** & **of Samples** & **Time (ms)** & **Time (ms)** & **Length (m)** & **Rate** \\ \hline floor1 & 309 & 2489.4 & 11.24 & 152.63 & 92\% \\ corridor1 & 231 & 1757.75 & 8.77 & 114.36 & 92\% \\ corridor2 & 135 & 1128.89 & 7.09 & 39.42 & 100\% \\ corridor3 & 79 & 375.62 & 4.39 & 50.92 & 92\% \\ \hline \hline \end{tabular}
\end{table}
Table 7: Optimized the number of samples.  
Figure 14: Reward obtained by the robot on each training episode. The line in green represents a moving average using a 100 episodes window. The line in red shows the cumulative average until the current episode.  
going in the direction of a wall, then turning back to avoid a collision. This behavior slowed down the navigation and was less predictable than the path followed by the PRM-RL approach. Both methods had two failures. The RL method timed out after going back and forth in the first corridor. This failure happened depending on how close the robot was to the trashcan when approaching the corner. If the robot was too close, the laser scanner would not be able to see the second corridor's entrance, believe it was a dead-end, and turn back. In the case of the PRM-RL approach, the failures were caused because the PRM algorithm created a sample too close to trashcan, which led to a collision. We believe this could be solved by using a more robust sampling algorithm than the one currently used.  
The experimental results show that our hybrid approach added an extra layer of robustness to autonomous navigation based on reinforcement learning policies. It was able to perform a navigation task 40% faster than the baseline while maintaining the success rate.