## 3 Reinforcement Learning-Based Mobile Robot Path Optimization
### Results  
First, we tested two basic reinforcement learning algorithms: the Q-Learning algorithm and the Dyna-Q algorithm. The experiment was performed five times for each algorithm, and there may be deviations in the results for each experiment, so the experiment was performed under the same condition and environment. The following points were verified and compared using the experimental results. The optimization efficiency of each algorithm was checked based on the length of the final path, and the path optimization performance was confirmed by the path search time. We were also able to identify learning patterns by the number of learning steps per episode. The path search time of the Q-Learning algorithm was \(19.18\) min for the first test, \(18.17\) min for the second test, \(24.28\) min for the third test, \(33.92\) min for the fourth test, and \(19.91\) min for the fifth test. It took about \(23.09\) min on average. The path search time of the Dyna-Q algorithm was \(90.35\) min for the first test, \(108.04\) min for the second test, \(93.01\) min for the third test, \(100.79\) min for the fourth test, and \(81.99\) min for the fifth test. The overall average took about \(94.83\) min. The final path length of the Q-Learning algorithm was \(65\) steps for the first test, \(34\) steps for the second test, \(60\) steps for the third test, \(91\) steps for the fourth test, \(49\) steps for the fifth test, and the average was \(59.8\) steps. The final path length of the Dyna-Q algorithm was \(38\) steps for the first test, \(38\) steps for the second test, \(38\) steps for the third test, \(34\) steps for the fourth test, \(38\) steps for the fifth test, and the average was \(37.2\) steps. The Q-Learning algorithm was faster than Dyna-Q in terms of path search time, and Dyna-Q selected shorter and simpler final paths than those selected by Q-Learning. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameter** & **Q-Learning** & **Dyna-Q** \\ \hline Learning Rate (\(\alpha\)) & 0.01 & 0.01 \\ Discount Rate (\(\gamma\)) & 0.9 & 0.9 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Parameter values for experiment. Figure 10: Selected area for dynamic reward adjustment based on action. Figures 11 and 12 show the learning pattern through the number of training steps per episode. Q-Learning found the path by repeating many steps for 5000 episodes. Dyna-Q stabilized after finding the optimal path between 1500 and 3000 episodes. In both algorithms, the maximum number of steps rarely exceeded 800 steps. All tests ran 5000 episodes for each test. In the case of Q-Learning algorithm, the second test exceeded 800 steps 5 times (0.1%) and fourth test exceeded 800 steps 5 times (0.1%). So, Q-Learning algorithm test exceeded 800 steps at a total level of 0.04% in the five tests. In the case of Dyna-Q algorithm, the first test exceeded 800 steps once (0.02%), third test exceeded 800 steps 7 times (0.14%), fifth test exceeded 800 steps 3 times (0.06%). So, Dyna-Q algorithm test exceeded 800 steps at a total level of 0.044 % in the five tests. To compare the path search time and the learning pattern result, we compared the experiment (e) of the two algorithms. The path search time was 19.9 s for Q-Learning algorithm and 81.98 s for Dyna-Q. Looking at the learning patterns in Figures 11 and 12, the Q-Learning algorithm learned by executing 200 steps to 800 steps per each episode until the end of 5000 episodes, and the Dyna-Q algorithm learned by executing 200 steps to 800 steps per each episode until the middle of the 2000 episode. However, the time required for the Dyna-Q algorithm is more than 75% slower than the Q-Learning algorithm. In this perspective, it has been confirmed that Dyna-Q takes longer to learn in a single episode than Q-Learning. Figure 11: Path optimization learning progress status of Q-Learning: (**a**) 1 st test; (**b**) 2 nd test; (**c**) 3 rd test; (**d**) 4 th test; (**e**) 5 th test.