## 5 Simulation Experiments and Analysis
### Path Planning Scenarios
#### 5.2.1 Simulation Scenarios That Contain Only Convex Obstacles  
The structure of the scenario uses the Tkinter module in Python, and the position of each agent is represented by the position coordinates of the two points of the upper left corner and the lower right corner of the square, and the side length of a grid is set to 20. The simple scenario is equipped with eight regular-shaped obstacles, 38 \(\times\) 38 environment size. The top yellow square, blue square, and red square in the environment diagram represent UGV1, UGV2, and UGV3, respectively, and the square at the bottom of the environment map is the target point of the UGV corresponding to the color. The coordinates of the start point and target point of each UGV are shown in Table 2.  
In the simple scenario, the path planning simulation results of the SARSA algorithm, Q-learning algorithm, speedy Q-learning algorithm, and the OWS Q-learning algorithm are shown in Figure 7.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Numeric Value** \\ \hline Learning rate \(\alpha\) & 0.02 \\ Learning rate \(\alpha_{1}\) & \(1/(k+10)\) \\ Explore factors \(\varepsilon\) & 0.9 \\ Discount factor \(\gamma\) & 0.9 \\ Reward function \(r_{1}\) & 120 \\ Reward function \(r_{2}\) & \(-120\) \\ Reward function \(r_{3}\) & \(-100\) \\ Reward function \(r_{4}\) & \(-3\) \\ \(m\) & 400 \\ \(n\) & 3 \\ \(l\) & 10 \\ Max episode & 1800 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Experimental parameter settings.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Number** & **The Coordinates of the Starting Point** & **The Coordinates of the Target Point** \\ \hline
1 & (120, 40, 140, 60) & (120, 680, 140, 700) \\
2 & (360, 40, 380, 60) & (360, 680, 380, 700) \\
3 & (620, 40, 640, 60) & (620, 680, 640, 700) \\ \hline \hline \end{tabular}
\end{table}
Table 2: The starting and target points of the UGV obstacle avoidance path.  
From the path planning results in the simple scenario, the SARSA algorithm plans the route from the starting point to the target point for UGV2 and UGV3, and the route is tortuous. The Q-learning algorithm mapped out a route from the start point to the target point for the three UGVs, but the route was complicated. Compared with the Q-learning algorithm, the SARSA algorithm is more conservative in action selection, so under the same training and learning round, the SARSA algorithm does not complete the task of planning the path for all three UGVs. The speedy Q-learning algorithm and the proposed OWS Q-learning algorithm both plan the route from the starting point to the target point for the three UGVs. The specific experimental results are shown in Table 3.  
The experimental results in Table 3 show that the calculation time of the OWS Q-learning algorithm is the shortest in the same experimental scenario. In general, compared with the other three comparison algorithms, the OWS Q-learning algorithm has the shortest calculation time, the most complete planned path, and the shortest planned path compared with the other three comparison algorithms.  
Figure 7: Path planning that contains only convex obstacles. (**a**) SARSA algorithm path planning. (**b**) Q-learning algorithm path planning. (**c**) Speedy Q-learning algorithm path planning. (**d**) OWS Q-learning algorithm path planning.