# Research on Dynamic Path Planning of Multi-AGVs Based on Reinforcement Learning
## 5 Experimental Results and Discussion
### Experiment 1: Experiments to Verify the Effectiveness of the K-L Algorithm  
This experiment is carried out to prove the effectiveness of the K-L algorithm according to the AGVs driving path and Q-value curve during the training process. The experiment uses a 10 \(\times\) 10 raster map, as shown in Figure 6. The starting point is located in the lower-left corner of the map, and the end point is in the upper-right corner of the map. The obstacles are randomly placed inside the map, as represented by black raster. The AGV uses the K-L algorithm to accomplish the path optimization task. If the AGV collides with an obstacle, it immediately returns to the original position to speed up the training. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \begin{tabular}{c} _Fixed_ \\ _Amealing_ \\ _Rate_ \\ \end{tabular} & \begin{tabular}{c} _Initialization_ \\ _Temperature_ \\ \end{tabular} & \begin{tabular}{c} _Learning_ \\ _Efficiency_ \\ \end{tabular} & \begin{tabular}{c} _Penalty_ \\ _Coefficient_ \\ \end{tabular} & \
\begin{tabular}{c} _Penalty_ \\ _Value of the_ \\ _Reward_ \\ \end{tabular} &
\begin{tabular}{c} _Default_ \\ _Reward_ \\ \end{tabular} \\ \hline _\(\gamma\)= 0.95_ & _T=200_ & _a0= 0.2_ & _\(\delta\)= 0.9_ & \(1\) & \(-\)0.5_ \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Parameters of the K-L algorithm. Figure 5: Flow chart of the GA-KL method. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \begin{tabular}{c} _Average_ \\ _Speed_ \\ \end{tabular} & \begin{tabular}{c} _Turning_ \\ _Speed_ \\ \end{tabular} & \begin{tabular}{c} _Penalty_ \\ _Factor_ \\ \end{tabular} & \begin{tabular}{c} _Number of_ \\ _Nodes_ \\ \end{tabular} & \begin{tabular}{c} _Maximum_ \\ _Number of_ \\ _AGVs_ \\ \end{tabular} & \
\begin{tabular}{c} _Time_ \\ _Bonus_ \\ _Factor_ \\ \end{tabular} &
\begin{tabular}{c} _Maximum_ \\ _Value of_ \\ _Node Busy_ \\ _Factor_ \\ \end{tabular} \\ \hline _v1= 0.80_ & _v2=0.35_ & _a= 2.1_ & _M= 15_ & \(5\) & _T\({}_{w}=3\)_ & _P\({}_{max}=20\)_ \\ _mls_ & _mls_ & _\(\rho\)=1.3_ & _M= 15_ & \(5\) & _T\({}_{w}=3\)_ & _P\({}_{max}=20\)_ \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Parameters of the GA-KL method. As shown in Figure 7, the green raster represents the path that the AGV travels during the training process. Figure 7a-d show the results of the 256 th, 644 th, 960 th, and 1445 th training iteration, respectively. As shown in Figure 7a, because the training process is in the early stage, the probability of exploration is high, so the AGV can easily collide with the obstacle. However, as the number of training sessions increases, the K-L algorithm accumulates more experience with the environment, and the AGV is eventually able to avoid obstacles, as shown in Figure 7b-d. Figure 8 shows the average Q-values of K-L for every 100 iterations. (a), (b), (c), and (d) in Figure 7 correspond to curves (a), (b), (c), and (d) in Figure 8, respectively. The path lengths of curves (a), (b), (c), and (d) are 10 m, 24 m, 20 m, and 19 m, respectively. Except for curve (a), as the training progresses, the path lengths of each curve gradually decrease, which proves that the K-L algorithm is converging to a better solution. As for curve (a), because the AGV collides with the obstacle and returns to the starting point, the path length of curve (a) remains constant at 10. A comparison of Figures 7 and 8 shows that when the AGV collides with an obstacle far away from the end point, the Q-value will continue to decrease until it is 0; when the AGV bypasses the obstacle and reaches the end point, the Q-value continues to increase and remains stable in later training stages. Figure 6: Raster map. The yellow raster is the starting point, and the green raster is the ending point. Figure 7: AGV roadmap. The red grid represents not reaching the end point, and the blue grid means reaching the end point successfully. (**a–d**) represent the trajectories at different numbers of iterations, respectively.