# Autonomous Intersection Management by Using Reinforcement Learning
## 4 Results and Discussion
### Experimental Results
#### 4.3.1 Evaluation of Training Results of DRLAIM
The total queue length is computed by adding the queue lengths of all incoming lanes. We will calculate the total queue length once every 0.1 s.  
**Average Waiting Time (AWT)**: The time that a vehicle is waiting in a queue is called waiting time. We will calculate the waiting time of all vehicles and record the number of vehicles waiting over the whole simulation. The summation of the waiting time of vehicles divided by the number of waiting vehicles is called the average waiting time (AWT). Tables 4 and 5 show the average performance between first 10 epochs and the last 10 epochs of the 8 reward functions. We can see that the average performance of the last 10 training epochs among the 8 reward functions outperforms that of first 10 training epochs which means our DRLAIM model did learn a good policy after training. Table 6 shows the average performance over another 10 simulations for the 8 reward functions. We can see that the average performance of reward function 7 outperforms other reward functions. According to this simulation result, we will use reward function 7 to do a further evaluation by using different training epochs and different action time steps. The performance of the proposed DRLAIM while training is shown in Figures 8-10. Figures 8-10 show the value of the total reward obtained after training the model by using 25, 50, and 100 and 200 training epochs and 0.2, 0.5, and 1 s action time steps, respectively. The X-axis represents the epochs, and the Y-axis represents the total reward in Figure 8. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Reward Function** & **AQL (Vehicles)** & **AWT (s)** & **Throughput (Vehicles)** \\ \hline
1 & 51.46 & 34.67 & 152.5 \\ \hline
2 & 56.27 & 37.61 & 176.7 \\ \hline
3 & 60.31 & 38.68 & 141.5 \\ \hline
4 & 44.02 & 33.70 & 202.4 \\ \hline
5 & 58.59 & 36.83 & 159.7 \\ \hline
6 & 53.24 & 34.18 & 187.8 \\ \hline
7 & 42.83 & 31.71 & 211.3 \\ \hline
8 & 58.39 & 36.95 & 148.9 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Average performance difference among 8 reward functions. Figure 8: Total reward while training by using action time step 0.2 s with epochs (**a**) 25, (**b**) 50, (**c**) 100, and (**d**) 200. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Performance Indicator** & \multicolumn{2}{c}{**Throughput (Vehicles)**} \\ \hline
**Compared with** & **First 10 Epochs** & **Last 10 Epochs** \\ \hline Reward 1 & 121.4 & 161.3 \\ \hline Reward 2 & 112.7 & 177.1 \\ \hline Reward 3 & 110.9 & 161.9 \\ \hline Reward 4 & 119.0 & 183.7 \\ \hline Reward 5 & 118.8 & 167.1 \\ \hline Reward 6 & 109.9 & 185.4 \\ \hline Reward 7 & 116.8 & 214.1 \\ \hline Reward 8 & 109.8 & 142.5 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Comparison of training results among 8 reward functions for throughput. The total reward generated by DRLAIM is trained with 25 epochs, as shown in Figure 8a, and it gives the total reward at 25 epochs of \(-34,357.28\). Then we have increased the epoch as 50, which is depicted in Figure 8b, which gives the total reward as \(-34,397.38\), which has higher negative values at 25 epochs; thus we have again increased the epochs to 100 and recorded the total reward, which is \(-34,303.90\). From Figure 8c, we can understand that during the training of the DRLAIM model at 100 epochs, we are getting lower negative reward values than at 25, 50, and 200 epochs. We can conclude that even if the number of epochs increases, the model will not give good performances. We need to decide the optimal epochs for the training model. We can see that except for the model trained with 100 epochs and time step 0.2 s, the total reward increases gradually as the number of epochs increases. Figure 9 shows a total reward value of 25, 50, 100, and 200 epochs during time step value 0.5 From Figure 9, we can understand that our model performs well in the 50-epoch range.