# Autonomous Intersection Management by Using Reinforcement Learning
## 4 Results and Discussion
### Experimental Results
#### 4.3.1 Evaluation of Training Results of DRLAIM  
In this experiment, we first evaluate the performance of training results by using different combinations of reward functions as shown in Table 2 where \(r_{q}\) is queue size and \(r^{q^{i}_{t+1}}\) is the queue size of lane \(i\) (the same lane as \(r^{q^{i}_{t}}_{\min}\)) at time \(t+1\). The usage of elements \(r_{q}\) and \(r_{th}\) is to maximize the queue size difference and the throughput to enhance the efficiency of DRLAIM. The usage of element \(\max_{i}w^{i}_{t+1}\) is to avoid a situation where vehicles are waiting for a long time and still cannot go through the intersection to enhance the fairness of DRLAIM. We use these three elements to combine seven different reward functions, as shown in Table 2. The 8 th reward function lets the agent learn a policy to select an action to let vehicles in the lane with minimum queue size to pass through the intersection. We train our DRLAIM model by using different combinations of reward functions for 50 training epochs, and the action time step is taken as 0.5 s (time duration in which the agent selects an action). We then evaluate the training results of the reward function with the best performance by using a different number of training epochs and action time steps. Table 3 specifies the parameters used in this experiment. In each training epoch, we simulate for 200 s by using a high inflow rate (100 vehicle arrivals per minute). The capacity of memory which stores the experience sequence set \((s_{t},a_{t},r_{t+1},s_{t+1})\) is a set of 100,000 where \(s_{t}\) is the state of traffic environment at time \(t\), \(a_{t}\) is the action selected by agent at time \(t\), \(r_{t+1}\) is the reward received at time \(t\) and \(s_{t+1}\) is the state of traffic environment at time \(t+1\). The batch size used in training DRLAIM is 64, and the value of the discount factor used in calculating the action value function is 0.99. \begin{table}
\begin{tabular}{c c} \hline \hline
**Numbering of Reward Function** & **Formula of Reward Function** \\ \hline
1 & \(r=r_{q}\) \\ \hline
2 & \(r=-\max_{i}w^{i}_{t+1}\) \\ \hline
3 & \(r=r_{th}\) \\ \hline
4 & \(r=r_{q}-\max_{i}w^{i}_{t+1}\) \\ \hline
5 & \(r=r_{q}+r_{th}\) \\ \hline
6 & \(r=-\max_{i}w^{i}_{t+1}+r_{th}\) \\ \hline
7 & \(r=r_{q}-\max_{i}w^{i}_{t+1}+r_{th}\) \\ \hline
8 & if \(r^{q^{i}_{t}}_{\min}>r^{q^{i}_{t+1}}\): \(r=10\), else: \(r=-10\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Combination of reward functions used for training. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Simulation length & 200 s \\ \hline Inflow rate & 100 vel/min \\ \hline Memory capacity & 100,000 set \\ \hline Batch size & 64 \\ \hline Discount factor & 0.99 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Parameters used in training experiment. We calculate the value of \(\epsilon\), shown in Equation (9), which is used to determine the probability for an agent to choose an action:  
\[\epsilon=\epsilon_{\min}+(\epsilon_{\max}-\epsilon_{\min})\times e^{-0.02\times steps} \tag{9}\]  
where  
* \(\epsilon_{\min}\) is the minimum value of \(\epsilon\) which is taken as 0.01 in this work. * \(\epsilon_{\min}\) is the maximum value of \(\epsilon\) which is taken as 1 in this work. * _steps_ is the the current training epoch. As the value of _steps_ increases, the value of \(\epsilon\) will decrease from 1 to 0.01. The reason we use Equation (9) to decrease the value of \(\epsilon\) gradually is as follows. At the beginning of training, the agent will randomly choose an action to explore the environment because the DRLAIM has not yet learned anything; thus, we cannot use it to predict the best action. As the training proceeds, we can gradually increase the ratio to use the trained DRLAIM to predict actions. We now introduce two performance indicators to evaluate the performance of our model. **Average Queue Length (AQL)**: If the velocity of a vehicle in an incoming lane is smaller than 0.1 m/s, then the vehicle is considered to be in a queue.