# Autonomous Intersection Management by Using Reinforcement Learning
## 3 Methodology
### Deep Reinforcement Learning for Intersection Control
#### 3.4.4 Learning  
The definition of action-value function \(Q(s_{t},a_{t})\) is  
\[Q(s_{t},a_{t})=r_{t+1}+\gamma\max_{a\in A}Q(s_{t+1},a), \tag{6}\]where the \(r_{t+1}\) is the received reward, where \(0\leq\gamma\leq 1\). Then we introduce the temporal difference (TD). The concept of TD is the core of Q-learning, which uses the difference of action-value \(Q\) in learning temporally. The formula of TD is introduced below:  
\[TD(s_{t},a_{t})=r_{t+1}+\gamma\max_{a\in A}Q(s_{t+1},a)-Q_{t-1}(s_{t},a_{t}). \tag{7}\]  
The steps of the Q-Learning based intersection scheduling algorithm are given in the Algorithm 1. The new Q-value \(Q(s_{t},a_{t})\) equals the old Q-value \(Q_{t-1}(s_{t},a_{t})\) that has been calculated before and is stored in memory plus the temporal difference that is calculated by Equation (7) times \(\alpha\), where \(\alpha\) is the learning rate and \(0\leq\alpha\leq 1\). The learning rate determines the importance of new estimated information, i.e., temporal difference. ```
Input:\(N\): Number of simulations; \(sim\_len\): The length of simulation time step; \(\epsilon\): A user-defined value that \(0<\epsilon<1\); Output:\(f\): A neural network that approximate the action-value function; Variable:\(a_{t}\): An action at time \(t\); \(s_{t}\): A state at time \(t\); \(r_{t+1}\): A reward received after executing \(a_{t}\); \(episode\): One of the simulations; \(t\): The time step; \(\epsilon\): A variable contains \((s_{t},a_{t},r_{t+1},s_{t+1})\); \(M\): A list stores \(e\); Capacity: Maximum capacity of \(M\);
1 forepisode \(=1\)to\(N\)do
2 for\(t=1\)to\(sim\_len\)do
3 Observe \(s_{t}\);
4 Choose \(a_{t}\) randomly with probability \(\epsilon\), else choose \(a_{t}=\max f(s_{t})\);
5 Send action \(a_{t}\) to Priority Assignment Model;
6 Take action \(a_{t}\), observe \(r_{t+1},s_{t+1}\);
7 iflen(\(M\)) ==\(Capacity\)then
8 delete \(M[0]\);
9 Append \(e=(s_{t},a_{t},r_{t+1},s_{t+1})\) to \(M\);
10 Run Algorithm 2 to train \(f\);
11\(s_{t}\gets s_{t+1}\);
12
13 return\(f\);
```  
**Algorithm 1**Q-Learning based intersection scheduling algorithm  
We update \(Q(s_{t},a_{t})\) at every time step, and the Q-table store all Q-values of state-action pairs that will eventually converge. A learning rate of value 0 means that the agent learns nothing and only uses the old Q-value in memory, whereas a learning rate of value 1 means the agent only considers the new information just estimated. Every time the agent steps into a new state \(s_{t}\), it can look up the Q-table to see which action \(a_{t}\) is the best choice. However, suppose the Q-table is too large. This means the combination of state space \(S\) and action space \(A\) is too large, and we will suffer from the problem called the curse of dimensionality, which refers to the phenomenon that when the dimensionality of the Q-table increases, the training time needed to converge will increase exponentially; i.e., the time to convergence will be extraordinarily long. Therefore, we use the method called deep Q learning [8] by combining a deep neural network with traditional Q learning to solve the problem of the curse of dimensionality. With the help of neural networks, we no longer need to maintain the large Q-table. At each time step \(t\), the input is \(s_{t}\) and the output is \(c_{t}\) that represents a vector consisting of all action-values. The neural network architecture is first the input layer using \(s_{t}\) as input with the ReLU activation function. There are three hidden layers, and each layer has 64 neurons with a relu activation function. The output layer has \(|A|\) neurons (i.e., number of actions) with a softmax activation function. Each layer is fully connected with the other. The loss is computed by using Equation (8):  
\[Loss=\frac{1}{2}\Big{(}r_{t+1}+\gamma\max\big{(}f(s_{t+1})\big{)}-f(s_{t}) \Big{)}^{2}. \tag{8}\]  
The loss value is back-propagated to the network and then updates the weights of network according to the loss to better approximate the action value.