# Traffic Light Cycle Configuration of Single Intersection Based on Modified Q-Learning
## 3 Traffic Light Cycle Recommendation Methods
### Q-Learning
\[\varepsilon-greedy\;action\;A=\left\{\begin{array}{c}\;\;\;\;\;\;\;\;\;\; \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;  
* If the New-AVDT is better than the Best-AVDT, update the best state and jump back to step (2); otherwise give a negative reward.
* If the action value is not zero, the agent randomly selects the new state and jumps to step (2.1).
* If the agent continues to find no better average vehicle delay time, the output is the best state.
* Agent recommends the best state cycle configuration.  
The purpose of this experiment is to quickly find the minimum average vehicle delay time, so the selection action only looks at the current best state. If the action is not better than the previous average vehicle delay time (AVDT), it will give a negative reward (i.e., increase the AVDT), otherwise it will give a positive reward (i.e., decrease the AVDT). The reward function is listed in Equation (10). By using the Q-table to record the poor cycle configuration mode, the agent will not go to the poor state before the previous attempt, and then proceed to the better direction. However, such a method is prone to a local optimum. In order to solve this problem, this study adds a random state selection mechanism that effectively prevents the model from falling into a local optimum. The algorithm of the modified Q-learning approach is described in Algorithm 2, where \(Q_{best}\) represents the current small average delay time, and \(S_{best}\) is the state of the current minimum average delay time.  
\[\textit{Reward Function}=-AVDT \tag{10}\]  
```
Algorithm parameters: Step size \(\alpha\in(0,\ 1]\), small \(\varepsilon>0\) let \(S_{best}\) be the traffic light cycle time of the previous time period let \(Q_{best}\) be the negative average delay time of the original traffic light cycle Loop for each episode:  Initialize \(Q(s,\ a)\), for all \(s\ \in\ S^{+}\), \(a\in A(s)\), arbitrarily, except that (terminal, \(\cdot\)) = 0 \(S\ \leftarrow\ S_{best}\)  Loop for each step of episode:  Choose \(A\) from \(A\) using policy derived from \(Q\) (e.g., \(\varepsilon\)-greedy)  Take action \(A\), observe \(R\), \(S^{\prime}\) \(Q(S,A)\leftarrow(1-\alpha)Q(S,A)\ +\ \alpha[R\ +\ \gamma max_{q}Q(S^{\prime},a)\ -\ Q(S,A)]\) \(S\gets S^{\prime}\)  If \(Q(S,A)>\ Q_{best}(S_{best},A)\) then \(S_{best}\ \leftarrow\ S^{\prime}\) until \(S\) is terminal
```  
**Algorithm 2** Modified Q-learning. In Algorithm 2, the optimal \(Q\) value cannot be known before training, so the \(Q_{best}\) value must be updated by each training. Since the conventional Q-learning is a random initialization state, the Q value that causes the optimal state has a chance to be successively selected in the initial stage. Therefore, the Q value that causes the optimal state becomes too small, so the optimal state cannot be selected afterwards. In order to solve this problem, we modify the initial state by continuously updating \(S_{best}\), which ensures that the current state finds a better state in a shortest time. In this experiment, \(\alpha\) is setting to 1, and \(\gamma\) is setting to 0; that is, only the current reward is considered. Because the purpose of this experiment is to quickly find the minimum average delay time, it is done by constantly updating the current rewards.