# An Overview of Reinforcement Learning Methods for Variable Speed Limit Control
## 3 Research and Implementation
### Results of Research Questions
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Paper** & **Year** & \begin{tabular}{c} **RL** \\ **Method** \\ \end{tabular} & \
\begin{tabular}{c} **Compared** \\ **against** \\ \end{tabular} &
\begin{tabular}{c} **Control** \\ **Strategy** \\ \end{tabular} & **Improvements** \\ \hline
[35] & 2020 & Q-Learning & no-control & VSL &
\begin{tabular}{c} 19.4\% lower \\ crash risk \\ \end{tabular} \\ \hline
[49] & 2020 & W-Learning & \begin{tabular}{c} no-control, single \\ and independent \\ VSL agents \\ \end{tabular} & \
\begin{tabular}{c} multi-agent \\ VSL \\ \end{tabular} &
\begin{tabular}{c} 18.1\% lower \\ bottleneck density \\ \end{tabular} \\ \hline
[34] & 2019 & \
\begin{tabular}{c} Distributed \\ Q-Learning (kNN-TD) \\ \end{tabular} & no-control &
\begin{tabular}{c} multi-agent \\ VSL \\ \end{tabular} & 51\% lower \(TTS\) \\ \hline
[19] & 2019 & Maximax MARL & \
\begin{tabular}{c} no-control, feedback, \\ independent, \\ hierarchical MARL \\ \end{tabular} &
\begin{tabular}{c} multi-agent \\ RM and VSL \\ \end{tabular} & 11\% lower \(TTS\) \\ \hline
[23] & 2018 & \begin{tabular}{c} Deep-RL \\ (GRU) \\ \end{tabular} & \begin{tabular}{c} no-control, \\ feedback RM \\ \end{tabular} & \
\begin{tabular}{c} centralized agent \\ VSL to AVs \\ \end{tabular} &
\begin{tabular}{c} 25\% higher \\ bottleneck throughput \\ \end{tabular} \\ \hline
[20] & 2018 & \begin{tabular}{c} Q-Learning \\ linear approx. \\ \end{tabular} & \
\begin{tabular}{c} no-control, Q-tabular, \\ (Tile, Coarse, RBF) \\ \end{tabular} & VSL &
\begin{tabular}{c} 10\% lower overall \\ \(TTS\) \\ \end{tabular} \\ \hline
[21] & 2017 &
\begin{tabular}{c} Q-Learning \\ (kNN-TD) \\ \end{tabular} & no-control, FB-VSL & VSL & 21.8\% lower \(TTT\) \\ \hline
[18] & 2016 & \
\begin{tabular}{c} O-Learning \\ (ANN) \\ \end{tabular} &
\begin{tabular}{c} no-control, \\ certain fixed amount of VSL \\ \end{tabular} & VSL & 30\% lower \(TTS\) \\ \hline
[17] & 2014 & R-MART & no-control & VSL & 18\% lower \(TTS\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The most representative Reinforcement Learning (RL) frameworks for VSL control. the area where congestion occurs. The set of actions contains four speed limits \(\{60,80,100,120\}\) km/h. The reward function is proportional to the negative \(TTS\) measured between the two control intervals (duration of each action). An additional condition is added to the reward that prevents oscillations of the speed limit, and at the same time, penalizes changes in speed limit in which the consecutive absolute difference is greater than \(20\) km/h. An element is also added, representing a reward (amount 0) if the agent recognizes the free-flow conditions and allows vehicles to drive at a maximum speed of \(120\) km/h. The macroscopic model METANET [50] was used to simulate the proposed solution. The Tile coding method was used to generalize the continuous state variables in the case of linear Q-function approximation [16]. The results were also compared with a nonlinear approximation using an Artificial Neural Network (ANN). In the case of ANN, an additional variable describing the traffic density on the managed segment was added to classify the state conditions on the managed motorway. Further, an important difference between previous work is that additional variables were included in the state vector, such as the predicted speed calculated using a parallel METANET simulation based on the current traffic situation and expected traffic demand. As shown by the results, in this case, they were better by \(2\%\) in using cell coding, and \(0.8\%\) in the case of ANN compared to the case without speed prediction. The no-control case was taken as a starting point, in which the mentioned methods showed a decrease in \(TTS\) by approximately \(30\%\). The case for the best \(TTS\) achieved for a certain fixed amount of speed limit is also taken into account. In this case, the VSL controllers failed to achieve a better result suggesting that the proposed VSL learning system needs to be further improved.