# An Overview of Reinforcement Learning Methods for Variable Speed Limit Control
## 4 Discussion  
The application of advanced traffic management solutions in the field of ITS enables further improvement of the level of service on urban motorways. One such solution is the VSL control system. VSL can produce a satisfying performance when traffic on the urban motorway is consistent in a spatial-temporal context. However, the effectiveness of VSL decreases in cases when the traffic conditions are exposed to the rapid oscillations in traffic demand or in the case when the motorway capacity is being reduced. RL as one of the approaches for online ML provides an optimal trade-off between the complexity and efficiency among various model-free data-driven traffic control methods. As shown, it is worth applying RL-VSL since its continuous self-adaptation features are able to tackle control problems related to the new, not yet anticipated traffic conditions [18; 21; 35]. From the completed survey, several potential research directions to address the limitations of the existing methods were identified. The limitations of current FB-VSL controllers are the need for an accurately estimated fundamental diagram and delay. The emphasis of the survey is placed on the description of VSL controllers in which algorithms are based on the RL type of ML. The main efficiency drawback of classical VSL algorithms is their inability to adapt control policy to a new traffic situation in which case they operate sub-optimally [21]. During the last five years, according to Table 1, it is  
Figure 7: Traffic-to-image conversion on an urban motorway network. possible to notice an increase in the number of studies looking forward to improving and proposing the new RL-based VSL control algorithms. Regarding different RL techniques and state-action modeling of the environment, including various types of different objectives, the development of RL-VSL strives to bring such approaches closer to a real-world application. A few RL based approaches have been successfully applied in VSL control. Mostly used is the QL algorithm, which has been used in its base form using Q-table. It is well integrated with a more sophisticated method such as function approximation (linear [19; 20], nonlinear [18]) for Q-function generalization. Even though the authors in [22] stated that the ETRL algorithm can find a better optimum compared to the QL, the results obtained by ETRL-VSL are insufficient to confirm that, since there is a lack of comparison with the QL-VSL. Likewise, in [17], further studies are needed to assess if the R-MART based VSL has any benefits over the QL-VSL. An example of the kNN-TD algorithm proposed in [19] has been successfully applied for Q-function approximation in [35]. As shown in [23; 37], an application of DL can be useful when state and action space in the RL-VSL model becomes too large in which case the basic form of RL-VSL (Q-table) cannot be used efficiently. In [19], an interesting approach is explained regarding the third state variable (density \(\rho_{us}\)) measured on the motorway section further upstream from the primary VSL area. This variable is chosen originally to provide a predictive component in terms of motorway demand. Also, it is expected to provide the agent with an indication of the severity of the congestion in cases where it has split back beyond the application area of VSL. In that way, there is no need for predictive models (like in [18]). In that way, the VSL agent can recognize a platoon of vehicles coming into the VSL and downstream bottleneck area. This allows the VSL agent to react even earlier (proactively). The newly proposed concept in [35] has incorporated a risk model into RL-VSL for safer traffic in the case of traffic oscillations on motorways. It is tested on a macroscopic CTM based simulator. Results showed that after the training process, the proposed QL-VSL control successfully reduced the crash risks by 19.4% while only increasing the \(TTT\) by 1.5%.