# An Overview of Reinforcement Learning Methods for Variable Speed Limit Control
## 1 Introduction  
Today's available real-time traffic data from motorways enables a practical application of services from Intelligent Transportation Systems (ITS) to be used as traffic control measures for improving traffic [1]. This improvement is made in terms of better utilization of existing infrastructure rather than building an additional one. This is specially done on urban motorways that present an integral part of the urban traffic network. Urban motorways serve as bypasses for transit traffic, but they also serve the local traffic originating from the city or connect city districts. They are well connected with the local urban traffic network through a larger number of near on- and off-ramps. Periodically, they become oversaturated, mostly near an on-ramp caused by local traffic entering the motorway. This happens because urban motorways are also heavily used by the local commuters that also generate severe congestion at its on- and off-ramps. The cost of indirect consequences of traffic congestion is represented by the form of time spent by passengers in the traffic jams, air pollution, and accidents. This presents a significant problem today. For example, emissions from road traffic are responsible for 72% of total greenhouse gas emissions from the transport sector in EU [2]. Thus, the EU plans to reduce emissions produced by the transport sector significantly, and any such reduction on motorways can lead to tremendous gains for the cost-effectiveness in general. Building additional road infrastructure is one way to adopt the traffic network to the constant increase in traffic demand. However, expending the motorway infrastructure is not always a solution as proved by the theory of induced traffic demand, where increases in motorway capacity will induce additional traffic demand [3]. This is known as the Braess paradox [4]. Two of the commonly used traffic control strategies for the mainstream traffic flow control on urban motorways are Variable Speed Limit (VSL) control and Ramp Metering (RM). VSL influences the traffic flow dynamics by changing the allowed speed limit on a particular motorway section [5], RM influences the motorway mainstream by allowing only a portion of the traffic flow from a particular on-ramp to enter the motorway according to the current traffic flow conditions [6]. Thus, preventing the occurrence of congestion on the motorway mainstream. The focus of this study is on VSL. VSL control systems can be classified as static or dynamic as it is explained in [7]. Static VSL is based on hourly or seasonal changes in the speed limit, while the dynamic one is directly dependent on the current traffic flow or weather conditions. The main objective of VSL control systems is to improve throughput and traffic safety on motorways due to the concept of speed harmonization [8; 9; 10]. VSL aims to ensure stable traffic flow in motorway areas affected by recurrent bottlenecks. Thus, it has a twofold influence of preventing and alleviating congestion as well. Several classic strategies for VSL have been developed in order to optimize the traffic flow [11; 12; 13]. The control logic of such VSL controllers is based on the classical feedback control theory. They only react to imminent changes in the traffic, which results in a delayed response. Additionally, they require information of the traffic flow dynamics in terms of the fundamental diagram (relation between flow, speed, and density) to tune the controller parameters [14]. The fundamental diagram has to be estimated for each controlled section to ensure the optimal behavior of the controller. Machine Learning (ML), specifically Reinforcement Learning (RL) techniques [15; 16], are promising alternatives for modeling VSL and addressing the mentioned problems. Especially the problem of knowing the fundamental diagram of the controlled section. Various RL algorithms have been successfully applied for solving traffic flow control on motorways [17; 18; 19; 20; 21; 22; 23].