# An Overview of Reinforcement Learning Methods for Variable Speed Limit Control
## 2 Application of RL in VSL Control
### Variable Speed Limit Control  
VSL is a traffic control approach that optimizes the mainstream traffic on motorways by adapting the speed limit according to real-time traffic and weather conditions. Nowadays, the speed limit is being displayed on VMS installed at motorways notifying drivers about the current speed limit. Correctly positioned VMSs on the motorway enable the VSL to operate efficiently in terms of flow harmonization or to alleviate congestion occurring in the areas near on- and off-ramps [11]. The significant contributions of VSL on the traffic flow is seen through the speed harmonization [8; 9; 38], which is reflected in the smaller speed differences of vehicles between lanes and within the lane, as well as in reduced speed variance between upstream free-flow and the downstream flow in the congested area. Consequently, a safer and more stable traffic flow is obtained [39; 40; 41]. Preventing traffic breakdown by managing the flow rate entering the bottleneck area by inducing an artificial \"controllable\" bottleneck using VSL upstream of the real bottleneck is studied in [11; 42; 43]. The reduced mean speed of vehicles under the active VSL area produces higher traffic density and a decrease in the mainstream flow. A reduction in flow will be temporary for higher speed limits and lower traffic loads [44], while permanent reduction is evident for lower speed limit values at higher traffic loads [45]. Such artificially created bottleneck located upstream of occurrence of an unmanageable bottleneck (congestion) allows the VSL controller to limit the inflow rate to the congested area to the approximate value of the bottleneck operational capacity (\(\approx q_{cap}\))) which prevents traffic flow breakdown and, thus, enables higher throughput through a congested area. This is the basic working principle of VSL when used for bottleneck control, as presented in Figure 1.  
As shown in [11], in addition to obtaining speed limits, the positioning and choosing of the appropriate length of the section covered by VSL is the crucial part for optimal utilization of the VSL  
Figure 1: Application of Variable Speed Limit (VSL) for the bottleneck control. system. A recent overview of VSL control strategies, which summarizes the VSL influence on the managed traffic flow, can be found in [31]. The benefit of using RL over classical feedback control methods is that there is no need for knowing the explicit model of traffic flow dynamics and the influence of VSL on it. RL in general faces this problem by a learning agent that learns to act by trial-and-error interactions within its environment [16; 46] (see Figure 2). In the standard RL paradigm, an agent is connected to its environment via the perception and action framework. At each step of interaction, the agent senses the environment and then selects an action to change the state of the environment. This state transition generates a reinforcement signal (reward or penalty) received by the agent [47]. The agent's environment is assumed to be modeled as a Markov Decision Processes (MDP). An MDP is a tuple \(\langle S,A,P,R\rangle\) where \(S\) is a finite set of \(n\) discrete states and \(A\) is a finite set of actions available to the agent. The actions are stochastic and Markovian in the sense that an action \(a_{t}\) in a given state \(s_{t}\in S\) results in a state \(s_{t+1}\) with fixed probability \(P(s_{t+1}|s_{t},a_{t})\), while \(R\) represents the reward received after state transition. Policy \(\pi\) is a function representing a mapping from states to actions \(\pi:S\mapsto A\), which optimizes performance, for example, expected accumulated reward received. RL methods differ according to the exact measure and optimization criteria (total reward optimization, discounted reward optimization, or average reward optimization) used to select actions.