# An Overview of Reinforcement Learning Methods for Variable Speed Limit Control
## 3 Research and Implementation
### Results of Research Questions
In order to reduce the difference in speed limit from 120 km/h to \(VSL_{app}\), the speed limit at the upstream section is adjusted using the rule given with the following Equation (2):  
\[VSL_{us}=max[(VSL_{app}+\delta),120], \tag{2}\]  
where \(\delta=10\) km/h and denotes the maximal difference between the upstream \(VSL_{us}\) and applied \(VSL_{app}\) speed limit. This ensures a smoother speed transition of vehicles between the adjacent upstream (free-flow) segment and the segment managed by VSL. Thus, preventing sudden changes in vehicle speed reduces the likelihood of shock waves on the mainstream. A microscopic simulation model is based on a motorway stretch of the \(N1\) national motorway outbound from Cape Town in South Africa's Western Cape Province and was developed within the AnyLogic 7.3.5. University Edition software suite using built-in Road Traffic and Process Modeling Libraries. In [34], a novel VSL control algorithm under the V2I environment aiming to optimize the motorway traffic mobility and safety is presented. The control system is a multi-agent system consisting of several VSL agents. The agents work cooperatively using the proposed distributed RL approach DQL-VSL where kNN-TD algorithm is used as the general function approximator (see Figure 3). The goal of the control algorithm is to improve traffic mobility by maintaining motorway traffic density slightly under the critical point \(\rho_{c}\) to produce the maximum traffic volume. In the case of traffic safety, the objective is to reduce the speed difference between adjacent segments (Figure 4). The control system is tested using the open-source microscopic traffic simulation software MOTUS. Separated results were analyzed regarding the motorway section and motorway bottleneck as well regarding the mentioned two objectives. Results revealed that compared with no-control cases, the proposed DQL-VSL can noticeably decrease the system \(TTT\) and increase the bottleneck outflow. Moreover, the speed difference between motorway segments indicating the potential rear-end collision risk is significantly reduced. In the case of traffic mobility control and measured traffic parameters in the bottleneck, \(TTT\) is lower for 51%, mean speed is higher for 35% and outflow from the bottleneck is higher for 24%. Interesting results are obtained regarding the average bottleneck speed difference of speed between bottleneck and the adjacent segment where lower reduction is achieved by using traffic safety control strategy compared to traffic mobility control. This could imply a lack in the modeling of the safety control objective. Figure 3: Scheme of traffic state collection and speed limit executing process [34]. In [22], a control strategy based on RM and VSL was proposed. To calculate the RM and VSL control output, the Eligibility Traces based Reinforcement Learning (ETRL) algorithm was used. The state variables are density measured within each section, while action contained RM rate and VSL values at the same time. Therefore, there is no actual cooperation between RM and VSL as it was introduced in the work of [19]. The reward function is modeled as the \(TTT\) minimization problem. The motorway model is based on \(M62\) eastbound motorway and traffic data provided by England Highway Agency. Testing is done in MATLAB by using macroscopic simulations based on the Cell Transmission traffic flow Model (CTM). During the rush hours, the average speed is maintained above 70 km/h, which is how the authors state high average speed for rush-hour congestion. In [49], authors introduced W-learning based VSL (WL-VSL): a novel multi-agent RL-based VSL control approach. WL-VSL is implemented on a motorway simulation scenario where two agents are learning, using the W-learning algorithm [55], to control two segments upstream of a congested area jointly (Figure 5).