# An Overview of Reinforcement Learning Methods for Variable Speed Limit Control
## 3 Research and Implementation
### Results of Research Questions
The reward function (3) for each agent is based on the agent's local performance by sensing of average speed \(\bar{v}_{i,l+1}\) within agent's segments as well as the downstream bottleneck by sensing the \(TTS\) value measured in sections \(L_{1}\), \(L_{2}\) and \(L_{3}\), as denoted in Figure 5.  
\[r_{i,l+1}=-\big{[}(1-\frac{\bar{v}_{i,l+1}}{\delta})(\frac{1}{2}(TTS_{1}+TTS_{ 2})+TTS_{3})\big{]} \tag{3}\]  
WL-VSL is evaluated on microscopic simulations using two traffic scenarios with dynamic and static traffic demand in the SUMO simulator. WL-VSL was compared with base cases: no control, single agent, and two independent agents. WL-VSL outperforms these baselines with respect to measured traffic parameters (\(TTS\), density, and average speed) in the downstream congested area of the simulated urban motorway. Improvement of traffic parameters up to 18.18% regarding the decrease of traffic density and 7.3% higher average speed is achieved in the bottleneck during the peak hour in the case of multi-agent WL-VSL. The study [35] proposed an RL-based VSL control algorithm to reduce crash risks associated with oscillations near the bottleneck. More precisely, the state, action, and reward for the QL-based VSL control were designed to improve safety. Density within three motorway sections are used to be the state variables (Figure 6). The speed limit ranges from 20 to 65 mph with an increment of 5 mph. The objective of QL-VSL control is to minimize the total risks of crashes associated with oscillations near motorway bottlenecks. The total crash risk was calculated using a crash risk prediction model developed by the same authors [56], taking into account the characteristics of vehicle  
Figure 4: Time to the collision between two adjacent segments [34]. Figure 5: Motorway model [49]. deceleration trajectories. The QL-VSL was trained to learn the optimal speed limit for various traffic states to achieve a goal of safety optimization. The CTM based simulation model was modified as the simulation platform for evaluating the control effects. The results showed that after the training process, the proposed QL-VSL control successfully reduced the crash risks by 19.4% while only increased the \(TTT\) by 1.5%. A continuous online learning function was developed in RL to enhance the robustness of the control strategy regarding the overspeed issue. The results showed that with continuous learning, the QL-VSL control performed reasonably well under lower driver compliance situations. In [23], a per lane VSL, based on Lagrangian control using Deep-RL (DRL) is proposed. It is a learning structure based on Recurrent Neural Network (RNN) [57] where the application of the so-called Gated Recurrent Unit (GRU) [58] mechanism solves the problem of vanishing gradient (decay of information through time) that occurs when learning a classic RNN using the backpropagation through time procedure. GRU is a simplified version of the Long-Short Term Memory (LSTM) recurrent network introduced in [59] that combines forget and input gates to form an 'update' gate. In general, the LSTM can maintain temporal information in the state for a long number of timesteps and is widely used in sequential data analysis. Since the traffic flow used in research contains AVs, a centralized agent based on deep reinforcement learning DRL-VSL can directly adjust the speed to AVs within specific traffic lanes and, in this way, control remaining traffic flow, instead of using classical VMSs. State space of the controller is described with: density and average speed of human drivers in each lane for each observed section, density and average speed of AVs in each lane for each observed section, and the outflow at the final bottleneck. The reward function is modeled to sense the outflow from the bottleneck. The proposed solution was tested using _Flow_, which is a new library for application of DRL to traffic micro-simulators SUMO.