# An Overview of Reinforcement Learning Methods for Variable Speed Limit Control
## 3 Research and Implementation
### Results of Research Questions
Also, the authors performed an analysis of the robustness of action selection strategies (control policy) by adding noise (modeled using the Gaussian distribution) to the measured traffic parameters speed and density. Control policy behaves well for noise up to \(10\%\) otherwise the learned control policy can no longer be used to appropriately select speed limits. Research in [21] has shown that VSL based on the QL algorithm (QL-VSL) can achieve better results compared with close loop-based algorithms, which uses speed limits adjusted based on control feedback loops (FB-VSL). The FB-VSL sets a speed limit slightly after the traffic jam has already been formed. This can be too late because the control error can be large. Thus, in order to correct the control error, the VSL controller must significantly reduce the speed limit to reduce inflow rate into the congested area in order to maintain traffic parameters (measured density \(\rho\) in the bottleneck) around the desired value \(\rho_{d}\) (often critical density \(\rho_{c}\) for which the maximal traffic flow is achieved). Occasional, more restrictive actions (such as lower speed limits) can negatively impact the flow upstream of the controlled sections and induce new congestion in front of the VSL area. A comparative analysis of controllers has shown that QL-VSL can recognize a potentially problematic condition leading to bottleneck activation and, thus, proactively correct the speed limit before congestion occurs. It can be said that the agent, through interaction with the environment through trials and errors, has the ability to successfully learn how to recognize certain critical traffic (pattern) situations. In that way, QL-VSL is able to predict disturbances in traffic flow and the potential to act preventively somewhat earlier than the classic FB-VSL controllers. QL-VSL succeeds with much less intervention (less speed limit correction) to achieve a better effect in preventing bottleneck activation or capacity drop on the segment of the urban motorway where congestion occurs. Proactive VSL control reduces the likelihood of the creation of larger congestion and reduces the need for stronger actions (lower speed limit values). The environment in which the QL-VSL agent operates is described by the discrete values of measured traffic density in the bottleneck area (mainstream motorway segment immediately after the on-ramp) and the density at the problematic on-ramp. The final set of actions in [21] is ranging from \(20\) to \(65\) mph with an increment of \(5\) mph. The agent's indirect goal is to minimize the Total Travel Time (\(TTT\)) by preventing the occurrence or alleviating the consequences of an already active bottleneck. The effect of each action at a specific traffic flow condition is assessed by the reward function defined by the Poisson density distribution (veh/km) in the bottleneck area stimulating the agent to strive to maintain a flow around \(\rho_{c}\) at maximum vehicle flow and thus minimize \(TTT\). In order to accelerate the convergence of the learning process, an additional parameter \"prize\" (\(+200\)) is introduced for two adjacent statesaround \(\rho_{c}\). To penalize the congested state (density much higher than the \(\rho_{c}\)), an additional \"penalty\" value of \(-400\) is added to the reward function. The learning process takes place offline, i.e., in the background on continuously collected data (states, actions, and rewards). In this way, the VSL agent is able to periodically refresh its knowledge based on continuously collected new traffic data and thus adapt its control policy if needed. In the case of the overspeed issue in terms of relaxing the driver compliance rate to the speed limits, the \"_continuous learning_\" approach for QL-VSL outperformed the basic QL-VSL approach regarding reduction in measured \(TTT\). Eventually, the proposed QL-VSL strategy outperforms baselines (FB-VSL and no-control) with an improvement of \(TTT\) up to \(21.84\%\) in the fluctuating traffic demand scenario.