# Variable Speed Limit and Ramp Metering for Mixed Traffic Flows: A Review and Open Questions
## 3 Research and Implementation
### VSL in Mixed Traffic Flows
#### 3.4.3 RL-Based VSL  
In [84], a DRL model was proposed for Differential Variable Speed Limit (DVSL) control where dynamic and distinct speed limits can be imposed among lanes. It is based on the Actor-Critic (AC) architecture to learn a large number of discrete speed limits in the continuous action space. Four reward scenarios were tested. The reward scenarios \(r_{1}\) and \(r_{4}\) are based on several factors such as on-ramp inflow and off-ramp outflow. The reward scenario \(r_{2}\) was the average speed in the downstream bottleneck. The reward scenario \(r_{3}\) was computed by the total number of emerging braking vehicles in the controlled section. DVSL-\(r_{1}\), DVSL-\(r_{2}\), and DVSL-\(r_{4}\) methods showed Average Travel Time (\(ATT\)) reduction of 1.08%, 4.1%, and 2.94%, respectively, while DVSL-\(r_{3}\) showed an increase in \(ATT\) of 6.2%. DVSL-\(r_{2}\) had the best emission reduction in \(HC\) and \(No_{x}\), while DVSL-\(r_{4}\) showed the best emission reduction in CO and PMx. The AC-based DVSL agent was compared to the baseline scenario with no control, and baseline scenarios with AC and Deep Q-Network (DQN)-based VSL controllers. The DVSL improved \(ATT\) by 8.1% in scenarios with incidents and 5.8% in scenarios without incidents, while the VSL-DQN improved \(ATT\) by 1.7% and 2.9%, respectively. The VSL-DQN and QL performed better than the VSL-AC by a very slight margin.  
In [90], a traffic flow control system was tested that includes AVs whose speeds are controlled by a centralized agent based on deep learning. The learning structure is based on Recurrent Neural Networks (RNNs), where the Gated Recurrent Unit (GRU) algorithm solves the vanishing gradient problem that occurs when learning a classical RNN network. Introducing 10% AVs into the traffic flow enables better control of the traffic flow in a structure where each AV is a mobile actuator of the control system. The state of the agent's environment is determined by the density and average speed of the HDVs and the AVs.  
In both cases, the data are measured for each lane on each model segment. The value of the outflow measured at the exit of the bottleneck was also taken into account to describe the state. The actions were defined by the speed selection for AV vehicles. The goal of the agent is to maximize the outflow from the bottleneck. Therefore, the reward function is formulated as the output flow from the bottleneck area, measured at intervals of 20 s. Compared to the case without control, the results showed that the AV controller for medium traffic demand (inflow \(\geq\) 1800 veh/h) achieves a \(\approx\)25% higher outflow. For the lower traffic demand scenario, the case without control showed a better result. The results compared to the case where the mainline is controlled by RM (control feedback) showed that the AV controller provides approximately the same outflow from the bottleneck for increased traffic demand (inflow \(\geq\) 1800 veh/h) as RM. The case with RM showed a much better result for lower traffic demands.