# Variable Speed Limit and Ramp Metering for Mixed Traffic Flows: A Review and Open Questions
## 2 Application of VSL and RM Control in HDV Traffic Flows
### RL-Based RM  
An RM framework that uses RL as a robust and nonparametric generic optimization scheme that allows controlling of discretized partial differential equations was presented in [54]. An algorithm that enables experience sharing between agents and specialization of each agent due to multi-task learning to train Neural Networks (NNs), that was denoted from Mutual Weights Regularization (MWR), was also tested. MWR is an NN training approach that allows RL to learn a policy in a multi-agent environment without the curse of dimensionality due to the number of agents. Applied to RM's actual traffic control problem, a model-free approach achieves a comparable control level to the currently used  
Figure 2: Block scheme of a general Ramp Metering (RM) controller. model-dependent implementation of the ALINEA algorithm. The results show that the average speed is globally increased while the total number of vehicles on the motorway is decreased. The MWR approach resulted in a significant performance improvement, almost reaching the performance of the ALINEA algorithm in both cases. In [55], an algorithm based on the k-Nearest Neighbor Temporal Difference (kNN-TD) was developed and tested on a microscopic traffic simulation model, which was used as a replica of the real world and a test environment for evaluating three scenarios: do-nothing (no metering), ALINEA controller, and kNN-TD controller. The results show that the ALINEA controller reduced \(TTT\) by 27%, while kNN-TD reduced \(TTT\) by 44% compared to the no control scenario. The average queue length was reduced to 112 vehicles in the proposed method compared to 231 vehicles in the case of ALINEA controller. In [56], an RL-based density Control Agent (RLCA) was proposed. The RLCA objective function is to optimize the mainline motorway density to minimize \(TTT\) and maximize traffic flow. RLCA was tested with two different traffic network architectures and demand scenarios. In the first case study with a 6 km long mainline, the proposed RLCA guaranteed satisfactory performance by retaining the flow close to the motorway operational capacity. The simple QL algorithm managed to keep the mainline density close to 80 veh/km/2 lanes, compared to the no control case where the density surpassed 110 veh/km/2 lanes. The optimal control policy was determined based on two reference points: motorway capacity and \(\rho_{c}\). In the second case study with a 500 m long mainline, RLCA always performed the optimal action. In [57], a network-level RM framework was proposed based on modeling by the collaborative Markov Decision Process (MDP) and an associated cooperative QL algorithm based on a payoff propagation algorithm under the coordination graph framework. Three design strategies were analyzed. The first design was independent learning, where the greedy policy for agents was implemented, and an action was chosen to maximize its local reward. The second network-level design was fully distributed and based on the collaborative MDP and cooperative QL algorithm based on the payoff propagation algorithm. In this design, the actions were coordinated between the control agent and its neighboring agents. The cooperative Q-function was updated globally. The third design is a centralized cooperative multi-agent system that can be considered as a single large agent. The cooperative Q-function was updated with a single Q-function. The first design reduced the \(TTT\) by 0.5%, the second by 6.5%, and the third by 6.9%. The multi-agent Deep RL (DRL) RM algorithm, based on loop detectors data, was proposed in [58]. A multi-agent DRL framework is used to generate an appropriate RM scheme for each RM in real time to improve the operational efficiency of the urban motorway. Multi-agent proximal policy optimization architecture is introduced to solve the RM problem. The proposed algorithm was compared with no control, fixed-time control and ALINEA.