## 2 Related Works
### Application of Reinforcement Learning in VSL and LCC  
In traffic control, feedback control and MPC control face challenges, such as frequent strategy adjustments leading to secondary congestion, system oscillation instability, poor convergence in complex traffic environments, and occasional failure to converge [19]. To address these issues, more flexible and intelligent control strategies are introduced. Reinforcement learning (RL) can learn optimal strategies directly from interactions with the environment without pre-modeling the system. This adaptability allows RL to better handle complex and dynamic traffic environments, providing robust responses to system changes and uncertainties [20]. Reinforcement learning, a popular intelligent control method in automatic control, has achieved notable results in previous traffic control studies [21; 22; 23; 24]. However, most RL-based traffic control research has focused on small-scale road networks, mainly targeting single-point ramp control. Recently, researchers have turned to more realistic multi-ramp coordinated control. Applying RL to multiple ramp traffic networks presents challenges, as agents typically react to environmental changes individually, without considering coordination among agents (e.g., VSL controllers). This lack of coordination can result in suboptimal overall behavior [25]. To maximize regional traffic efficiency, more studies combine traffic control systems with collaborative Multi-agent Reinforcement Learning (MARL) algorithms to achieve cooperative control among multiple ramps [26]. [PERSON] et al. [27] were the first to transform the VSL problem into a MARL problem, considering a distributed MARL system in a V2I environment. The system comprises multiple ramp control agents that communicate and cooperate to maximize highway traffic flow and safety benefits. To address non-stationarity issues among multiple traffic control agents, [PERSON] et al. [28] employed the Centralized Training and Decentralized Execution (CTDE) paradigm with the MADDPG algorithm to optimize large-scale VSL control for continuous traffic bottlenecks. Compared to traditional independent agent or feedback-based VSL control strategies, their method effectively improves overall highway network flow and safety by coordinating cooperation among multiple VSL controllers. Research combining RL with multiple traffic control strategies is relatively sparse, likely due to significant coordination challenges: determining the mutual influence of different control methods, setting control cycles for different controllers, and accurately and efficiently controlling multiple traffic controllers [29]. [PERSON] et al. [30] proposed a decentralized reinforcement learning method for highway traffic control in South Africa. They simultaneously applied RM and VSL, using decentralized RL to solve control problems online. Their study claims to be the first to apply MARL methods to real traffic control scenarios, particularly in handling concurrent RM and VSL issues. [PERSON] et al. [31] proposed a method combining speed harmonization and lane merging control to improve traffic flow and fuel efficiency by controlling CAVs. CAVs adjust speed based on sensor data, optimizing vehicle-following speeds and prioritizing merging in bottleneck areas to enhance traffic efficiency. In their study, speed harmonization and merging control are treated as a homogeneous group, using a single neural network for training and taking coordinated control actions based on the same network. The results indicate significant benefits in fuel consumption and traffic congestion reduction from the synergy between speed harmonization and merging control. [PERSON] and [PERSON] [32] proposed a control strategy combining VSL and lane change guidance to enhance highway traffic safety and efficiency through distributed deep reinforcement learning. Their study simultaneously employs VSL and LCC to address complex traffic scenarios that could lead to secondary collisions.