## 3 Construction of Integrated Control Problems Based on Heterogeneous FACMAC
### Heterogeneous FACMAC Algorithm
This approach limits the integrated control of the VSL and LCC controllers in single entry ramp merging areas, as it fails to account for differences between various traffic controllers. In this paper's highway environment, the two controllers are heterogeneous agent controllers because the traffic states they need to obtain and their control methods differ. Thus, the FACMAC algorithm cannot be directly applied to train and provide effective control strategies in such a multi-agent environment. To solve this problem, this paper draws on the concept of classes used in the object-oriented distributed reinforcement learning method by [PERSON] et al. [35], constructing heterogeneous agents into different classes through object-oriented principles. Specifically, in this paper, when considering the optimization problem of traffic flow in the highway merging area within a multi-agent system, an effective method is to model the VSL controller and the lane LCC controller in the ramp merging area as independent class objects. The integrated control is then achieved by integrating the state inputs and action outputs of these entities. In this design, each type of controller is considered an independent agent class AC, with its unique properties and behaviors. The VSL agent class \(AC_{VSL}\) determines speed limits based on traffic density and flow information within the ramp merging area. Its input state dimension is denoted as \(D_{s}^{\text{vsl}}\). The LCC agent class \(AC_{LCC}\) focuses on the density differences between different lanes in the merging area, with its input state dimension represented as \(D_{s}^{\text{vsl}}\). By concatenating the state inputs of these two agent types, the overall input dimension becomes \(D_{s}^{\text{vsl}}+D_{s}^{\text{lcc}}\). Unnecessary state dimensions for different agent types are filled with null values to maintain data consistency. Furthermore, since the control methods of the two agent types vary, the action output dimensions also differ. Thus, the output dimension of the policy network is designed to match the maximum action output dimensions among the agent classes. Agents with fewer action output dimensions extract only the required dimensions for their actions. The two controllers in an integrated control group belong to class objects of different intelligences, each having its own critic and actor networks. These controllers differ from other agent classes only in terms of input and output dimensions, while agents of the same class within different integrated control groups share network parameters. This design leverages similar observation spaces and behavior strategies, enabling the sharing of network parameters among homogeneous agents. This approach effectively addresses the failure of the FACMAC algorithm in heterogeneous intelligent body environments. Although the approach may encounter dimensionality issues due to the presence of multiple agent classes in a heterogeneous environment, it remains applicable for the highway merging bottleneck control problem discussed in this paper. Prior studies on traffic active control strategies have focused on variable speed limit control, ramp control, and lane change control.