## 3 Construction of Integrated Control Problems Based on Heterogeneous FACMAC
### FACMAC-Based Combined VSL and LCC Controller  
Consider two controllers, VSL and LCC, for each entrance ramp area, and call such an integrated entrance ramp controller involving VSL controllers and LCC controllers an integrated control group. As shown in Figure 1(a), the data fusion process integrates key information from the highway environment. The integrated control group monitors real-time traffic speed and density near the entrance ramp. Centralized training simultaneously provides speed limit information and lane change information to alleviate bottlenecks in the merging area. Interaction between the two controllers within the integrated control group and between different integrated control groups impacts the overall system. Figure 2: Architecture diagram of FACMAC-based ramp merge integration control system, where (**a**) depicts the data fusion process, which integrates crucial information from the surrounding highway environment, and (**b**) represents the integrated control framework designed for the merging area in this study. Consider two controllers, the VSL and LCC, for each entrance ramp area. These controllers form an integrated control group that observes traffic speed and density related to the entrance ramp. Centralized training simultaneously provides speed limit and lane change information to alleviate bottlenecks in the merging area. The interaction between the controllers within the integrated control group and across different integrated control groups impacts the overall system performance. Figure 2b illustrates the merging area integrated control framework designed in this study. Each controller agent has a value network and a policy network, which are trained with data \((s,s^{\prime},r,a)\) collected during interactions with the traffic environment and stored in a replay buffer. The output value of each policy network serves as the input to the mixer network. The detailed pseudocode is provided in Algorithm 1. Each agent's actor network is designed using a three-layer fully connected architecture to map states to actions, with two hidden layers following ReLU activation functions to introduce nonlinearity. The output layer uses the tanh function to constrain the action output to the range \([-1,1]\), matching continuous action requirements. The critic network uses a fully connected layer to evaluate state-action values, combining ReLU-activated hidden layers for processing, and directly outputs the estimated value, supporting the optimization of agent policies. For the mixer network, two super networks are employed to calculate and generate its parameters. Each super network consists of a linear layer and an absolute value activation function. These two super networks calculate the weights for the first layer neurons \(w_{1}\) and the hidden layer to output the global \(Q_{tot}\) weights \(w_{2}\) by receiving the global state \(s\). Similar network configurations are used to generate biases for the two-layer network, ultimately forming the nonlinear mixer network. In the policy gradient optimization process of reinforcement learning, the high nonlinearity and dimensionality of the objective function often result in multiple local optima. To address this, we adopt the \(\varepsilon\)-greedy strategy to balance exploration and exploitation. By dynamically adjusting the intensity of exploration, this strategy ensures that the agent extensively explores diverse policy spaces during the early stages and gradually converges to the optimal policy in the later stages of training. Additionally, Ornstein-Uhlenbeck (OU) noise is introduced to further enhance exploration diversity. In continuous action spaces, OU noise helps prevent premature convergence to local optima. In traffic system control tasks, it effectively balances exploration and exploitation, avoids abrupt action jumps, and strengthens the diversity and robustness of policy exploration.