## 3 Construction of Integrated Control Problems Based on Heterogeneous FACMAC
### Heterogeneous FACMAC Algorithm  
The FACMAC algorithm is a MARL algorithm that follows the CTDE paradigm. In FACMAC, a multi-agent actor-critic framework is employed, where a centralized critic is trained by considering the joint actions of all agents. The joint action value function is optimized by decomposing it across individual agents, utilizing a weighted distribution trained with a mixer network. This distribution more accurately reflects the true traffic environment changes in the global \(Q_{tot}^{\mu}\). In the actor, decentralized execution of individual actions is achieved through deep deterministic policy gradient learning. The process of the FACMAC algorithm is illustrated in Figure 1.  
Specifically, FACMAC recognizes the challenge of learning in a centralized critic due to the large number of agents or action spaces. Therefore, the global action value function is decomposed into individual agents, which not only reduces training difficulty but also preserves the decentralized execution of the algorithm. \[Q_{tot}^{\mu}(\tau,a,s;\phi,\psi)=g_{\psi}\Big{(}s,\Big{\{}Q_{i}^{\mu_{i}}( \tau_{i},a_{i};\phi_{i})\Big{\}}_{i=1}^{n}\Big{)}, \tag{2}\]  
In Equation (2), the global action value function \(Q_{tot}^{\mu}\) depends on the joint action-observation history \(\tau\), joint action \(a\), and global state \(s\). Here, \(\phi\) represents the parameters of the agent's policy network. Each agent has its own critic with an action value network \(Q_{i}^{\mu_{i}}\), which is used to evaluate the performance of its policy. The individual action value \(Q_{i}^{\mu_{i}}\) can be computed using the nonlinear monotonic function \(g_{\psi}\) as part of \(Q_{tot}^{\mu}\). The parameters  
Figure 1: FACMAC algorithm diagram. \(\psi\) are computed using a mixer network. The loss function in Equation (3) is used to train \(Q_{tot}^{\mu}\) centrally:  
\[\mathcal{L}(\phi,\psi)=\mathbb{E}_{\mathcal{D}}[(y^{tot}-Q_{tot}^{\mu}(\tau,a,s; \phi,\psi))^{2}], \tag{3}\]  
where \(y^{tot}=r+\gamma Q_{\text{tot}}^{\mu}\left(\tau^{\prime},\mu\left(\tau^{\prime} ;\theta^{-}\right),s;\phi^{-},\psi^{-}\right)\) and the parameters \(\theta^{-},\phi^{-},\psi^{-}\) represent the target actor, critic, and mixer network parameters, respectively. To address the issues arising from agents' dependence on other agents' policy estimates, FACMAC centralizes the policy gradient update across the entire joint action space. This centralization corrects strategy errors and inaccurate evaluations during the learning process:  
\[\
abla_{\theta}J(\mu)=\mathbb{E}_{\mathcal{D}}\big{[}\
abla_{\theta}\mu\
abla _{\mu}Q_{tot}^{\mu}(\tau,\mu_{1}(\tau_{1}),\dots,\mu_{n}(\tau_{n}),s)\big{]}, \tag{4}\]  
where all agents use a unified centralized actor policy network. When estimating the global action value function \(Q_{tot}^{\mu}\), the current policies of all agents, \(\mu=\mu_{1}(\tau_{1}),\dots,\mu_{n}(\tau_{n})\), are sampled to obtain joint actions, and the overall joint action space is optimized through gradient ascent. This updating method is particularly important for optimizing the overall efficiency of highway traffic control. For a highway traffic system involving multiple entrance and exit ramps, traffic integration control points are generally set in merging and diverging areas prone to bottlenecks. Due to the continuity of the highway system and the fluidity of vehicles, decisions made in a single traffic integration control area affect not only the traffic flow near the ramp but also the entire system, especially downstream areas. The FACMAC algorithm can optimize across the entire joint action space rather than being limited to a single control point. However, considering that the FACMAC algorithm primarily addresses control issues in homogeneous cooperative multi-agent environments where all agents share the same policy network, this requires that all agents in the network have identical environmental perceptions and behavior strategies. This means the input and output dimensions of all agents' control processes are fixed.