## 3 Construction of Integrated Control Problems Based on Heterogeneous FACMAC
### Collaborative Multi-Agent Reinforcement Learning  
In real-world scenarios, larger-scale highways are often influenced by multiple ramps and controllers, a situation that can be depicted using MARL. Unlike single-agent reinforcement learning, multi-agent systems, composed of numerous agents, form complex systems where accurately perceiving the global state of the environment is exceedingly difficult. Instead, agents can only access partial and local information, referred to as observations \(o\in O\). Consequently, the environment in MARL is influenced not solely by the actions of a single agent but by the interactions among multiple agents and their respective local environments. Additionally, direct or indirect interactions between agents lead to changes in the overall environment, influenced by the behavior strategies of different agents [33]. The rewards for all agents depend on the strategies of other agents. In MARL, each agent aims to learn the optimal Q-value or strategy, but the environment transition and reward function depend on the actions of all agents, making the environment appear unstable from the perspective of any single agent [34].  
This situation violates the Markov process assumption that agents have full observability of the environment. In the real world, due to sensor limitations, decisions are made based on limited or noise-interfered observations. In MARL, this is referred to as partial observability, leading to state uncertainty and interaction uncertainty among agents during the learning process. For example, even though traffic controllers on different road sections may share a common goal, limited communication capabilities prevent them from providing accurate traffic information to other agents. Additionally, coordinating respective traffic control signals can lead to mutual influences, making overall traffic complex and unpredictable. For this reason, training in partially observable MARL scenarios is conducted through a distributed partially observable Markov decision process.  
Dec-POMDP considers that agents in the environment cannot fully observe the state of the environment and face state uncertainty induced by noise. Dec-POMDP is defined as \(M_{\text{DecP}}=\langle I,S,A,R,T,O,o,\gamma\rangle\). Here, \(I\) represents a finite set of agents; \(S\) denotes the set of environmental states; \(A\) indicates the set of joint action spaces, which includes each agent's individual actions \(A_{i}\); \(R\) denotes the reward function for all agents; and \(T\) represents the joint state transition probability function that describes the probability of transitioning to a new state given the current state and joint actions. In this paper, this function is modeled using macroscopic traffic simulation to obtain realistic information. The function's probability is regarded as \(1\), \(O\) represents the set of observations, with \(o\) indicating the joint observation set, where \(o_{i}\) represents the local observations of the corresponding agent; \(\gamma\) represents the discount factor reflecting the parameter of reward change over time.  
In Dec-POMDP, agents cannot obtain the true state of the entire environment and can only take actions based on their current local observations, without sharing observational information with other agents. Under these conditions, the goal of Dec-POMDP is to achieve effective cooperation among agents, finding a set of optimal strategies to maximize the total expected cumulative return for agents in the environment:  
\[Q_{tot}^{\pi}(s_{t},a_{t})=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{ t},a_{t})|s_{0}=s,a_{0}=a\right], \tag{1}\]