## 3 Hybrid Multi-Agent Reinforcement Learning (Hybrid MARL)
### Multi-Agent Reinforcement Learning Framework  
In large-scale road networks, optimizing individual intersections in isolation is insufficient to achieve network-wide efficiency. Therefore, a multi-agent reinforcement learning (MARL) framework is proposed to enable decentralized traffic signal controllers to cooperate. However, MARL presents unique challenges, such as non-stationarity (as agents continuously adapt their strategies) and exponential growth of the action space. To address these issues, we propose a Hybrid MARL framework that balances local decision-making with global coordination.The detailed hyperparameter settings used for training our Hybrid MARL framework are presented in Appendix A.  
Figure 2: Single-agent reinforcement learning framework for traffic signal control, where each agent operates independently without considering network-wide coordination.  
The proposed Hybrid MARL framework consists of two levels of agents:  
1. Sub-region Agents: These agents control a small cluster of intersections and optimize traffic flow within their local region. Each sub-region agent is responsible for managing a local cluster of intersections. In our experimental setting, each sub-region typically contains 4 to 8 signalized intersections, depending on the density and structure of the urban road layout. This decomposition ensures both manageable learning complexity and sufficient spatial coordination.
2. Global Agent: This centralized agent aggregates traffic information from all sub-region agents and provides high-level guidance to ensure network-wide coordination.
3. This hierarchical structure reduces computational complexity while maintaining effective coordination across large road networks. The overall structure of the Hybrid MARL model is presented in Figure 3, where local agents operate within defined regions, and a central agent optimizes traffic flow at a global level.  
The learning process follows a Centralized Training and Decentralized Execution (CTDE) paradigm. During training, all sub-region agents share experience and receive coordinated feedback from the global agent, allowing them to collectively learn optimal traffic control policies. During real-time execution, each sub-region agent operates independently, making decisions based on local observations. This hybrid approach achieves scalability while retaining the benefits of collaborative learning.  
Each agent in the Hybrid MARL framework is trained using the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, which extends the standard Deep  
Figure 3: Hybrid MARL framework with sub-region agents handling local intersections and a global agent ensuring network-wide coordination.  
Deterministic Policy Gradient (DDPG) to a multi-agent setting. The policy gradient update is computed as  
\[\
abla_{\theta}\mathrm{J}(\theta_{i})=\mathrm{E}\big{[}\
abla_{\theta_{i}}\mathrm{ log}\pi_{\theta_{i}}(\mathrm{a}_{i}|\mathrm{s}_{i})\mathcal{Q}_{\pi}(\mathrm{s}, \mathrm{a})\big{]} \tag{1}\]  
\(\pi(\theta_{i})\) represents the policy network of agent \(i\), and \(Q\pi(s,a)\) is the action-value function. The global agent provides an auxiliary reward signal to sub-region agents to encourage network-wide optimization.  
To improve stability and convergence, we implement Experience Replay and Target Networks, which mitigate policy fluctuations and enhance learning efficiency. Additionally, Graph Attention Networks (GATs) are incorporated into agent architectures to improve communication between neighboring intersections, allowing agents to learn cooperative traffic control strategies.