## 3 Construction of Integrated Control Problems Based on Heterogeneous FACMAC
### FACMAC-Based Combined VSL and LCC Controller
#### 3.3.4 Reward  
In Dec-POMDP problems, cooperative multi-agent systems typically share a global reward function. This means that all agents aim to maximize the overall performance of the entire system when making decisions, rather than just their individual performance. This emphasizes the importance of teamwork and collective goals. In the multi-agent merging bottleneck integrated control model discussed in this paper, local integrated control groups optimize traffic flow efficiency and alleviate high lane density at specific ramps through VSL and LCC control. Considering the upstream and downstream interactions in the global traffic integrated control system, the integrated control agents at different ramps influence each other's traffic conditions, creating a chain reaction. The propagation of traffic flow can impact the broader highway network globally. Previous studies aimed at mitigating merging traffic bottlenecks often focused on reducing the total travel spend (TTS) within the entire road network system, which includes the total travel time within the study area and the queuing time upon entry [27; 36]. This approach is somewhat effective in simplified single-agent scenarios or with single traffic control measures. However, using TTS in multi-agent integrated control can lead to unnecessary confusion among agents. Considering the distinction between VSL and LCC control, as well as the relationship between the global and local levels, this paper proposes a mixed reward function. Specifically, based on the objective of integrated control, we aim to maximize the traffic efficiency \(BN_{effect}\) in the merging bottleneck area. Although the traffic flow \(F_{BN}\) is a direct indicator of traffic efficiency, the nonlinear relationship between flow and density in the fundamental diagram better represents road traffic conditions. The FD reveals that as road density approaches the critical density \(\rho_{cr}\), the traffic flow increases, but it sharply decreases when exceeding \(\rho_{cr}\). Therefore, maintaining road density near the critical density \(BN_{effect}\) is crucial for improving traffic efficiency. For outer lanes, the density \(\rho_{r}\) of the ramp lane is also considered due to the impact of merging traffic. The reward function is defined by Equations (11)-(13):  
\[r_{i,j}^{BN}= \begin{cases}\frac{\rho_{i,j}^{2}}{\rho_{cr}^{2}},&\rho_{i,j} \leq\rho_{cr}\\ \max\biggl{\{}0,\frac{\rho_{cr}^{2}-(\rho_{i,j}-\rho_{cr})^{2}}{\rho_{cr}^{2} }\biggr{\}},&\rho_{i,j}>\rho_{cr}\end{cases} , \tag{11}\]  
\[r_{i,out}^{BN}= \begin{cases}\frac{\rho_{i,out}^{2}}{\rho_{sinh}^{2}},&\rho_{ i,out}\leq\rho_{wish}\\ \max\biggl{\{}0,\frac{\rho_{sinh}^{2}-(\rho_{i,out}-\rho_{wish})^{2}}{\rho_{ wish}^{2}}\biggr{\}},&\rho_{i,out}>\rho_{wish}\end{cases} , \tag{12}\]  
\[r_{i,j}^{upstream}=-\frac{\rho_{i,j}^{2}}{\rho_{cr}^{2}}, \tag{13}\]  
At time step \(t\), the system receives an overall reward \(r_{t}\), which is determined by the density of the the merging bottleneck area and the upstream region. In Equation (11), the inner lane reward \(r_{i,j}^{BN}\) of the merging bottleneck area is designed to optimize traffic flow when the density \(\rho_{i,j}\) is below the critical density \(\rho_{cr}\), ensuring maximum traffic throughput. We also aim to avoid a rapid decrease in capacity when \(\rho_{i,j}\) exceeds \(\rho_{cr}\). Similar to Equation (11), the density \(\rho_{i,out}\) of the outer lane in the merging bottleneck area is significantly influenced by the merging traffic at the entry ramp. Therefore, the reward \(r_{i,out}^{BN}\),out for the outer lane in the merging bottleneck area is defined in Equation (12). The desired lane density \(\rho_{wish}=\rho^{\sigma}-\rho_{r}\), where \(\rho_{r}\) is the ramp lane density, indicates that the outer lane should transfer more excess traffic to the inner lanes via the LCC to obtain a higher reward.