## 2 System Architecture
The joint actions \(a=(a_{1},a_{2},\ldots,a_{N})\) collectively drive environmental transitions, leading to new states \(s^{\prime}\) and reward feedback \(r=(r_{1},r_{2},\ldots,r_{N})\). Using the MDP framework defined in (1), the cumulative discounted reward for agent \(i\) is modeled as Equation (2), which captures the expected long-term return starting from state \(s\) under policies \(\pi^{i}\) and \(\pi^{-i}\). \(\pi^{i}\) denotes the strategy of intelligent agent \(i\). \(\pi^{-i}\) denotes the joint strategy of intelligences other than intelligent agent \(i\). \[V^{i}_{\pi^{i},\pi^{-i}}(s)=\mathrm{E}\Bigg{[}\sum_{t\geq 0}\gamma^{t}r^{i}(s _{t},a_{t},s_{t+1})\Big{|}a_{t}^{i}\sim\pi^{i}(\bullet\big{|}s_{t}),s_{0}=s \Bigg{]} \tag{2}\]  
Figure 2: Schematic diagram of multi-agent interaction with the environment. For a multi-agent system, multiple agents will reach an equilibrium point. For any agent, no other strategy can be adopted to obtain a higher cumulative reward, as shown in (3). This equilibrium ensures that no single agent can unilaterally improve its cumulative reward by changing its strategy, assuming other agents' strategies remain optimal. \[V^{i}_{\pi^{i},\pi^{-i,*}}\geq V^{i}_{\pi^{i},\pi^{-i,*}},\forall i\in\text{ Agent }s \tag{3}\]  
In a multi-agent environment, a single agent interacts with the environment. The purpose of the agent is to start from any initial state \(s_{0}\), with the maximum expected return as shown in (4), where E{ } denotes the mathematical expectation, calculating the expected cumulative reward under the policy \(\pi\). \[\pi^{*}_{t}=\text{argmax}_{\pi}E\Bigg{\{}\sum_{m=0}^{\infty}\gamma^{m}r_{t+m} |s_{t}=s\Bigg{\}} \tag{4}\]  
where \(E_{\pi}\) represents the expected value under the strategy, \(m\) represents the subsequent time period, \(r_{t+m}\) denotes the reward at time \(t+m\), given the current state \(s_{t}=s\). Where \(\pi^{*}_{t}\) denotes the optimal policy at time \(t\), \(V^{t}_{\pi}(s)\) and \(Q^{t}_{\pi}(s,a)\) are the time-dependent value functions, explicitly linking to the state \(s_{t}\) and reward \(r_{t+m}\) at time \(t\). Under strategy \(\pi\), the maximum reward value of state \(s\) is represented as \(V^{t}_{\pi}(s)\). \(V^{t}_{\pi}(s)\) denotes the value function of state \(s\) under the strategy \(\pi\), which is the expected cumulative reward for following the strategy \(\pi\) from state \(s\) onwards. \(V^{t}_{\pi}(s)\) is defined as Equation (5). \[V^{t}_{\pi}(s)=\text{max}_{\pi}E_{\pi}\Bigg{\{}\sum_{m=0}^{\infty}\gamma^{m}r_ {t+m}|s_{t}=s\Bigg{\}} \tag{5}\]  
Defined under strategy \(\pi\), the value \(Q^{t}_{\pi}(s,a)\) of action \(a\) taken against state \(s\) represents the expected return of taking action \(a\) starting from state \(s\) and subsequently following strategy \(\pi\), as formalized in Equation (6). \[Q^{t}_{\pi}(s,a)=\text{max}_{\pi}E_{\pi}\Bigg{\{}\sum_{m=0}^{\infty}\gamma^{m} r_{t+m}|s_{t}=s,a_{t}=a\Bigg{\}} \tag{6}\]  
In the single-agent case, the state includes all information about the environment, while a random process satisfying the Markov property can only use a portion of the information. Multi-agent reinforcement learning processes are more challenging when training an agent; other agents are also considered part of the environment, and the agent's observation of the environment does not include action information. During the training process, the strategies of other agents often change, making the observation results more uncertain. A stochastic process that does not strictly satisfy the Markov property is defined as a Partially Observable Markov Decision Process (POMDP) [16]. POMDPs are crucial in multi-agent systems because each agent's observations are limited to local information, and the actions of other agents introduce uncertainty into the environment, violating the full observability assumption of standard MDPs. Information collection includes topology information, bandwidth, resource utilization, and latency.