## 3 Problem Formulation and Model Design
### Model Design  
A fully cooperative multiagent task can be described as a Dec-POMDP, which can be denote as a tuple \(G=\langle S,A,P,r,Z,O,n,\gamma\rangle\). \(s\in S\) denotes the true state of the environment. At each step, each agent \(m\in\mathcal{M}=\{1,...,M\}\) chooses an action \(a_{m}\in A\), and all actions form joint action \(\mathbf{a}\in\mathbf{A}=A^{M}\). \(P\) is the state transition function depending on the environment. When all agents complete the decision, the environment will transit from the state \(s\) to the next state \(s^{\prime}\) according to the state transition function \(P(s^{\prime}|s,\mathbf{a}):S\times\mathbf{A}\times S\rightarrow[0,1]\), and return the reward at the same time. All agents share the same reward function \(r(s,\mathbf{a}):S\times\mathbf{A}\rightarrow\mathbb{R}\). \(\gamma\in[0,1)\) is the discount factor. For a Dec-POMDP, the environment is partially observable for agents. Each agent obtains its individual observation \(z\in Z\) according to observation function \(O(s,m):S\times M\to Z\). Each agent has an action-observation history \(\tau_{m}=(a_{m,0},\tau_{m,1},...,a_{m,t-1},\tau_{2,t})\), on which it conditions a stochastic policy \(\pi_{m}(a_{m}|\tau_{m})\in[0,1]\). The joint policy \(\mathbf{\pi}\) has a joint action-value function \(Q_{\pi}(s_{t},\mathbf{a}_{t})=\mathbb{E}_{s_{t+1}:\infty,\mathbf{a}_{t+1}:\infty}[R_{ t}|s_{t},\mathbf{a}_{t}]\), where \(R_{t}=\sum_{i=0}^{\infty}\gamma^{i}r_{t+i}\) is the discounted return. The goal of agents is to find a joint policy \(\mathbf{\pi}\) to maximize the expected discounted reward. In MARL, agents learn to make decisions by exploring the unknown environment and using the feedback received from the environment. In this setting, the objective of each agent is to maximize the shared reward. Then the essential elements of a Dec-POMDP can be defined as follows:  
**State. ** In this scenario, due to the difference in factors, such as latency between each user and each edge server, we define the state of a user \(u_{j}\) under the coverage of a single edge server \(e_{i}\) as \(S_{ij}\), and each \(S_{ij}\) is a 4-tuple as shown in (3). \[S_{ij}=(d_{ij},rtt_{i}(u_{j}),th_{i}(u_{j}),b_{i}(u_{j})) \tag{3}\]  
Then the input state of the edge server \(e_{i}\) is  
\[S_{i}=(S_{i1},S_{i2},\cdots,S_{i|U_{i}|^{\prime}},|C_{i}|) \tag{4}\]  
In (4), we input not only the states of all users under the coverage of the edge server \(e_{i}\) into the neural network but also the number of remaining connections to \(e_{i}\). In AMARL, the input to the actor network is the respective state \(S_{i}\) of each agent, while in the critic network, since the training is to be centralized, the input is the set of states \(\mathcal{S}\) of each edge server \(e_{i}\), defined as shown in (5):  
\[\mathcal{S}= \begin{bmatrix}S_{11}&S_{12}&\cdots&S_{1|U_{1}|}&|C_{1}|\\ S_{21}&S_{22}&\cdots&S_{2|U_{2}|}&|C_{2}|\\ \vdots&\vdots&\ddots&\vdots&\vdots\\ S_{m1}&S_{m2}&\cdots&S_{m|U_{m}|}&|C_{m}|\end{bmatrix}  \tag{5}\]  
Because of the presence of other agents, the prediction of a single actor is not a global joint action. In order to reduce the impact of other agents on the current critic's evaluation of the actor's prediction, a masked array is used to eliminate useless information in the evaluation. The algorithm for state matrix construction is shown in Algorithm 1. **Action. ** For each agent, the action is initially set to a list of binary groups, where each element of the list is a binary group made up of the user index and the assigned video bitrate. Denote the action of \(e_{i}\) for \(u_{j}\) as \(A_{ij}\). Further, \(A_{ij}\) is a binary as shown in (6).