## 3 Problem Formulation and Model Design
### Model Design
\[A_{ij}=(i,b_{i}(u_{j})) \tag{6}\]  
Thus, the action vector \(A_{i}\) predicted by the edge server \(e_{i}\) is  
\[A_{i}=(A_{i1},A_{i2},\cdots,A_{i|U_{i}|}) \tag{7}\]  
However, it is a challenge to determine the actions because each agent will have access to a list of all user behaviors, requiring several decisions to be made about each user by various agents, which will certainly result in some conflict. Section 3.3 describes the detailed solution. **Reward. ** In the EUA problem, the goal is to maximize the number of assigned users and minimize the number of leased servers. However, if only the user's QoE and the number of leased servers are considered as the reward function, the capacity of each agent will be ignored. Therefore, a penalty function is introduced to prevent the number of server connections from exceeding their own capacity. Denote \(R_{i}\) as the reward of edge server \(e_{i}\) and \(EN\) as the number of leased servers; then  
\[R_{i}=\sum_{u_{j}\in C_{i}}QoE(u_{j})+ap(e_{i})-EN \tag{8}\]  
Furthermore, in order to serve as many users as feasible within the coverage area, the penalty function \(ap(\cdot)\) is defined in accordance with the number of connections and the number of users inside the coverage area as follows:  
\[ap(e_{i})= \begin{cases}Con_{i}-|C_{i}|,&|C_{i}|>Con_{i}\\ |C_{i}|,&|C_{i}|\leq Con_{i}\end{cases}  \tag{9}\]  
**Actor. ** For each UAV agent, there is an actor and a critic. The actor is the network parameterized policy function \(\mu(s_{i}^{j};\theta^{\mu})\); \(\theta^{\mu}\) is the parameter of the network. The action of the agent can be obtained by the deterministic policy \(a_{t}=\mu(s_{i}^{j};\theta^{\mu})\). After all agentscomplete the action selection, the joint action \(\mathbf{a}_{t}\) can be obtained. The gradient can be calculated as  
\[\
abla_{\theta_{i}}J=\frac{1}{S}\sum_{j}\
abla_{\theta_{i}}\mu_{i}(s^{j}_{i}) \
abla_{a_{i}}Q^{\psi}_{i}(\mathcal{S}^{j},a^{j}_{1^{\prime}},\cdots,a_{i^{ \prime}},\cdots,a^{j}_{N})|_{a_{i}=\mu_{i}(s^{j}_{i})} \tag{10}\]  
where \(i\) is the agent index, \(S\) is the batch size, \(\mu\) is the actor policy, and \(Q\) is the critic evaluation. **Critic. ** In order to more accurately evaluate the action, each agent queries other agents for information about its observations and actions and integrates it into the estimate of its value function. To compute the Q-value function \(Q^{\psi}_{i}(s,a)\) for \(agent_{i}\), the critic receives the states \(s=(s_{1},\ldots,s_{N})\) and actions \(a=(a_{1},\ldots,a_{N});i\in 1...N\). \(Q^{\psi}_{i}(s,a)\) is a function of states and actions of \(agent_{i}\) and the contributions of other agents:  
\[Q^{\psi}_{i}(s,a)=f_{i}(g_{i}(s_{i},a_{i}),x_{i}) \tag{11}\]  
where \(f_{i}\) is a two-layer MLP, while \(g_{i}\) is a one-layer MLP embedding function. Denote the contribution from other agents as \(x_{i}\), which is a weighted sum of each agent's value:  
\[x_{i}=\sum_{j\
eq i}\alpha_{j}v_{j} \tag{12}\]  
where the value, \(v_{j}\) is the embedding of \(agent_{j}\). The attention weight \(\alpha_{j}\) considers the correlation of \(agent_{i}\) and \(agent_{j}\); that is, it is calculated according to the distance between the two agents and the overlap rate of users in the service area. The details are shown in Figure 3.  
As shown in Figure 2, the critics of each agent can obtain the state and action information of other agents extracted by the attention mechanism during training for an objective action evaluation. After evaluation, each agent independently predicts the next action based on the evaluation value. Figure 3: Calculating \(Q^{\psi}_{i}(s,a)\) for \(agent_{i}\). Each agent encodes its observations and actions, sends them to the central attention mechanism, and receives a weighted sum of other agents’ encodings.