## 4 Methodology
### Solution  
The framework of DCMCE is shown in Figure 2, including actor network, critic network, environment and replay buffer. The network structure adopts DNN architecture with shared parameters, and the overall objective is to merge the critic network and actor network's error terms. The overall maximization objective function can be written as Formula (24). \[L^{PPO}(\theta)=\mathbb{E}_{n}\Big{[}L^{C}_{n}(\theta)-\epsilon L^{V}_{n}( \theta)\Big{]} \tag{24}\]  
First, we utilize DCMCE to calculate the UAV's trajectory for time slot t. Since DCMCE is dependent on DRL, let us first describe the three components of DRL: status, action, and reward. Figure 2: The structure of DCMCE algorithm. State;  
\[s(t)=(E_{\text{b}}\ (t),q(t),u_{1}(t)\dots u_{M}(t),w_{1}(t)\dots w_{M}(t)) \tag{25}\]  
\(E_{\text{b}}\ (t)\) reflects the battery's remaining power in the time slot \(t\), \(q(t)\) represents the UAV's location information, \(u_{k}(t)\) represents the position of the \(k\)-th user and \(w_{k}(t)\) shows whether the \(k\)-th user's signal is being obstructed by objects. Action;  
\[a(t)=(\sigma(t),v(t)) \tag{26}\]  
The UAV's flight angle and flight speed can be precisely optimized in the continuous action space. \(\sigma(t)\) represents the UAV's flight angle and \(v(t)\) represents the UAV's flight speed, where \(\sigma(t)\in[0,\ 2\pi],v(t)\in[0,\ v_{max}]\). Reward;  
\[r(t)=\eta(t)-\zeta \tag{27}\]  
where \(\eta(t)\) is the computational efficiency at time \(t\). It is obtained through the Formula (10), where \(\zeta\) is the penalty obtained by flying out of the active area. The details of the PPO-based DCMCE algorithm proposed in this paper are given in Algorithm 1. We initialize two DNNs with the same parameters (\(\theta_{\text{old}}\ \leftarrow\theta\)) and \(\pi_{\theta_{\text{old}}}\) for sampling, and \(\pi_{\theta}\) for optimization. The algorithm alternates between sampling (lines 6 to 10) and optimization (lines 12 to 14). In the sampling phase, the UAV performs action \(a_{t}\) to move to a certain position, and then performs Algorithm 2 to calculate the priority function to obtain the best user association and reach the next state \(s_{t+1}\) and calculate the current reward. After repeating the process for \(D\) times, trajectories \((s_{0},a_{0},r_{0},s_{1},\dots,s_{\text{D}})\) follow the old strategy \(\pi_{\theta_{\text{old}}}\) for sampling. At this stage, GAEs are computed in advance to increase training efficiency. Data from samples are cached for efficiency. The policy's parameters are changed in \(D\) epochs during the optimization phase. The approach uses a stochastic gradient ascent on the cached sampled data, based on the objective function to enhance the strategy once every \(D\) time slots. The sample strategy \(\pi_{\theta_{\text{old}}}\) is updated with the current \(\pi_{\theta}\) and the cached data are removed at the conclusion of the optimization phase. Then, the next iteration begins. ```
1. Randomly initialize the parameters \(\theta\) and \(\theta_{old}\)
2. Initialize the collection \(\mathcal{R}\), sample size \(\mathcal{D}\)
3.for\(\text{e}=1,2,\dots,\ Z\)
4. Initialize \(\text{S}_{t}\)
5.for\(t=1,2,3,\dots,N\)
6. Use \(s\) as the input of DNN to obtain the \(a_{t}\)
7. Execute action \(a_{t}\), obtain state \(s_{t+1}\)
8. Obtain user association through Algorithm 2
9. Calculate reward \(r_{t}\) through the Formula (27)
10. Store the transition \((s_{t},a_{t},r_{t})\) in \(\mathcal{R}\)
11.if\(\%D==0\) or t ==T
12. Compute advantage function estimates \(\hat{A}_{n}^{GAE(\gamma,\theta)}\)
13. Update the policy according to the objective function \(\theta\leftarrow\underset{\theta}{\text{argmax}}\underset



After the trajectory of the UAV is determined, we need to obtain the optimal user association to maximize energy efficiency. Algorithm 2 summarizes the details of the association algorithm based on Dinkelbach.