# PPO-Based Joint Optimization for UAV-Assisted Edge Computing Networks
## 4 PPO-Based Joint Optimization Algorithm
\begin{array}{c}1,s_{t+1}=s_{t}+a_{t},s_{t}\in\mathcal{S},a_{t}\in\mathcal{A}\\ 0,\text{others}\end{array} \right..\] (16)
* _Reward function_: the designed PPO-based joint optimization algorithm aims to minimize the average delay of the UAV-assisted edge computing network. Therefore, the average delay is designed as the body of the reward function. Meanwhile, in order to make the agent body transfer minimize the average delay, the average delay exists in the form of the opposite number in the reward function. Specifically, the reward function is expressed as \[r_{t}=p\left(q-\left(\frac{1}{K}\sum_{k=1}^{K}Delay_{total}^{k}|t\right)\right),\] (17) where \(p\) and \(q\) are constant terms to adjust the reward value to within a suitable range. Based on the modeling of the Markov decision process, the agent body can perform actions to obtain the reward value and complete the state transfer. The agent body in the PPO algorithm collects the state transfer trajectories for every \(T\) time steps, then performs a round of updating. Each round of updating uses the group of trajectory data to update \(D\) times, and the object of the updating is the parameter \(\theta\) of the Actor network and the parameter \(\phi\) of the Critic network. The role of the Actor network is to fit the agent's policy \(\pi_{\theta}(a|s)\), representing the policy in the form of a Gaussian distribution, which can be fully described by its mean \(\mu\) and standard deviation \(\sigma\). Then, the Actor network completes the state to \((\mu,\sigma)\) mapping. When the intelligence needs to take an action, it first recovers that Gaussian distribution through \((\mu,\sigma)\) and then randomly samples to pick an action. The PPO, in order to keep the old policy unchanged in each round of update, initializes a network with the same structure and parameters as the Actor network, called Actor-old. The Actor-old network does not participate in training and only copies parameters from the Actor network before each round of updates to keep the old strategy \(\pi_{\theta_{k}}\) for the current round; then, the Actor network can continue searching for the new strategy \(\pi_{\theta_{k+1}}\). The Actor network is trained to maximize the agent objective function, whose parameters are given by the following equation update  
\[\theta_{k+1}=\operatorname*{arg\,max}_{\theta}\frac{1}{T}\sum_{t=0}^{T} \min\biggl{(}\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{k}}(a_{t}|s_{t})}A _{t}^{\pi_{\theta_{k}}},\text{clip}\biggl{(}\frac{\pi_{\theta}(a_{t}|s_{t})}{ \pi_{\theta_{k}}(a_{t}|s_{t})},1-\varepsilon,1+\varepsilon\biggr{)}A_{t}^{\pi _{\theta_{k}}}\biggr{)}, \tag{18}\]  
where \(k\) denotes the first round of update and \(A_{t}^{\pi_{\theta_{k}}}\) is the dominance function calculated based on the state value estimated by the Critic network and the temporal difference method, where \(R_{t}\) and \(V(s_{t})\) are the reward value of the time step and the state value of \(s_{t}\), respectively. The Critic network completes the estimation of the state value \(V(s)\), and its training objective is to minimize the loss function based on the mean square error. Then, its parameter \(\phi\) is updated by the following equation:  
\[\phi_{k+1}=\operatorname*{arg\,min}_{\phi}\frac{1}{T}\sum_{t=0}^{T}\Big{(}V_{ \phi}(s_{t})-\widehat{R}_{t}\Big{)}^{2}, \tag{19}\]  
where \(\widehat{R}_{t}\) is the accumulated reward computed from the state transfer trajectory data. Based on the above neural network design, the basic framework of PPO is built. According to the above description, both the main framework and Markov decision process of the PPO algorithm have been designed. Next, we reduce the average delay of the network by jointly optimizing the UE-ECS matching and the UAV three-dimensional hovering position. The designed optimization algorithm is called the PPO-based joint optimization algorithm, and the detailed algorithm flow is shown in Algorithm 1.