# PPO-Based Joint Optimization for UAV-Assisted Edge Computing Networks
## 4 PPO-Based Joint Optimization Algorithm  
We provided the optimization problem for joint UE-ECS matching and deployment of UAVs with three hovering positions in the previous section. However, it can be observed by looking at the problem model that the problem is a complex mixed integer nonlinear programming problem, which is usually difficult to solve by traditional optimization methods. Fortunately, the advancement of artificial intelligence has introduced novel research techniques for mobile edge computing. PPO has gained widespread acceptance as a reinforcement learning algorithm due to its stability and reliability. In contrast to traditional policy gradient algorithms and other RL algorithms based on the actor-critic framework, PPO incorporates an older actor network to constrain the variance of the new policy. This approach effectively mitigates instability issues that could potentially arise in other algorithms. Secondly, the PPO algorithm can consistently perform well in various environments and can effectively suit the UAV-assisted edge computing environment that is being studied. Hence, in this work, we consider designing solution algorithms for joint optimization based on the PPO framework. Specific algorithmic details are given next. Note that deep reinforcement learning frameworks, including PPO, can usually only handle discrete or continuous decision problems. However, the joint optimization problem in this work is mixed integer programming with both discrete and continuous variables, which is difficult to handle for PPO. Therefore, we first introduce matching factors to convert the original mixed integer programming into a continuous variable problem. We design \(M\) matching factors for each UE, which represent the matching priority of each of the \(M\) ECSs at that UE. The value range of the matching factors is \([0,1]\). Following this design principle, the matching factors of UE \(k\) are \(o_{1}^{k},o_{2}^{k},\cdots,o_{M}^{k}\); then, the ECS matched by UE \(k\) is the one with the largest matching factor, denoted by \[\arg\max\left(o_{1}^{k},o_{2}^{k},\cdots,o_{M}^{k}\right).\] (15) So far, the original integer variable \(\mathbf{u}\) is transformed into a continuous variable, i.e., the matching factor, and the original problem is transformed into a continuous-variable decision process that can be solved by the PPO implementation. We first model the problem as a Markov decision process, where the state space, action space, state transfer function, and reward function are described as follows. * _State space_: the environment is described to the agent body by the state space, which needs to provide enough information to make the agent body take appropriate actions. For the considered UAV-assisted edge computing network, the hovering position of the UAV determines the sending delay, while the matching of the UE and the ECS determines the processing delay and the waiting delay. Therefore, the state space designed in this work contains the following points: the three-dimensional hovering position of the UAV and the matching factor at all UEs. The state space is defined as \(\mathcal{S}\), and the state at time step \(t\) is defined as \(s_{t}\). * _Action space_: based on the decision variables and state space in the problem model, we design the action space for the problem. The action space also contains two parts: one is the displacement of the UAV in three-dimensional space, and the other is the amount of change of the matching factor. The action space is defined as \(\mathcal{A}\), and the state at time step \(t\) is defined as \(a_{t}\). * _State transfer probability function_: according to the state and action of the agent body designed above, the next state of the agent body can be obtained from its current state plus the action, i.e., the current three-dimensional position of the UAV base station plus the displacement to the next three-dimensional position. Therefore, the state transfer probability function is defined as \[\mathcal{P}(s_{t+1}|s_{t},a_{t})=\left\{