## 4 PPO-Based Optimization of the UAV Trajectory
### PPO Algorithm  
In this section, we present the proposed PPO learning algorithm for UAV trajectory design, and the steps are written as pseudocode in Algorithm 2. PPO is an on-policy, model-free reinforcement learning algorithm in which decision-making is based on data collected from the most up-to-date policy. The goal of the PPO model is to enable the agent to execute optimal actions that maximize long-term cumulative rewards. Consequently,  
Figure 4: The PPO network.  
in the PPO model, the selected action may not be the optimal choice for the current time slot, but it aims to be the optimal choice for pursuing long-term benefits. The central controller, which controls the UAV, acts as the agent in this process.  
At each time slot, the agent observes a state from the state space \(S\), which consists of the coordinates of the RIS-equipped UAV. The action space \(A\) represents the set of available actions, the agent based on the current state and decided by stochastic policy \(\pi_{v_{k}}\), where the actions decided by policy rules can be denoted as \(a_{t}\). After executing the actions, the agent receives a reward determined by the average capacity under the current FSO connectivity condition.  
In order to adjust the surrogate objective problem where the huge ratio can be denoted as  
\[r_{t}(v_{k})=\frac{\pi_{v_{k+1}}\big{(}a_{t}|s_{t}\big{)}}{\pi_{v_{k}}\big{(}a _{t}|s_{t}\big{)}}, \tag{21}\]  
where \(v_{k}\) is the policy parameter that includes the operation movement rules established for the UAV. Between the current policy \(\pi_{v}\) and new policy \(\pi_{v_{k+1}}\), via the PPO-clip update policy  
\[v_{k+1}=\text{arg max }L^{clip}(v_{k}), \tag{22}\]  
to maximize the clipped surrogate objective \(L^{clip}(v_{k})\), which can be denoted as  
\[L^{clip}(v_{k})=\hat{E}_{t}\bigg{[}\min\bigg{(}r_{t}(v_{k})\hat{D}_{t}, \tag{23}\] \[\text{clip}(r_{t}(v_{k}),1-\varrho,1+\varrho)\hat{D}_{t}\bigg{)} \bigg{]},\]  
where the clip function can be denoted as  
\[\text{clip}(\varrho,\hat{D}_{t})=\begin{cases}1+\varrho&\hat{D}_{t}\geq 0\\ 1-\varrho&\hat{D}_{t}<0\end{cases} \tag{24}\]  
and \(\varrho\) is a hyperparameter, \(\hat{E}_{t}\) is the estimator of the advantage function \(\hat{D}_{t}\). The conventional update policy between two iterations relies on the Monte Carlo approximation, where the surrogate objective is maximized, denoted as  
\[v_{k+1}=\text{arg max }L(v_{k}). \tag{25}\]  
When the value of advantage \(\hat{D}_{t}\) for the state-action pair is positive, the objective reduces to  
\[L^{pos}(v_{k})=\hat{E}_{t}\bigg{[}\min\bigg{(}\frac{\pi_{v_{k+1}}\big{(}a_{t} |s_{t}\big{)}}{\pi_{v_{k}}\big{(}a_{t}|s_{t}\big{)}},(1+\varrho)\bigg{)}\hat{D }_{t}\bigg{]}, \tag{26}\]  
when the value of advantage \(\hat{D}_{t}\) for the state-action pair is negative, the objective reduces to  
\[L^{neg}(v_{k})=\hat{E}_{t}\bigg{[}\max\bigg{(}\frac{\pi_{v_{k+1}}\big{(}a_{t} |s_{t}\big{)}}{\pi_{v_{k}}\big{(}a_{t}|s_{t}\big{)}},(1-\varrho)\bigg{)}\hat{D }_{t}\bigg{]}. \tag{27}\]  
After updating the policy by maximizing the PPO-clip objective, the value function needs to be fitted by regression on mean-squared error.