## 4 Proposed Solution
### Trajectory Optimization Algorithm Based on DDPG
#### 4.3.3 Reward Function  
We assume that the UAV observes the state \(S_{t}\) and takes action at the beginning of each time slot. Then, it transitions to state \(S_{t+1}\) based on \(K(s_{t},a_{t})\). \(S\times A\to R\) represents the expected immediate reward received by the UAV after transitioning from state \(S_{t}\) to \(S_{t+1}\) by taking action \(a^{k}_{t}\). For each UAV, we take into account coverage and fairness and define the real-time coverage efficiency as follows:  
\[\Delta\eta^{k}_{t}=f^{k}_{t}\sum_{k=1}^{K}\Delta c^{k}_{t} \tag{20}\]  
The reward function in this article consists of three parts. The first part is coverage efficiency, which includes two indicators: coverage score and fairness index, where \(\Delta c^{k}_{t}=c^{k}_{t}-c^{k}_{t-1}\).  
The second part defines the reward function that represents the relative energy consumption of the UAVs. Since SP-UAVs are used in the model of this paper, the relative energy consumption of the UAV swarm is made up of the energy consumed by the UAV's flight and the energy supplemented by solar power. In this paper, relative energy consumption is defined as the ratio of collected solar energy to flight energy consumption. Here, we  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Notation** & **Explanation** & **Value Range** \\ \hline \(x_{f}\) & horizontal x-coordinate of UAV & \(\left(0,a_{range}\right)\) \\ \hline \(y_{f}\) & horizontal y-coordinate of UAV & \(\left(0,a_{range}\right)\) \\ \hline \(z_{f}\) & horizontal z-coordinate of UAV & \(\left(h_{fmin},h_{fmax}\right)\) \\ \hline \(E^{f}_{remain}\) & remaining energy of UAV & \(\left(0,E_{max}\right)\) \\ \hline \(D^{k}_{cover}\) & user coverage & \(\left(0,D_{range}\right)\) \\ \hline \(E^{f}_{cost}\) & energy consumed by the UAV in the current time slot & \(\left(0,E_{max}\right)\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Explains the notations, explanations, and value ranges of each quantity in the state space.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Notation** & **Explanation** & **Value Range** \\ \hline \(\theta_{f}\) & the direction of the UAV’s horizontal movement & \(\left[-\pi,\pi\right]\) \\ \(d^{f}_{v}\) & the distance of the UAV’s horizontal movement & \(\left[0,v_{max}\Delta\tau\right]\) \\ \(d^{f}_{h}\) & the distance of the UAV’s vertical movement & \(\left[0,v_{max}\Delta\tau\right]\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Explains the notations, explanations, and value ranges of each quantity in the action space.  
consider the overall relative energy consumption, because using the relative energy consumption of a single UAV instead of the overall relative energy consumption in the reward function may lead to a situation where one UAV has a high relative energy consumption while the relative energy consumptions of other UAVs are low. This will cause one UAV to quickly run out of power. Considering the overall relative energy consumption, however, will result in the remaining energy consumption of each UAV being more balanced, thereby extending the lifetimes of the UAVs.  
The third part sets a penalty term \(\rho_{t}^{f}\), where \(\rho_{t}^{f}=0\) when the UAV is flying within the target area. When the UAV flies out of the target area, \(\rho_{t}^{f}\) will be equal to a constant V, and the reward obtained for this action will decrease accordingly, encouraging the UAV to avoid actions that will cause it to fly out of the target area insofar as possible. It is worth noting that we do not set penalty terms for UAV collisions or disconnections, because these two situations are not tolerable in the context of this paper. Once a UAV collides or loses connection, the flight cycle of the UAV swarm will immediately stop, and the lifetime will be limited to the current total number of time slots. Therefore, the obtained reward and penalty functions are as follows:  
\[R_{t}=\Delta\eta_{t}^{k}+\frac{E_{solar}^{D}}{E_{f}^{D}}-\rho_{t}^{f},\;\forall t \tag{21}\]