## 5 Proposed Solution
### Reward Function  
We consider the occupancy of the UAV processor queue as a cost function to define the UAV overall reward function (\(r_{i}(\mathbf{s}_{t},\mathbf{a}_{t})\)) in each TTI as follows in Equation (29):  
\[r_{i}(\mathbf{s}_{t},\mathbf{a}_{t})=\gamma_{1}r_{i,i\
eq}(t)+\gamma_{2}r_{i, \mathcal{D}}(t)+\gamma_{3}r_{i,s}(t)+\gamma_{4}r_{i,\mathcal{D}}(t) \tag{29}\]  
where \(\gamma\) is a discount factor, \(r_{i,\mathcal{D}}(t)\) is the reward that achieves optimal model transmission from the vehicles to the UAV. The term \(r_{i,\mathcal{D}}(t)\) is the reward that achieves minimal delay. The term \(r_{i,s}(t)\) is the reward that achieves the maximum data rates, and \(r_{i,\mathcal{D}}(t)\) is the reward that achieves the maximum power utilization for UAV. The expectation (\(\mathbb{E}\)) is taken over all samples in the experience replay buffer and constitutes the global reward \(r_{i}(\mathbf{s}_{t},\mathbf{a}_{t})\). Based on the past UAV trajectories, the actor-critic LSTM provides model-free prediction based on the existing samples for action exploration. Consequently, it guides the UAV trajectory towards a more rewarding policy. In fed-DDPG, each vehicle learns a local model based on its data samples and uploads the model to the UAV to maximize the long-term discounted reward in Equation (30):  
\[r_{\pi_{i}}=\mathbb{E}_{\mathbf{s},\mathbf{a}\sim\mathbb{E}}[\sum_{t=0}^{ \infty}\gamma_{t}r_{i,t}(s,a_{1,t},\ldots,a_{i,t},\ldots,a_{I,t})] \tag{30}\]  
where \(a_{I,t}\) is the UAV action in each TTI. When trained on the _LTE I/Q_ dataset, the LSTM estimates an optimal coordinate in the next TTI based on the recent trajectory. The action estimation is input to the critic LSTM together with the action learned by the actor LSTM. The critic LSTM evaluates the two actions, and the critic decides an action to be executed. The UAV selects rewarding action with a higher probability to reduce the action space and improve the learning efficiency to map each coordinate to the data size collected from the vehicles to minimize the MSE. The UAV acts as an independent DRL agent, where the policy parameter outputs its action using the deterministic policy based on its observation of the channel states, where \(s\) is the channel state, \(a\) is the action of UAV, and \(\mathbb{E}\) refers to the expectation in the C-V2X environment. The gradient is calculated as per Equation (31) as  
\[\
abla_{\phi_{i}}r(\pi_{i})=\mathbb{E}_{\mathbf{s},\mathbf{a}\sim\mathbb{E}} \
abla_{\phi_{i}}\log\bigl{(}\pi_{\phi_{i}}(a_{i}\mid s_{i})\bigr{)}+e(s_{i},a _{i}) \tag{31}\]  
where \(e(s_{i},a_{i})\) is the entropy. The trade-off between maximizing entropy and reward is determined by Equation (32):  
\[Q_{i}^{\psi}(s,a)\gets Q_{i}^{\psi}(s,a)+\beta[r+\gamma\max_{a}Q_{i}^{ \psi}\bigl{(}s,a^{\prime}\bigr{)}-Q_{i}^{\psi}(s,a)] \tag{32}\]  
where \(Q_{i}^{\psi}\) is the actor-critic Q-function, \(\beta\in\) (0,1] is the learning rate, and \(r\) denotes the reward. As illustrated in Figure 5, we assume a random trajectory as well as an elliptical trajectory to model the UAV path. The elliptical trajectory leads to less MSE compared to the random trajectory, as the future coordinates can be accurately predicted. As per Equations (33) and (34), the reward is recursively updated, and the updated reward is used to calculate the loss function using MSE.  
\[r_{i}=r_{i-1}+\gamma E_{\mathbf{a}^{\prime}\sim\pi_{\mathcal{D}} (\mathbf{s}^{\prime})}\Bigl{[}Q_{i}^{\psi^{\prime}}(\mathbf{s},\mathbf{a})+ e(\mathbf{s}_{i},a_{i})\Bigr{]} \tag{33}\] \[\min f_{\mathcal{L}_{\mathcal{D}}}(\psi)=\sum_{i=1}^{1}\mathbb{E} _{(\mathbf{s},\mathbf{a},\mathbf{r},\mathbf{s}^{\prime})\sim\mathbb{E}}[(Q_{ i}^{\psi}(\mathbf{s},\mathbf{a})-r_{i})^{2}] \tag{34}\]Based on the available battery power (\(\mathcal{P}_{b}\)), the UAV adjusts its height (\(\mathcal{H}\)). For each height, the number of vehicles covered is depicted by a triangle in Figure 5.