## 6 Solutions to Optimization Problems
### Machine (Deep) Learning  
Machine learning and AI-assisted techniques have gained increased attention for handling the nonlinearity, nonconvexity, and computational complexity of complex engineering problems [169]. In addition to the nonconvex formulation associated with UAV-enabled, RIS-assisted communication and trajectory optimization, challenges such as imperfect CSI, dynamic environments, and uncertain channel conditions further complicate the problem. As a result, machine learning techniques have been increasingly considered in the wireless communication community. Aiming to improve the performance of RISeUAV communication channels, the authors in [170] proposed a machine learning-based beamforming policy for UAV-RIS by employing prioritized experience replay (PER) based deep Q-Network (DQN). Further, they used a DDPG-based beamforming approach for secure UAV-RIS beamforming under imperfect channel estimation [171]. Deep Reinforcement Learning (DRL)  
Among AI-based machine learning techniques, DRL is the most popular one in the community, see Table 1. DRL works based on Markov decision process (MDP) and dynamic programming for estimating future reward in the MDP for training DQN [136]. The most common DRL agent is the actor-critic deep deterministic policy gradient (DDPG) agent. A schematic of the DDPG DRL agent including actor-critic deep networks is illustrated in Figure 7. Based on the MDP, the actor network \(\pi_{\phi_{A}}\) receives the current state \(\mathcal{S}[k]\) (e.g., positions of the RISeUAV and IoVs) and takes action \(\mathcal{A}[k]=\pi_{\phi_{A}}(\mathcal{S}[k])\) in the permitted action space (e.g., the action may include UAV motion speed inputs or motion primitives as well as RIS-PhSh). Based on the taken action \(\mathcal{A}[k]\), the rewards \(\mathcal{R}[k]\) and new state \(\mathcal{S}[k+1]\) are determined based on the simulated environment. For example, the RISeUAV takes the given action and moves through the path or generated trajectory. Then, the channel gain and communication performance indexes are obtained to calculate the reward. The new state \(\mathcal{S}[k+1]\) is the new positions of the RISeUAV and IoVs. The critic networks \(Q_{\phi_{c}}(\mathcal{S}[k],\mathcal{A}[k])\), is embedded to predict future rewards, and is trained by minimizing the error function of the calculated actual reward \(\mathcal{R}[k]\) and its estimated rewards \(Q_{\phi_{c}}(\mathcal{S}[k+1],\pi_{\phi_{A}}(\mathcal{S}[k+1]))\). Therefore, critic \(Q_{\phi_{c}}(\mathcal{S},A)\) is a Q-value function based on the recursive Bellman equation:  
\[Q_{\phi_{c}}(\mathcal{S}[k],\mathcal{A}[k])=\mathbb{E}\big{[}\mathcal{R}[k]+ \gamma Q_{\phi_{c}}\big{(}\mathcal{S}[k+1],\pi_{\phi_{A}}(\mathcal{S}[k+1]) \big{)}\big{]} \tag{31}\]  
where \(\gamma\) is the discount factor for the future award, and \(\mathbb{E}\) denotes expectation. The critic parameters \(\phi_{c}\) are updated by minimizing the loss function \(L\) given by all sampled experiences out of \(M\) random mini-batch of experiences (trial-and-error episodes)  
\[L=\frac{1}{M}\sum_{i=1}^{M}\big{(}\textit{y}_{i}[k]-Q_{\phi_{c}}(\mathcal{S}[ k],\mathcal{A}_{i}[k])\big{)}^{2} \tag{32}\]  
where  
\[ \begin{split}&\textit{y}[k]=\mathcal{R}[k]+\gamma\mathbb{E}( \mathcal{R}[k+1])\\ &=\mathcal{R}[k]+\gamma Q_{\phi_{M}}\big{(}\mathcal{S}[k+1],\pi_{ \phi_{M}}(\mathcal{S}[k+1])\big{)};\end{split}  \tag{33}\]  
The actor parameters \(\phi_{A}\) are updated using the sampled policy gradient by applying the chain rule to the expected return that maximizes the expected discounted reward:  
\[\begin{split}&\
abla_{\phi_{A}}\However, the design of the DRL structure, including the deep networks and the architecture of the actor-critic, the method of observing the system state, the action-taking process, reward functions, and the training algorithm of the deep neural network, all significantly impact the performance of the DRL agents.