# DDPG-Based UAV-RIS Framework for Optimizing Mobility in Future Wireless Communication Networks
## 3 Problem Formulation
### Joint Optimization of UAV Location and RIS Phase Shifts  
To jointly optimize the UAV location and RIS phase shift in mmWave networks, a DRL-DDPG algorithm is proposed to address the formulated problem of finding the optimal UAV location and adjusting RIS phase shifts. The proposed DRL framework consists of a state set \(s(t)\), an action set \(a(t)\), a reward set \(r(t)\), and UAV-RIS as an agent by the DDPG algorithm [36]. \(s(t)\), \(a(t)\), and \(r(t)\) are clearly described as follows:  
**State Space:** The set of spaces, including the UAV's optimal location and RIS phase shifts, at time \(t-1\) is described as  
\[\mathbf{s_{t}}=\left[\frac{\theta_{1}^{(t-1)},\ldots,\theta_{ZF}^{(t-1)}}{\text{ RIS phase shift}}\right|\frac{x^{(t-1)},y^{(t-1)},h^{(t-1)}}{\text{UAV optimal location}}\Bigg{]} \tag{18}\]  
**Action Space:** The action space includes a UAV movement and RIS phase shifts when transitioning from the current to the next state. The suggested approach enables the agent to continuously determine the optimal movement while considering the long-term reward and identify the optimal phase shift for each time instance. The agent (UAV-RIS) inputs the state \(s_{t}\) at time step \(t\) to determine the appropriate action based on the current environment,resulting in the optimal UAV horizontal location and updated RIS phase shift to improve connectivity and mobility issues in urban environments. The action space is expressed as  
\[\mathbf{a_{t}}=\left[\theta_{1}^{(t)},\ldots.\theta_{2F}^{(t)},\mathbf{x}^{(t)},y^{(t)}, h^{(t)}\right] \tag{19}\]  
**Reward:** After performing action \(a_{t}\) in state \(s_{t}\) at time \(t\), the agent obtains a reward \(\mathbf{r_{t}(s_{t},a_{t})}\). Based on the objective of the paper, the sum rate per user group describes the reward and can be written as  
\[\mathbf{r_{t}}:\Theta_{sum}^{(t)}=\sum_{u=1}^{U}\Theta_{u}^{(t)},u=1,\ldots,U \tag{20}\]  
The DDPG algorithm aims to determine the optimal action that maximizes the Q-value, assessing state-action pair quality to maximize the expected commutative reward under an optimal policy \(\pi\). This approach facilitates the assessment of the agent's actions and state transitions based on the given state \((s_{t})\), action \((a_{t})\), and reward \((r_{t})\). The proposed DDPG algorithm adaptively optimizes the UAV location and RIS phase shifts (actions) in dynamic mmWave urban dense environments for achieving improved signal quality (rewards). This is accomplished by training two main networks, i.e., actor network and critic network. The actor network suggests actions, while the critic network evaluates those actions to enhance overall network efficiency and user experience in FWNs. Algorithm 2 summarizes the proposed DDPD algorithm, while Figure 2 depicts the flow diagram of the proposed scheme for optimizing the UAV location and RIS phase shift to achieve improved network performance. The flow diagram illustrates the interaction between the UAV-RIS system and the DDPG learning agent. The UAV-RIS observes environmental states, including user positions and channel conditions, and takes joint actions of UAV positioning and RIS phase adjustments. These actions yield corresponding rewards based on system performance. The agent stores transition tuples \((s_{t},a_{t},r_{t},s_{t+1})\) into the experience replay buffer, which are sampled as mini-batches to train the actor and critic networks. The critic network evaluates the Q-value of actions, while the actor network updates its policy through gradient ascent. Target networks are softly updated to stabilize learning. The step-by-step explanation of the Algorithm 2 is as follows:  
The algorithm initializes several key components, including a replay buffer \(D\) for storing learning experiences, a discount factor \(\gamma\) to weigh future rewards in the learning process, and a soft update coefficient \(\tau\) for updating target networks, and it sets the minibatch size \(N_{B}\) for determining sample transitions from \(D\) during training.