# Machine-Learning-Based Optimization for Multiple-IRS-Aided Communication System
## 4 Proposed Algorithm for Dynamic UE Scenario
### The Proposed DDPG-Based Algorithm
#### 4.2.1 Description of the Proposed DDPG-Based Algorithm  
Based on the aforementioned structure of the DDPG, we propose a DDPG-based algorithm to solve the optimization problem \((P1)\) in the dynamic UE scenario, as summarized in Algorithm 1. At the start, the main and target networks are initialized; specifically, the four training and targeting networks are initialized with uniformly distributed parameters \(\theta_{\mu}=\theta_{\mu^{\prime}}\), and \(\theta_{\mu}=\theta_{q^{\prime}}\). Additionally, all model hyper-parameters for training are initialized. Then, at the training stage, the agent's interactions in the surrounding environment are split into \(I\) episodes with finite \(T\) steps in each episode. During each training episode, the environment is reset to generate new UEs' positions and channel gains. Then, all direct and indirect channels between network nodes are calculated, and the beamforming vectors are randomly initialized. Accordingly, all the transmission and received powers are calculated; the state \(s_{t}\) is formulated by (18) and inputs as excitation to the actor, which feeds back the decision \(a_{t}\) by applying the current policy and adding generated sampled noise \(\mathcal{N}_{t}\), extracting the BS active beamforming and the deployed IRSs' passive beamforming from this action, whereby the current reward \(r_{t}\) is calculated by (28) and the next state \(s_{t+1}\) is formulated. Then, storing the transition \(\{s_{t},a_{t},r_{t},s_{t+1}\}\), the experience buffer is created. The algorithm needs at least \(N_{B}\) experience to start updating and learning networks; thus, the previous steps are repeated \(N_{B}\) times to form the starting mini-batch only in the first episode. Once the \(N_{B}\) experience is created, the algorithm will start training. At each time step \(t\), a random \(N_{B}\) sample is chosen from the replay. Accordingly, the target value function is calculated by the target network as the sum of the instantaneous reward and the discounted future reward as  
\[y_{t}=r_{t}+\gamma\ Q^{\prime}(s_{t+1},\mu^{\prime}(s_{t+1};\theta_{\mu^{ \prime}});\theta_  
```
1: Initialize the training networks for actor and critic \(\mu(s;\theta_{\mu})\) and \(Q(s,a;\theta_{q})\), with initial parameters \(\theta_{\mu}\) and \(\theta_{q}\), respectively. 2: Initialize the target networks for actor and critic \(\mu^{\prime}(s;\theta_{\mu^{\prime}})\) and \(Q^{\prime}(s,a;\theta_{q^{\prime}})\) with copied parameters as \(\theta_{\mu^{\prime}}\leftarrow\theta_{\mu}\) and \(\theta_{q^{\prime}}\leftarrow\theta_{q}\), respectively. 3: Initialize actor and critic learning rates \(\mu_{a}\) and \(\mu_{c}\), an empty experience replay \(D\), mini-batch size \(N_{B}\), target smooth factor \(\tau\), and discount factor \(\gamma\). 4:for episode \(i=1,2,\ldots I\)do
5: Randomly set \((x_{k},y_{k},z_{k}),\forall k\), obtain \(\{\mathbf{G}_{i}\}_{l=1}^{L}\), \(\{\mathbf{H}_{rl}\}_{l=1}^{L}\),\(\mathbf{H}_{d},\ \forall l,k\). 6: Initialize \(\{\theta_{l}\}_{l=1}^{L}\), and \(\mathbf{W}\). 7: Form initial state \(s_{1}\) using (18). 8:for t=1,2,...T do
9: Select an action as \(a_{t}=\mu(s_{t};\theta_{\mu})+\mathcal{N}_{t}\). 10: Form the BS beamforing \(\mathbf{W}\) and IRS beamforming matrices \(\{\mathbf{\Theta}_{l}\}_{l=1}^{L}\) from \(a_{t}\). 11: Obtain instant reward \(r_{t}\) using (28). 12: Obtain next state \(s_{t+1}\) using (18). 13: Store the experience \(\{s_{t},a_{t},r_{t},s_{t+1}\}\) in buffer \(D\). 14:if update then
15: Sample a random mini-batch of \(N_{B}\) experiences \(\{s_{t},a_{t},r_{t},s_{t+1}\}\) from the buffer. 16: Set the value function target as \[y_{t}= \begin{cases}r_{t},&\text{if $s_{t+1}$ is terminal state,}\\ r_{t}+\gamma\ Q^{\prime}(s_{t+1},\mu^{\prime}(s_{t+1};\theta_{\mu^{\prime}}); \theta_{q^{\prime}}),&\text{otherwise.}\end{cases} \] (28)
17: Update the main critic parameters by minimizing the loss in (24). 18: Update the main actor parameters by sampled policy gradient in (25).