# Deep Reinforcement Learning for RIS-Aided Multiuser MISO System with Hardware Impairments
## 3 Proposed Algorithm
### DDPG Algorithm
The network parameter \(w^{(t)^{\prime}}\) is periodically copied from \(w^{(t)}\), which uses the soft update method, and the soft update factor is \(\tau\) : \[w^{(t+1)}=\tau w^{(t)}+(1-\tau)w^{(t+1)}.\] At the same time, to increase some randomness and increase the coverage of learning in the learning process, the DDPG algorithm adds some noise \(\mathcal{N}\) to the selected action. A. That is, the expression of the final and interactive action A of the environment is \[a^{(t)}=\pi_{\theta}\Big{(}S^{(t)}\Big{)}+\mathcal{N}.\] (20)  
The structures of actor and critic networks are shown in Table 1 and both of them have three layers of neural networks. The overall algorithm is summarized in Algorithm 1. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Parameter Networks** & **Number of Neurons** & **Activation Function** \\ \hline \multirow{2}{*}{Actor} & \(128\) & ReLU \\ \cline{2-3}  & \(64\) & ReLU \\ \cline{2-3}  & \(N+M*K*2\) & \(\tanh(\cdot)\) \\ \hline \multirow{3}{*}{Critic} & \(64\) & ReLU \\ \cline{2-3}  & \(32\) & ReLU \\ \cline{2-3}  & \(1\) & None \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Network Structures. ```
1:Randomly initialize \(\theta^{(t)}\), \(w^{(t)}\), \(w^{(t+1)}=w^{(t)}\), \(\theta^{(t+1)}=\theta^{(t)}\). Empty the collection of experience playback \(D\). 2:forI = 1,2,..., T do
3: Initialize \(S^{(t)}\) as the first state of the current state sequence, and get its eigenvector \(\phi\Big{(}S^{(t)}\Big{)}\). 4: Get the action \(A^{(t)}=\pi_{\theta}(S)+\mathcal{N}\) in Actor's current network based on state \(S\). 5: Perform the action \(A^{(t)}\), get a new state \(S^{(t+1)}\), reward \(R^{(t)}\), and determine whether arrive the termination status 'end'. 6: Stores the array \(\Big{\{}\phi\Big{(}S^{(t)}\Big{)},A^{(t)},R^{(t)},\phi\Big{(}S^{(t+1)}\Big{)}, end\Big{\}}\) into the empirical playback set \(D\). 7:\(S^{(t+1)}=S^{(t)}\)
8: Get \(m\) samples \(\Big{\{}\phi\Big{(}S^{(t)}_{j}\Big{)},A^{(t)}_{j},R^{(t)}_{j},\phi\Big{(}S^{( t+1)}_{j}\Big{)},end_{j}\Big{\}}\), \(\forall j=1,2,\ldots,m\), from empirical playback sets, and calculate the current target \(Q^{\prime}\)s value \(y_{j}\) : \[y_{j}= \begin{cases}R^{(t)}_{j}\\ R^{(t)}_{j}+\gamma Q^{(t+1)}\Big{(}\phi\Big{(}S^{(t+1)}_{j}\Big{)},\pi_{\theta^{ \prime}}\Big{(}\phi\Big{(}S^{(t+1)}_{j}\Big{)}\Big{)}\Big{)}.\end{cases} \] (21)
9: Use the mean-variance loss function \[\frac{1}{m}\sum_{j=1}^{m}\Big{(}y_{j}-Q^{(t)}\Big{(}\phi\Big{(}S^{(t)}_{j} \Big{)},A^{(t)}_{j},w\Big{)}\Big{)}^{2},\] to update the Critic's current network parameter w through the gradient backpropagation of neural networks. 10: Use \[J(\theta)=-\frac{1}{m}\sum_{j=1}^{m}Q^{(t)}(s_{i},a_{i},w).\] to update the Actor's current network parameter \(\theta\) through the gradient backpropagation of neural networks. 11: If \(t\%\text{C}=1\), update Critic's target network and parameters of Actor's target network: \[w^{(t+1)}=\tau w^{(t+1)}+(1-\tau)w^{(t+1)}.\] \[\theta^{(t+1)}=\tau\theta^{(t)}+(1-\tau)\theta^{(t+1)}.\]
12: If \(S^{(t+1)}\) is at termination status, end the current time step's iteration; otherwise go to Step b. 13:endfor
```  
**Algorithm 1** The Proposed DDPG Algorithm.