# Deep Reinforcement Learning for RIS-Aided Multiuser MISO System with Hardware Impairments
## 3 Proposed Algorithm
### DDPG Algorithm  
As the objective function in (16) does not have a closed-form expression, it is difficult to solve this problem using conventional optimization algorithms. Therefore, in this paper, we adopt a deep reinforcement learning algorithm, which can efficiently process complex environmental parameters and a large amount of state information by utilizing techniques such as stochastic gradient optimization and inverse parameter transfer in deep neural networks. Specifically, the DDGP algorithm is employed to solve the optimization problem in this paper, which is one of the deep reinforcement learning algorithms. The DDPG algorithm can be used to solve this challenging optimization problem with continuous variables. The DDPG algorithm adopts the Actor-Critic architecture, which uses the policy network to output deterministic actions directly, and the functions of its four networks are introduced as follows:  
1. _Actor Current network_ The role of the actor current network is to iteratively update the policy network parameters \(\theta\) and select the current action according to the state \(S^{(t)}\) at time step \(t\), which is composed of three parts: the beamforming matrix \(\textbf{W}^{(t)}\in\mathbb{C}^{M\times K}\), the phase shift matrix \(\textbf{\Phi}^{(t)}\in\mathbb{C}^{N\times N}\) and the channel matrices, i.e., \(\textbf{H}^{(t)}_{SI}\in\mathbb{C}^{N\times M}\), \(\textbf{h}^{(t)}_{SDK}\in\mathbb{C}^{M\times 1}\), \(\textbf{h}^{(t)}_{ID,k}\in\mathbb{C}^{N\times 1}\). In addition, the actor current network also interacts with the environment to generate \(S^{(t+1)}\) and reward \(R^{(t)}\), which can be defined as \(R^{(t)}=\underset{\textbf{W},\textbf{\Phi}}{\text{max}}\underset{k}{\text{min }}\quad\mathbb{E}[R^{(t)}_{k}]\) and \(R^{(t)}_{k}\) is defined in (15). The expression of loss function \(J^{(t)}(\theta)\) can be expressed as \[J^{(t)}(\theta)=-\frac{1}{m}\sum_{j=1}^{m}Q^{(t)}(s_{i},a_{i},w).\] (17)
2. _Actor Target network_ The actor target network serves to select the optimal action \(\textbf{a}^{(t+1)}\) based on the state \(S^{(t+1)}\) at time \(t+1\) sampled in the empirical playback pool. The action \(\textbf{a}^{(t)}\in\mathbb{R}^{2 MK+N}\) is composed of two parts: the first \(N\) elements corresponding to the phase shifts of RIS reflecting elements and the remaining \(2 MK\) elements corresponding to the real part and imaginary part of the beamforming matrix, respectively. We take action \(\textbf{a}^{(t)}\) to optimize the beamforming matrix \(\textbf{W}^{(t)}\) and the phase shift matrix \(\textbf{\Phi}^{(t)}\), and the optimized results can be described as \[\textbf{W}^{(t+1)}=\frac{\sqrt{P_{\text{max}}}\textbf{W}^{(t)}}{\left\| \textbf{W}^{(t)}\right\|_{F}}.\] \[\phi^{a(t+1)}_{n}=\phi^{a(t)}_{n}+a^{(t)}_{j}\pi\] where \(\phi^{(t)}_{n}=\cos(\phi^{a(t)}_{n})+j\sin(\phi^{a(t)}_{n})\) is the \(n\)-th phase shift in \(\textbf{\Phi}^{(t)}\) and \(a^{(t)}_{j}\) is the \(j\)-th action value in \(\textbf{a}^{(t)}\), \(\forall n=j=1,2,\ldots,N\). Figure 2: Transmission Scheme  
The target network parameter \(\theta^{(t)^{\prime}}\) is periodically copied from the current network parameter \(\theta^{(t)}\), which uses the soft update method, and the soft update factor is \(\tau\). \[\theta^{(t+1)}=\tau\theta^{(t)}+(1-\tau)\theta^{(t+1)}.\] (3) _Critic Current network_ The critic current network is used to iteratively update the value network parameter \(w^{(t)}\) and calculate the current value of \(Q\Big{(}S^{(t)},a^{(t)},w^{(t)}\Big{)}\). The target value of \(Q^{(t)^{\prime}}\) is given by \[y_{i}=R^{(t)}+\gamma Q^{\prime}\Big{(}S^{(t)},a^{(t)},w^{(t)^{\prime}}\Big{)}.\] (18) The loss function is given by \[J\Big{(}w^{(t)}\Big{)}=\frac{1}{m}\sum_{j=1}^{m}\Big{(}y_{j}-Q\Big{(}\phi \Big{(}S^{(t)}_{j}\Big{)},a^{(t)}_{j},w^{(t)}\Big{)}\Big{)}^{2}.\] (4) _Critic Target network_ The critic target network aims to calculate the \(Q^{\prime}\Big{(}S^{(t)},a^{(t)},w^{(t)^{\prime}}\Big{)}\) portion of the target value \(Q\).