# Deep Reinforcement Learning for RIS-Aided Multiuser MISO System with Hardware Impairments
## 2 System Model
In addition, the array response vector is defined as  
\[\mathbf{x}=\sum_{k=1}^{K}(\mathbf{w}_{k}s_{k}+\mathbf{\eta}_{s}), \tag{1}\]1 \[\mathbf{x}=\sum_{k=1}^{K}(\mathbf{w}_{k}s_{k}+\mathbf{\eta}_{s}), \tag{1}\]2  
where \(X\) can be substituted as \(M\), \(N\) or \(K\) and \(\theta_{A}^{a}\) and \(\theta_{A}^{e}\) denote the azimuth and elevation angles, respectively; \(d\) and \(\lambda\) represent the antenna spacing at the BS and the wavelength, respectively. Moreover, \(\widetilde{\mathbf{H}}_{SI}\sim\mathcal{CN}(0,\mathbf{R}_{HR}\otimes\mathbf{ R}_{HB})\), \(\widetilde{\mathbf{h}}_{SD,k}\sim\mathcal{CN}(0,\mathbf{R}_{BB,k})\) and \(\widetilde{\mathbf{h}}_{ID,k}\sim\mathcal{CN}(0,\mathbf{R}_{HR,k})\) represent the non-line-of-sight component with \(\mathbf{R}_{HR}\), \(\mathbf{R}_{HB}\), \(\mathbf{R}_{HB,k}\) and \(\mathbf{R}_{HR,k}\) being the corresponding spatial covariance matrices, which are given by \([\mathbf{R}_{HB}]_{i,j}=\rho|^{i-j}|\), \([\mathbf{R}_{HR}]_{i,j}=\rho|^{i-j}|\), \([\mathbf{R}_{HB}]_{i,j}=\rho|^{i-j}|\) and \([\mathbf{R}_{HR}]_{i,j}=\rho|^{i-j}|\) and \(\rho\) represents the correlation coefficient. \[\mathbf{x}=\sum_{k=1}^{K}(\mathbf{w}_{k}s_{k}+\mathbf{\eta}_{s}), \tag{1}\]3 \[\mathbf{x}=\sum_{k=1}^{K}(\mathbf{w}_{k}s_{k}+\mathbf{\eta}_{s}), \tag{1}\]4  
\[\mathbf{x}=\sum_{k=1}^{K}(\mathbf{w}_{k}s_{k}+\mathbf{\eta}_{s}), \tag{1}\]5  
Thus, the signal received at the \(k\)-th user can be written as  
\[\mathbf{x}=\sum_{k=1}^{K}(\mathbf{w}_{k}s_{k}+\mathbf{\eta}_{s}), \tag{1}\]6 \[\mathbf{x}=\sum_{k=1}^{K}(\mathbf{w}_{k}s_{k}+\mathbf{\eta}_{s}), \tag{1}\]7 \[\mathbf{x}=\sum_{k=1}^{K}(\mathbf{w}_{k}s_{k}+\mathbf{\eta}_{s}), \tag{1}\]8where \(\mathbf{\Phi}=\text{diag}\big{(}e^{i\theta_{1}}e^{i\theta_{2}},\cdots,e^{i\theta_{N}} \big{)}\) denote the phase shift matrix at the RIS and \(\theta_{i}\) is the phase shift of the \(i\)-th reflecting element; \(\mathbf{\eta}_{B}\sim\mathcal{CN}(0,k_{B}\big{\{}|\widetilde{y_{k}}|^{2}\big{\}})\) denotes the user's additional distortion noise, which satisfies the Gaussian distribution with zero mean and \(k_{B}\in(0,1)\) denotes the normalized variance of the received distortion noise; \(\mathbf{\eta}_{k}\sim\mathcal{CN}(0,\sigma_{k}^{2})\) denotes the additive Gaussian white noise by the \(k\)-th user. Therefore, the \(k\)-th user's instantaneous signal-to-interference-plus-noise ratio (SINR) is given by (12) and (13) on the next page. Based on (12) and (13), the instantaneous data rate of the \(k\)-th user can be expressed as  
\[\mathbf{x}=\sum_{k=1}^{K}(\mathbf{w}_{k}s_{k}+\mathbf{\eta}_{s}), \tag{1}\]9  
Therefore, the optimization problem in this paper can be written as  
\[ \begin{array}{rl}\underset{\mathbf{W},\mathbf{\Phi}}{\text{max}}&\underset{k} {\mathbb{E}}[R_{k}]\\ \text{s.t.}&\text{C1}:\text{tr}\{\mathbf{WW}^{H}\}\leq P_{\text{max}},\\ &\text{C2}:|\theta_{i}|=1,\forall i=1,2,\ldots,N,\end{array}  \tag{16}\]  
where the expectation in the objective function is taken over the online-of-sight components in the CSI. In the above optimization problem, C1 represents the power constraint at the BS, while C2 means the unit modulus constraints of the phase shifts of the RIS. Unfortunately, it is challenging to derive the closed-form expression of the objective function since the average data rate contains the expectation operation over numerous random small-scale channel gains. In addition, this work studied the impact of hardware impairment on both the BS and the users. The average data rate expressions would be much more complicated. Hence, there are no existing mathematical algorithms that can solve these kinds of optimization problems.