# A Deep Evolution Policy-Based Approach for RIS-Enhanced Communication System
## 5 Numerical Results  
In this section, we present numerical simulations to evaluate the proposed DEP algorithm. We selected two state-of-art DRL algorithms as benchmarks: the DDPG algorithm [20], and the SAC algorithm [23]. The hyperparameter settings are detailed in Tables 1 and 2. It is important to distinguish between episodic and non-episodic tasks within the context of the environments. An episodic task concludes when a terminal condition is met, whereas a non-episodic (continuing) task does not have a specific endpoint. In this paper, the communication environment is classified as continuing, as the BS continuously performs beamforming and configures RIS elements. Consequently, we have set the discount term to 1, as the agent must equally prioritize both instant and future rewards, given that the objective is to maximize the cumulative reward (sum rate). From Figure 1, we can see that the direct channel from the BS to the User is blocked, and the transmission signal is transmitted through the reflection line-of-sight (LoS) links \(\mathbf{H}_{1}\) and \(\mathbf{h}_{k,2}\). \(\mathbf{H}_{1}\) and \(\mathbf{h}_{k,2}\) follow the Rician fading which can be modeled by  
\[\mathbf{H}_{1}=L_{1}\Bigg{(}\sqrt{\frac{K_{1}}{K_{1}+1}}\overline{\mathbf{H}}_{1}+\sqrt {\frac{1}{K_{1}+1}}\widetilde{\mathbf{H}}_{1}\Bigg{)}, \tag{12}\]  
\[\mathbf{h}_{k,2}=L_{2}\Bigg{(}\sqrt{\frac{K_{2}}{K_{2}+1}}\overline{\mathbf{h}}_{k,2} +\sqrt{\frac{1}{K_{2}+1}}\tilde{\mathbf{h}}_{k,2}\Bigg{)}, \tag{13}\]  
where \(L_{1}\) and \(L_{2}\) denote the large-scale path losses, which can be modeled by \(32.4+20\lg(f)+19.8\lg(d)+2\) according to the 3 GPP propagation environment. \(f\) is the signal frequency, and \(d\) is the distance between the transceiver. In this paper, the carrier frequency, the transmission bandwidth, and the noise power spectral density are set at 3.5 GHz,200 kHz, and \(-170\) dBm/Hz, respectively. \(K_{1}\) and \(K_{2}\) are the Rician factors. It can be seen that when the Rician factor is 0, the reflect link becomes a non-line-of-sight (NLoS) channel which follows the Rayleigh fading, and we set \(K_{1}=K_{2}=10\) for simulation. \(\widetilde{\mathbf{H}}_{1}\) and \(\widetilde{\mathbf{h}}_{k,2}\) denote NLoS components whose elements are chosen from \(\mathcal{CN}(0,1)\), while \(\overline{\mathbf{H}}_{1}\) and \(\overline{\mathbf{h}}_{k,2}\) denote LoS components whose elements consist of the antenna array responses of the transmitter and receiver. We have conducted an evaluation of the proposed DEP-based approach, as outlined in Algorithm 2, in comparison to two benchmark methods. Figures 7 and 8 illustrate the relationship between the converged sum rate and transmit power. Two sets of system parameters were considered: \(M=4\), \(N=4\), \(K=4\), and \(M=4\), \(N=32\), \(K=4\). The results indicate that the performance of our proposed DEP-based approach surpasses that of the state-of-the-art benchmarks, specifically the DDPG and SAC algorithms. Furthermore, it is observed that the converged sum rate increases with transmit power across all the considered algorithms and scenarios. Figure 9 illustrates the convergence behavior of our proposed DEP-based algorithm in comparison to the DDPG and SAC algorithms. After approximately 10,000 time steps, the system sum rate of the SAC algorithm converges to 6.7 bps/Hz. In contrast, the sum rate of the DDPG algorithm reaches approximately 4.8 bps/Hz after 30,000 time steps. The slower convergence of the DDPG algorithm can be attributed to the continuous interactions between a single agent and the environment. Additionally, the actions generated by the DDPG algorithm exhibit limited exploration due to their deterministic nature. The SAC algorithm incorporates an additional critic network compared to DDPG. To enhance exploration within the actor network, the SAC algorithm employs an entropy-based method. In contrast to these DRL algorithms, our proposed DEP-based approach facilitates the simultaneous interactions between multiple agents and the environment.