# A Deep Evolution Policy-Based Approach for RIS-Enhanced Communication System
## 5 Numerical Results
Notably, after approximately 10,000 steps, equivalent to 20 generations (with a population size of 500 for each generation), the average reward converges to 7.3 bps/Hz. It is important to highlight that the final policy of a DRL algorithm depends on the last agent that interacts with the environment, while the DEP-based approach incorporates multiple agents in the final generation. Due to the robust exploration capabilities of the DEP algorithm, the agent \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Optimizer & Adam \\ Total time steps per training & 30,000 \\ Experience replay buffer size & 20,000 \\ Mini-batch size & 16 \\ Network update interval & after each environment step \\ Discount term & 1 \\ Learning rate for training critic network & 0.001 \\ Learning rate for training actor network & 0.001 \\ Learning rate for target critic network & 0.001 \\ Learning rate for target actor network & 0.001 \\ Entropy target & -action dimension \\ SAC log standard deviation clipping & (\(-20\), \(2\)) \\ SAC \(\epsilon\) & \(10^{-6}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: DRL-based algorithm hyperparameter settings. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Population size \(P\) & 500 \\ Subspace proportion \(\eta\) & 0.5 \\ Mutation rate \(\mu\) & 0.001 \\ Generation \(G\) & 60 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: DEP-based algorithm hyperparameter settings. yielding the highest instant reward can be selected as the final policy in the last generation. Consequently, the optimal sum rate achievable through the DEP algorithm can reach approximately \(9\,\mathrm{bps}/\mathrm{Hz}\). To investigate the influence of hyperparameter settings on system performance, we established several groups of comparative simulations that included variations in mutation rate, population size, and subspace proportion. The results of these simulations are presented in Figures 10-12. Figure 10 demonstrates the sum rate versus time steps under different mutation rates. The data indicate that mutation rates significantly affect the performance of the DEP algorithm. Specifically, the system achieves optimal performance in terms of converged sum rate at a mutation rate of 0.001. Conversely, an increase in the mutation rate to 0.1 results in a deterioration of the sum rate convergence. This convergence behavior can be attributed to the loss of advantageous genetic information within the population when subjected to a high mutation rate. Furthermore, it is important to highlight that excessively low mutation rates may lead the population to become trapped in local optima due to the insufficient introduction of new genetic material. In conclusion, it is essential to select an appropriate mutation rate that aligns with environmental parameters, ensuring it is neither excessively high nor excessively low. Figure 8: Converged sum rate versus transmit power to show the proposed DEP-based algorithm in comparison with two benchmarks, \(M=4\), \(N=32\), \(K=4\). Figure 7: Converged sum rate versus transmit power to show the proposed DEP-based algorithm in comparison with two benchmarks, \(M=4\), \(N=4\), \(K=4\). Figure 11: Sum rate versus total time steps under different population sizes, i.e., \(\{300,500,800\}\), \(P_{t}=5\) dBm, \(M=4\), \(N=32\), \(K=4\), mutation rate = 0.001, subspace proportion = 0.5. Shaded regions represent 95% confidence interval over 10 random seeds for each result. Figure 10: Sum rate versus total time steps under different mutation rates, i.e., \(\{0.1,0.01,0.001\}\), \(P_{t}=5\) dBm, \(M=4\), \(N=32\), \(K=4\), population size = 500, subspace proportion = 0.5. Shaded regions represent 95% confidence interval over 10 random seeds for each result. Figure 9: Sum rate as a function of total time steps at \(P_{t}=10\) dBm, \(M=4\), \(N=4\), \(K=4\). Shaded regions represent 95% confidence interval over 10 random seeds for each result.