# A Deep Evolution Policy-Based Approach for RIS-Enhanced Communication System
## 4 Joint Active and Passive Beamforming Optimization Based on DEP  
The RIS-aided MU-MISO downlink communication system is considered in this paper, and the active beamforming at the BS and the passive beamforming at the RIS are jointly optimized with the DEP-based algorithm. The initial population size, the matching population size, and the offspring population size are all set as \(P\) in \(G\) generations. A few basic elements of the DEP-based algorithm are shown as follows:  
**State**: Different from gradient descent-based DRL algorithms, the neural network structure is optimized without backpropagation in the DEP-based algorithm. Thus, the state \(s^{(t)}\) only includes the channels \(\mathbf{H}_{1}^{(t)}\) and \(\mathbf{h}_{k,2}^{(t)}\) at the time step \(t\). \(\mathbf{H}_{1}=\text{Re}\{\mathbf{H}_{1}\}+\text{Im}\{\mathbf{H}_{1}\}\), \(\mathbf{h}_{k,2}=\text{Re}\{\mathbf{h}_{k,2}\}+\text{Im}\{\mathbf{h}_{k,2}\}\). The real part and the imaginary part of each entry of \(\mathbf{H}_{1}\) and \(\mathbf{h}_{k,2}\) are also used as entries of the state, and the dimension of the state space is \(2 NM+2 NK\). **Action**: After receiving the input state \(s^{(t)}\), the deep natural network which represents the agent outputs the action \(a^{(t)}\). The action is simply constructed by the transmit beamforming matrix \(\mathbf{W}^{(t)}\) and the phase shift matrix \(\mathbf{\Phi}^{(t)}\). Likewise, to tackle the real input problem, \(\mathbf{W}=\text{Re}\{\mathbf{W}\}+\text{Im}\{\mathbf{W}\}\) and \(\mathbf{\Phi}=\text{Re}\{\mathbf{\Phi}\}+\text{Im}\{\mathbf{\Phi}\}\) are separated as a real part and imaginary part, and both are entries of the action. The dimension of the action space is \(2 MK+2N\). **Reward**: The reward \(r^{(t)}\), that is the system sum-rate, is produced by the interaction between the action \(a^{(t)}\) and the fitness function as shown in (9). \[r^{t}=f\big{(}\mathbf{W}^{t},\mathbf{\Phi}^{t}\big{)}=\sum_{k=1}^{K}\log_{2}(1+\gamma_{ k}). \tag{9}\]The DNN utilized in this paper is characterized by a uniform network architecture, wherein each agent is represented as a parameter vector comprising distinct weights \(w\) and biases \(b\). It is assumed that both hidden layers have the same dimension \(h_{\text{dim}}\). Consequently, all agents can be expressed as \(\text{Para}_{p}(p=1,2,\ldots,P)\) with a consistent dimensional size of \(D\), where \(D=2(NM+NK)h_{\text{dim}}+(h_{\text{dim}}+2)h_{\text{dim}}+2(MK+N)h_{\text{dim}}+ 2(MK+N)\). Figure 5 shows the crossover process between parent agents. Specifically, parent agents \(\text{Para}_{1}\) and \(\text{Para}_{2}\) choose a random genetic locus to crossover, and then produce offspring agents \(\text{Para}_{1}^{*}\) and \(\text{Para}_{2}^{*}\). \[(\text{Para}_{1}^{*},\text{Para}_{2}^{*})=\text{cross}(\text{Para}_{1},\text{ Para}_{2}), \tag{10}\]  
where \(\text{cross}(\cdot)\) denotes the crossover process. The parent agents will continue to execute the crossover process until the offspring population reaches the size of \(P\). Figure 6 illustrates the mutation process designed to reduce the risk of the population converging to local optima. The primary aim of the mutation process is to incorporate new genetic information into the population. \[\text{Para}_{p}^{**}=\text{mut}(\text{Para}_{p}^{*},\mu) \tag{11}\]  
where \(\text{mut}(\cdot)\) represents the mutation process, while \(\mu\) represents the individual mutation rate. Specifically, agent \(p\) with mutation rate \(\mu\) randomly selects \(\mu D\) parameters for mutation from the set \(\text{Para}_{p}^{*}\). It is important to note that agents with high mutation rates may cause the most fit individuals to lose advantageous genetic information. Conversely, agents with low mutation rates may struggle to identify the optimal policy due to the limited introduction of new information. Therefore, selecting an appropriate mutation rate is crucial for the DEP algorithm.