# A Deep Evolution Policy-Based Approach for RIS-Enhanced Communication System
## 3 Algorithm Description
### Deep Evolution Policy  
A typical M-P neuron model is illustrated in Figure 2, where \(x_{i}\) represents the \(i\)-th input signal from other neurons. \(\omega_{i}\), \(b\), and \(f\) denote the weight, bias, and activation function, respectively. The output of the M-P neuron can be expressed as follows:  
\[y=f\left(\sum_{i=1}^{n}\omega_{i}x_{i}-b\right) \tag{8}\]  
The limited learning capacity of a single neuron necessitates the adoption of DNNs, which comprise multiple layers of neural networks, to effectively address nonlinear separable problems. Typically, the process of DL can be represented as nonlinear function approximation, wherein multi-layer neural networks decompose complex mappings into a series of simpler mappings, each represented by different layers of the model. DRL is an RL algorithm that leverages the robust nonlinear fitting capabilities of DL to optimize the parameters of neural networks through interactions with the environment. Additionally, DEP is an evolutionary learning algorithm that treats distinct DNNs as individual entities within a population. A DNN featuring two hidden layers is illustrated in Figure 3.  
The proposed DEP-based algorithm integrates the advantages of RL and DL by employing an evolutionary method inspired by unnatural selection. Specifically, the natural selection process inherent to biological evolution is simulated through the use of DEP in conjunction with DNNs and evolutionary algorithms. In each generation, multiple deep neural networks with varying structures and parameter vectors are concurrently generated, which then interact with the environment to yield diverse fitness scores. The environment may be regarded as a fitness function, with the fitness score serving as a reward according to the current state and characteristics of the agents (individuals). During the evolutionary process, agents that yield higher rewards are typically favored in the selection process. The schematic diagram of the DEP-based algorithm is illustrated in Figure 4.  
Figure 2: M-P neuron model. exploitation strategy, often referred to as the greedy strategy, ranks all agents based on their rewards and selects the two agents with the highest rewards as the parent agents. Conversely, the pure exploration strategy, commonly known as the random strategy, selects parent agents at random. Most existing selection strategies integrate both the pure exploitation and pure exploration strategies, with typical mixed selection strategies including the soft-max strategy, \(\epsilon\)-greedy strategy, and attenuated \(\epsilon\)-greedy strategy. Most mixed selection strategies emphasize the accumulation of experience during the initial stages and the exploration of actions in the later stages. In contrast to gradient descent-based algorithms, the DEP-based algorithm needs more extensive exploration space for generating offspring through population reorganization and mutation in each generation. In this paper, we propose a random subspace selection (RSS) strategy for the selection of parent agents, as summarized in Algorithm 1. Specifically, \(P\eta\) agents are randomly selected from the matching population to form the subspace population, where \(\eta\) represents the proportion of the subspace. The agents within the subspace population are ranked according to their fitness scores, and parent agents are selected based on the highest fitness scores. Notably, (1) when \(\eta=1\), the subspace encompasses the entire population, resulting in the RSS degenerating into a pure exploitation strategy, and (2) when \(\eta=2/P\), the subspace consists of only two agents, leading to the RSS degenerating into a pure exploration strategy. Consequently, the appropriate selection of the subspace proportion \(\eta\) can effectively balance exploration and exploitation. ```
1:Input: matching population, \(P\). 2:Initialize: \(\eta,m=1\). 3:Repeat:
4: Random select agent from matching population. 5:\(m=m+1\). 6:Until: \(m=P\eta\)
7: Select the agents with two highest fitness scores as parents from \(m\) agents.