# Evolutionary Reinforcement Learning: A Systematic Review and Future Directions
## 4 Methods of Evolutionary Reinforcement Learning
### Genetic Algorithm  
Genetic algorithm-based EvoRL methods are different from traditional methods; they focus on applying genetic diversity to the policy search process, which makes the algorithm find an effective and stable policy in a highly complex environment. This characteristic has been particularly beneficial in real-world applications, such as autonomousdriving, game AI, and resource management, where the environment is dynamic and highly unpredictable.  
One of the valuable contributions of genetic algorithms is the ability to introduce significant diversity in parameters and strategies through crossover operations. This diversity enables the exploration of a broader solution space, which is crucial for finding effective and robust policies in highly complex environments. In [54], the authors introduced an MAERL method that mainly focuses on parameter optimization in the processing industry. The method consists of multi-agent reinforcement learning, Graph Neural Networks (GNNs), and genetic algorithms. The ERLGA [55] method involves the combination of a genetic algorithm and off-policy reinforcement learning, finally reaching a better performance on return than existing methods. Moreover, ref. [59] proposed a MetaPG algorithm to optimize different reinforcement learning targets using multi-objective searching standards and consider individual reinforcement learning targets via the Non-Dominated Sorting Genetic Algorithm II (NSGA-II). MetaPG is able to improve performance and generalization by about 3% and 7% compared to Soft Actor-Critic (SAC) by adjusting the loss function. In addition, ERL-TD [27] similarly utilizes a genetic algorithm-based policy gradient framework but optimizes the learning process by incorporating truncated variance and distillation mutations to enhance the data efficiency and asymptotic performance. Simultaneously, genetic algorithms play a crucial role in fine-tuning the intricacies of policy evolution; ref. [61] developed a PDERL method to solve the scalability issues caused by simple genetic encoding in traditional algorithms. The PDERL method applies the following formula to define the proximal mutation operator:  
\[\text{Sens}=\frac{1}{|A|N_{M}}\sum_{i}\|\
abla_{\theta}\mu_{\theta}(s_{i})\|^{ 2}, \tag{8}\]  
where Sens represents the sensitivity of the action to weight perturbation, \(|A|\) is the size of the action space, \(N_{M}\) denotes the sample size used for calculating the sensitivity, and \(\
abla_{\theta}\mu_{\theta}(s_{i})\) is the gradient of the policy network to its parameter, \(\theta\), used to evaluate the sensitivity of policy changes to the parameter under state \(s_{i}\).  
In addition, genetic algorithms can be applied to tackle complex reinforcement learning problems that demand extensive interaction with the environment; ref. [62] proposed an SERL which contains a surrogate-assisted controller module. The module combines a genetic algorithm and actor-critic algorithm, where the genetic algorithm here is mainly used for evaluating the fitness of the genetic population. The method applies surrogate models to predict the environmental performance of individuals, which decreases the requirements of direct interaction with the environment and leads to a lower computational cost.