# Evolutionary Reinforcement Learning: A Systematic Review and Future Directions
## 4 Methods of Evolutionary Reinforcement Learning
### Evolutionary Strategy  
In the expansive field of reinforcement learning, the value function method has always been one of the core research directions, mainly focusing on how to effectively estimate and optimize the expected return under a given policy. Against this background, the natural parameter search mechanism of evolutionary strategy provides a unique approach by simulating the processes of natural selection and heritable variation. This approach has been successfully applied in various real-world scenarios, such as robotic control, automated trading, and industrial process optimization, where traditional gradient-based methods face challenges due to high dimensionality.  
Mutation, one of the most essential operations in evolutionary strategy, introduces diversity to the algorithm process by randomly modifying policies or parameters. This operation enables EvoRL to effectively adapt to complex learning environments and avoid falling into local optima. In [44], the authors employed a simple but effective EvoRL algorithm called AEES, which contains two distinct, coexisting mutation strategies. Each of the two strategies is connected with their population subsets. That is, each subset mutates in accordance with one related mutation strategy. AEES applies the cumulative return and convergence rate as evaluation metrics, and the proposed model shows better performance compared to A2C, SAC, and other DRL methods. Compared to [44], the OHT-ES algorithm in [45] focuses more on adjusting key parameters of the reinforcement learning method through evolutionary strategy, hence improving the adaptability and efficiency of the algorithm. Ref. [45] proves that OHT-ES performs better than traditional DRL (e.g., TD3) in terms of the learning speed.  
Differently from the value function, the introduction of evolutionary strategy in the policy gradient method provides a brand new perspective. The idea for the realization of R-R1-ES [46] is distinct from [44]; R-R1-ES put special emphasis on the direct optimization of the policy itself, which applies the Gaussian distribution model \(N(\theta_{t},\sigma^{2}C_{t})\) and a restart mechanism to update the searching direction, where \(\theta_{t}\in\mathbb{R}^{n}\) represents the distribution mean, \(\sigma_{t}>0\) denotes the mutation strength, and \(C_{t}\) is an n-dimensional covariance matrix at \(t\) iterations. The update rule of \(C_{t}\) is given as  
\[C_{t}=(1-c_{cov})I+c_{cov}p_{t}p_{t}^{T}, \tag{7}\]  
where \(c_{cov}\in(0,1)\) is the changing rate of the covariance matrix, \(I\) denotes the unit matrix, and \(p_{t}\) is a vector which represents the primary search direction. The model performs better than NS-ES (Novelty Search-Evolutionary Strategy) according to reward evaluation. The application of evolutionary strategies is anticipated to augment the global search proficiency and yield refined gradient information, through which they can potentially accelerate the convergence towards optimal policy solutions [48].  
In addition, the ZOSPI model [50] reveals the potential of combining evolutionary strategy and an actor-critic algorithm. Compared to R-R1-ES [46], ZOSPI chooses to optimize the policy from global and local perspectives, which both exploit the advantages of the global value function and the accuracy of the policy gradient fully. The approach not only improves the sample efficiency but expands the possibility of multi-modal policy learning, creating a novel trajectory for the actor-critic algorithm.