# Evolutionary Reinforcement Learning: A Systematic Review and Future Directions
## 5 Challenges in Evolutionary Reinforcement Learning
### Issues Encountered by Evolutionary Reinforcement Learning
#### 5.2.1 Sample Efficiency  
The sample efficiency typically refers to the effectiveness of an EvoRL algorithm in achieving a certain level of performance with a minimal number of data samples. Algorithms with higher sample efficiency can extract useful information from a relatively small dataset, whereas those with lower sample efficiency require a larger dataset to achieve comparable performance. Consequently, sample efficiency holds significant importance in RL-related domains due to the time and resource consumption involved in acquiring samples through interactions with the environment. In [99], a projection technique was employed to address the challenge of low efficiency in sample transfer across different tasks. Specifically, the transfer of samples from one task to another can rapidly approach zero when the parameter space is sparse, potentially impeding updates to cross-task sample solutions. To maintain consistency in the distance, ref. [99] sampled parameter vectors from the distribution of one task and projected them back to the target distribution of another task.  
Furthermore, ref. [74] introduced the EAS-TD3 method, aiming to focus on enhancing the sample utilization efficiency by optimizing action selection. Through evolving actions and adjusting the Q-value, EAS-TD3 significantly improves EvoRL's ability to learn from limited samples. This approach not only enhances the algorithm's adaptability to complex environments but also effectively shortens the learning cycle and improves the  
\begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Issue** & **Solution** & **Method** & **RL Algorithm** & **Evo Algorithm** \\ \hline \multirow{3}{*}{Sample Efficiency} & Projection techniques are applied & NuEMT [99] & Actor–critic & Evolutionary strategy \\ \cline{2-5}  & Optimize action selection & EAS-TD3 [74] & Actor–critic & Evolutionary strategy \\ \hline \multirow{3}{*}{Algorithm Complexity} & Different mutation policies for each subset & AEES [44] & Value function & Evolutionary strategy \\ \cline{2-5}  & Parallel exploration & CCNCS [47] & Policy gradient & Evolutionary strategy \\ \hline \multirow{7}{*}{Performance} & Unify search space & A-MFEA-RL [53] & Value function & Genetic algorithm \\ \cline{2-5}  & Lagrange relaxation coefficient & ECRL [60] & Actor–critic & Genetic algorithm \\ \cline{2-5}  & Incorporate surrogate models into EvoRL framework & SERL [62] & Actor–critic & Genetic algorithm \\ \cline{2-5}  & Enhance mutation strength adaptation & R-R1-ES [46] & Policy gradient & Evolutionary strategy \\ \cline{2-5}  & Combine with a novelty-based fitness function & NS-MERL [93] & Actor–critic & Evolutionary strategy \\ \cline{2-5}  & Novel mutation operator & PDERL [61] & Actor–critic & Genetic algorithm \\ \hline \hline \end{tabular}
\end{table}
Table 3: Issues encountered by evolutionary reinforcement learning.  
sample efficiency by making precise use of each sample. The main framework is based on the following proposition:  
\[\mathbb{E}_{a^{\prime}\sim\mu_{e}(s)}[Q_{\mu_{e}}(s,a^{\prime})]\geq\mathbb{E}_{a ^{\prime}\sim\mu_{\theta}(s)}[Q_{\mu_{\theta}}(s,a)], \tag{18}\]  
The core concept of this proposition is that for any state, \(s\), the expectation \(\mathbb{E}_{a^{\prime}\sim\mu_{\theta}(s)}\) generated by policy \(\mu_{\theta}\) is surpassed by the expectation under the EAS-optimized policy \(\mu_{e}\). This indicates that the actions, \(a_{z}\), optimized through the EAS approach yield a higher expected return than those selected by the original policy, thereby enhancing the sample efficiency and overall learning performance.