# Evolutionary Reinforcement Learning: A Systematic Review and Future Directions
## 5 Challenges in Evolutionary Reinforcement Learning
### Issues Encountered by Evolutionary Reinforcement Learning
#### 5.2.3 Performance  
Exploring the performance of EvoRL unveils a range of challenges, especially concerning its core attributes. Despite its demonstrated advancements and potential in multiple domains, limitations such as its generalization, adaptation, sparsity, and scalability become apparent upon deeper investigation. These issues impact the effectiveness of EvoRL algorithms and restrict their application in wider and more complex settings. The generalization issues in EvoRL were discussed by [53]. They highlighted the significance of model transferring across multiple tasks in multi-task EvoRL environments. A common approach involves exploring a vast search space to identify the optimal policy. To address this challenge, ref. [53] introduced the A-MFEA-RL method, which aims to harness the potential synergies among tasks. This is achieved by employing evolutionary computing to tackle multiple optimization tasks simultaneously. At the heart of A-MFEA-RL is the concept of a uniform search space, wherein each candidate solution can be decoded and evaluated as a potential solution for any given task, thereby facilitating the efficient sharing and transfer of knowledge. Although EvoRL successfully generates diverse experiences for training reinforcement learning agents by evolving a series of actors, enabling it to outperform some advanced reinforcement learning and EA algorithms in unconstrained problems, this approach struggles to balance rewards and constraint violations in constrained environments. Ref. [60] introduced an ECRL algorithm designed to adaptively balance rewards and constraint violations through stochastic ranking and a Lagrange relaxation coefficient:  
\[\min_{\lambda}\max_{\theta}[J_{\pi}^{R}-\lambda(J_{\pi}^{C}-\delta)], \tag{19}\]  
where \(J_{\pi}^{R}\) and \(J_{\pi}^{C}\) represent the expected cumulative values of rewards and constraints, respectively. \(\lambda\geq 0\) is the Lagrange multiplier, and \(\delta\) denotes the constraint threshold. Simultaneously, ECRL restricts the policy's behavior by maintaining a constraint buffer. Adaptation issues in EvoRL also emerge from the high cost associated with interactions between policies in genetic populations and the real environment, as noted by [62]. These interactions often incur prohibitively high costs and can be impractical. To handle this issue, ref. [62] introduced a novel module, the surrogate-assisted controller (SC), and proposed the SERL method. This method aims to partially substitute the expensive environmental assessment process by incorporating surrogate models into the existing EvoRL framework. Central to this approach is the utilization of an approximate fitness function, derived from a surrogate model, to evaluate the fitness of individuals within a genetic population. This significantly reduces the necessity for direct interactions with the real environment, thus mitigating the adaptation challenges in EvoRL. In addition, unlike the previous articles focusing on the reward-constraint balance and cost of environment interaction, ref. [46] zeroed in on the mutation adaptability within EvoRL. R-R1-ES tackles this by simplifying CMA-ES for Deep Reinforcement Learning, enhancing mutation strength adaptation with key formulas:  
\[\sigma_{t+1}=\sigma_{t}\exp(\frac{s_{t+1}}{d_{\sigma}}), \tag{20}\]  
where \(\sigma_{t}\) is the current mutation strength, \(s_{t+1}\) denotes the cumulative rank rate, and \(d_{\sigma}\) presents a damping parameter that controls the adaption rate. EvoRL's sparsity issue is also noteworthy, as [93] identified challenges in discovering complex joint policies due to many policies yielding similar or zero fitness values. The proposed NS-MERL algorithm addresses this by combining a novelty-based fitness function for individual exploration and a sparse fitness function for team performance evaluation. Key to NS-MERL is the introduction of a count-based estimate of state novelty, fostering broader exploration within episodes and enhancing the likelihood of identifying effective joint policies.