# Evolutionary Reinforcement Learning: A Systematic Review and Future Directions
## 1 Introduction  
Reinforcement learning utilizes agents for autonomous decision-making, focusing on long-term action strategies, particularly effective in tasks like industrial automation and personalized recommendation systems [1, 2, 3]. However, reinforcement learning faces challenges such as parameter sensitivity and sparse rewards, leading to issues in its learning efficiency and adaptability [4, 5]. On the other hand, Evolutionary Algorithms (EAs),inspired by [PERSON]'s natural selection, excel in solving complex, multi-objective problems in large solution spaces [6; 7], showcasing their robust search capabilities in a wide range of optimization tasks. Hence, Evolutionary Reinforcement Learning (EvoRL), which integrates EAs with reinforcement learning, has been introduced to address the limitations of each method [8]. EvoRL maintains multiple policies within a population and utilizes evolutionary operations like crossover and mutation to refine these policies, enhancing the policy-making process inherent in reinforcement learning. Simultaneously, EvoRL leverages the global search capabilities of EAs for exploring the policy space and optimizing various components like agents and actions. EvoRL's core mechanism, combining the precision of policy gradients with EAs' global search, enables effective solutions in complex, high-dimensional environments [9; 10]. Additionally, approaches like EGPGs and CERL within EvoRL focus on the collaborative efforts of multiple agents, boosting performance in intricate tasks [11]. EvoRL has been applied in various domains, demonstrating its versatility and effectiveness. For instance, EvoRL enhances the sample efficiency in reinforcement learning, a crucial aspect for practical applications [12; 13]. In embodied intelligence, EvoRL fosters complex behavior through the integration of learning and evolution, offering new perspectives in this field [14]. Another significant application of EvoRL lies in quality diversity for neural control, contributing to the advancement of neural network-based control systems [15]. EvoRL's integration with Deep Reinforcement Learning (DRL) has been instrumental in promoting novelty search, expanding the boundaries of exploration in reinforcement learning [10]. Furthermore, the SUPER-RL approach within EvoRL, which applies genetic soft updates to the actor-critic framework, has been shown to improve the searching efficiency of DRL [16]. Early research highlighted the importance of meta-parameter selection in reinforcement learning, a concept that remains relevant in current EvoRL applications [17]. Lastly, the combination of population diversity from EAs and policy gradient methods from reinforcement learning has led to novel approaches like differentiable quality diversity, further enhancing the gradient approximation capabilities of EvoRL [18]. More specifically, conventional reinforcement learning focuses on obtaining the optimal policy through interactions with the environment. However, it is unstable in high-dimensional and complex environments. EvoRL, by combining reinforcement learning with EAs and taking advantage of global search, not only considers local optimization when searching for optimal policies but also avoids the local optimal problems in reinforcement learning methods. For relatively simple, well-defined tasks, traditional reinforcement learning may be sufficient. However, EvoRL is more effective when faced with complex, dynamic, and high-dimensional environments, especially when a lot of exploration is required. Recent advances in EvoRL [19; 20] have demonstrated significant improvements in both reinforcement learning and EAs. From the perspective of reinforcement learning, EvoRL has been shown to notably enhance the sample efficiency and expand the exploratory capabilities, which are essential in addressing reinforcement learning's limitations in complex and high-dimensional problem spaces [10; 13].