# Evolutionary Reinforcement Learning: A Systematic Review and Future Directions
## 4 Methods of Evolutionary Reinforcement Learning
### Population-Based Training  
PBT-based EvoRL shows potential in multiple research fields. The core concept of it is to adjust the parameters and hyper-parameters of the algorithm dynamically in the training process, so that more effective and flexible learning is realized in complicated environments.  
Therefore, ref. [4] introduced a parameter control strategy training method for EAs and Swarm Intelligence (SI) algorithms called SBARL. PBT was applied in SBARL to evolve the parameters and hyper-parameters in reinforcement learning; the results of the experiment demonstrate that SBARL performed better than the traditional DRL method TD3. PKAEO [89], an EvoRL algorithm using PBT, evolves both the parameters and policies in reinforcement learning, facilitating more efficient optimization and adaptation. EARL [73] is a framework that combines EAs (more specifically, PBT) and reinforcement learning. The key concept of EARL is that the collaborative work of EAs and reinforcement learning could facilitate the learning and evolving process. Ref. [78] proposed an MERL algorithm that realizes individual and team objectives by combining gradient-less and gradient-based optimizers. Among them, the policy of gradient-based optimizers is added to the regularly evolving population, which enables EAs to leverage skills learned through training in individual-specific rewards to optimize team goals without relying on reward shaping. Besides the policy space, PBT can influence the action space in reinforcement learning, emphasizing the importance of optimizing action decisions. Ref. [74] proposed the EAS-TD3 method, which uses actions selected by the reinforcement learning policy to generate a population and then applies particle swarm optimization to iteratively evolve the population. The key concept of EAS-RL is choosing to optimize the action space but not the policy space; the definition of the loss function in the action space is  
\[L_{evo}(\theta,\mathcal{A})=\mathbb{E}_{(s_{i},a_{t}^{\prime})\sim\mathcal{A} }\{\|\mu_{\theta}(s_{i})-a_{t}^{\prime}\|^{2}\}, \tag{10}\]where the state \(s_{i}\) and the evolutionary action \(a_{i}^{e}\) in this formula are sampled, respectively, from the archive \(\mathcal{A}\), while \(\theta\) denotes the learning parameters in the reinforcement learning policy \(\mu_{\theta}\). The proposed model behaves better than TD3 in MuJoCo environments.  
Beyond optimizing parameters and actions, PBT places specific emphasis on automatically adjusting reward functions, e.g., ref. [80] focused on game AI generation. Therefore, they proposed a PBT-based EvoRL method, the EMOGI framework. EMOGI considers the reward function as a part of the candidate objects, realizing the auto-adjustment of the parameters by EAs. EMOGI applies multi-objective optimization to select policies with distinct behaviors to ensure population diversity. The key process of the initialization of EMOGI is to randomly initialize a candidate population consisting of policy parameters and reward weights:  
\[P=\{\pi_{\theta_{1}}R_{w_{1}},\ldots,\pi_{\theta_{n}}R_{w_{n}}\}, \tag{11}\]  
where \(P\) denotes the population of candidates; each candidate consists of two parts, the policy parameter \(\pi_{\theta}\) and the weight of the reward, \(R_{w}\). The size of the population is decided by \(n\).