# Evolutionary Reinforcement Learning: A Systematic Review and Future Directions
## 4 Methods of Evolutionary Reinforcement Learning
### Other EAs  
The preceding chapters covered classical algorithms based on evolutionary strategies, genetic algorithms, the CEM, and PBT, each demonstrating notable results in their respective application scenarios. Additionally, there are other promising EvoRL methods that, while less commonly employed, deserve attention. These include random search-based approaches designed to enhance efficiency by streamlining the search process, genetic programming methods that optimize strategies through the simulation of biological genetic processes, and evolutionary computing algorithms that emphasize the use of the principles of evolutionary theory to improve the learning process. Random search simplifies the parameter optimization process while maintaining a certain level of exploration ability. This approach can efficiently find solutions in complex tasks. DDQN-RS [84] applies random search to randomly sample individuals from the population using a Gaussian distribution and evaluate their fitness according to the reward obtained from one round of running in the environment. Compared to the Double Deep Q-Network (DDQN), the proposed model performs better than it does in the mission of keeping a vehicle close to the center of a lane for the longest distance. Compared to the efficient parameter optimization discussed in [84], the genetic programming in [83] demonstrated a capability for the in-depth optimization of computation graphs. In this framework, genetic programming is applied to search in the computation graph space, and these graphs compute the minimal objective function required by the agent. Genetic programming enhances these computation graphs by simulating the evolutionary process of organisms. The proposed model applies the loss function from the Deep Q-Network (DQN) as a key component:  
\[L_{DQN}=(Q(s_{t},a_{t})-(r_{t}+\gamma\cdot\max_{a}Q_{\text{targ}}(s_{t+1},a))) ^{2}, \tag{12}\]  
where \(Q(s_{t},a_{t})\) indicates the Q-value under the current state \(s_{t}\) and action \(a_{t}\), \(r_{t}\) represents an instant reward, \(\gamma\) denotes the discount factor, and \(\max_{a}Q_{\text{targ}}(s_{t+1},a)\) stands for the maximal expectation of the Q-value under the next state \(s_{t+1}\). By applying this loss function, the proposed method focuses more on a more accurate estimation of the Q-value to solve the overestimation issue. Experimental results showed that a DQN modified by genetic programming behaved better than the original DQN in Q-value estimation [83]. Similarly, ref. [85] proposed an EGPRL method that applies genetic programming to search in the computation graph space, finally minimizing the objective function. EGPRLallows agents to operate under multiple environments, including OpenAI Gym's Classic Control Suite. Experimental results showed that the proposed model possessed a competitive generalization ability and efficiency. Differently from [83] which considers the accurate estimation of the Q-value as its core concept, EGPRL focuses more on the hierarchical structure of memory coding and multi-tasking. In addition, ECRL [81] applies evolutionary computing to optimize the parameters in reinforcement learning. More specifically, evolutionary computing uses the fitness function to assess each set of parameters, iteratively searching for the optimal solution in the parameter space. Similarly, the RL-RVEA [28] also applies evolutionary computing, combined with the Q-learning from reinforcement learning, to optimize solutions for multi-objective problems by dynamically adjusting reference vectors, effectively enhancing the adaptability and performance of the algorithm. Additionally, RLHDE [90] utilizes Q-learning to dynamically adjust the mutation strategies in differential evolution. Compared to traditional hybrid EAs, RLHDE demonstrates a better global search capability and convergence speed. It outperforms existing state-of-the-art algorithms on several trajectory optimization benchmark problems provided by ESA-ACT, such as Cassini2 and Messenger-full.