# Evolutionary Reinforcement Learning: A Systematic Review and Future Directions
## 4 Methods of Evolutionary Reinforcement Learning
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Evolutionary Algorithm** & **RL Algorithm** & **Method** & **Evaluation Metric** & **Compared Algorithm** & **Benchmark Task** \\ \hline \multirow{6}{*}{Evolutionary Strategy} & \multirow{2}{*}{Value function} & AEES [44] & Cumulative return & A2C, SAC, DDPG, TRPO & Continuous control \\  & & OHT-ES [45] & Asymptotic performance & TDS & Continuous control \\  & & R-RI-ES [46] & Reward & OpenAI-ES, NS-ES & Continuous/Discrete Control \\  & & CCNCS [47] & Time budget & PPO, A3C & Discrete Control \\  & & cMA-ES [48] & Error & CMA-ES & Continuous control \\  & & MARL-ES [49] & Convergence & MADDPG & Continuous control \\ \cline{2-6}  & \multirow{4}{*}{Actor-critic} & ZOSPI [50] & Reward & SAC, TDS & Continuous control \\  & & EPG [51] & Return, KL & PPO & Continuous control \\  & & ES-TDS [52] & Mean, std error, median & TD3, CEM-RL & Continuous control \\ \hline \multirow{6}{*}{Genetic Algorithm} & \multirow{6}{*}{Value function} & A-MFEA-RL [53] & Time budget & SAC, PPO & Multi-task \\  & & MAERL [54] & MAE, RMSE, MME & TD3 & Parameter Optimization \\  & & RELA [53] & Return & NA & Continuous control \\  & & COE-RI-[56] & Time budget, episode & NA & Continuous control \\  & & PGGRL [57] & Time budget & NA & Discrete Control \\  & & DRL-GA [58] & Time budget & IGA, KBGA, DPABC, NS & Multi-task \\ \cline{2-6}  & \multirow{4}{*}{Policy gradient} & MERL [26] & Convergence & PPO, IRG & Multi-objective \\  & & MetaP [59] & Entropy, return & SAC & Continuous control \\  & & ERL-TD [27] & Time budget & SAC, TD3, DDPG & Continuous control \\ \cline{2-6}  & \multirow{4}{*}{Actor-critic} & ECRL [60] & Constraint, reward & IPO & Continuous control \\  & & PDERL [61] & Reward & TD3, PPO & Continuous control \\  & & SRL [62] & Step & DDPG & Continuous control \\  & & QD-PG-PF [63] & Training curve & DPG & Continuous control \\  & & ERL-Re [64] & Return & TD3 & Continuous control \\ \hline \multirow{6}{*}{Cross-Entropy Method} & \multirow{2}{*}{Value function} & QT-PPT [65] & Success rate & NA & Continuous control \\  & & R-RVEA [28] & HIV values & MoEAAGD, RVEA & Multi-objective \\ \cline{2-6}  & \multirow{2}{*}{Policy gradient} & GRAC [66] & Return & TD3, SAC, DDPG, TRPO, CEM & Continuous control \\  & & CEM-RL [67] & Training curve & TD3 & Continuous control \\  & & PGPS [68] & Return & CEM, PPO, DDPG, CERL, SAC & Continuous control \\  & & CSPC [69] & Return & SAC, PPO, CEM & Continuous control \\  & & SAC-CEP [70] & Return & SAC & Continuous control \\  & & DEPI [71] & Return & CEM, TD3 & Continuous control \\ \hline \multirow{6}{*}{Population-Based Training} & Value function & RLMODE [72] & Mean, STD & NSGAII & Multi-objective \\ \cline{2-6}  & \multirow{2}{*}{Policy gradient} & SBARL [4] & NA & TD3 & Parameter optimization \\  & & EARL [73] & NPV & SAC & Parameter optimization \\ \cline{2-6}  & \multirow{4}{*}{Actor-critic} & EAS-TDS [74] & Return & TD3 & Continuous control \\  & & PS-TDS [75] & Reward & TD3 & Continuous control \\  & & ARC-YC [76] & Return & CERL, TD3 & Continuous control \\  & & SOS-PPO [77] & Reward & PPO & Multi-objective \\  & & MERL [78] & Success rate & MADDPG, MATD3 & Multi-agent \\  & & EFC-MARL [79] & Reward & MADDPG & Multi-agent \\  & & EMOGI [80] & Win rate, duration & A3C & Multi-objective \\ \hline \multirow{6}{*}{Other EAs} & \multirow{4}{*}{Value function} & ECRL [81] & Reward & NA & Parameter optimization \\  & & AGPIR [82] & Q-value & NA & Continuous control \\  & & TDDQN [83] & Q-value & DQN & Multi-task \\  & & DDQN-RS [84] & MLHP & DDQN & Continuous control \\ \cline{2-6}  & \multirow{2}{*}{Policy gradient} & EGFRL [85] & Reward & NA & Multi-task \\  & & GPPL [40] & Error & NA & Continuous control \\  & & GPPLM [86] & Reward & NA & Continuous control \\ \cline{2-6}  & \multirow{2}{*}{Actor-critic} & FLR-RL [43] & NA & NA & Continuous control \\  & & TERT [87] & Convergence & ESAC, CEM-RL, ERL & Multi-objective \\  & & ERL-A2S [88] & Task result & PDERL, ERL, TD3, PPO & Multi-objective \\ \hline \hline \end{tabular}
\end{table}