# Evolutionary Reinforcement Learning: A Systematic Review and Future Directions
## 5 Challenges in Evolutionary Reinforcement Learning
### Issues Encountered by Reinforcement Learning
#### 5.1.1 Parameter Sensitivity
\begin{table}
\begin{tabular}{l l l l l} \hline
**Issue** & **Solution** & **Method** & **RL Algorithm** & **Evo Algorithm** \\ \hline \multirow{4}{*}{Parameter Sensitivity} & EA dynamically adjusts parameters & SBARL [4] & Policy gradient & Population-based training \\  & AAEDyS [42] & Policy gradient & Evolutionary computing \\ \cline{2-5}  & EA enhances parameter space diversity & GA-DRL [92] & Actor–critic & Genetic algorithms \\  & NS-MERL [93] & Actor–critic & Evolutionary strategy \\ \hline \multirow{3}{*}{Sparse Rewards} & EA accelerates search process & PS-T3D [75] & Actor–critic & Population-based training \\  & EAS enhances policy space diversity & PNS-RL [5] & Actor–critic & Evolutionary strategy \\ \cline{2-5}  & EA enhances policy space diversity & GEATL [94] & Actor–critic & Genetic algorithm \\  & RACE [95] & Actor–critic & Genetic algorithm \\ \hline \multirow{2}{*}{Local Optima} & \multirow{2}{*}{EA enhances policy space diversity} & EARL [73] & Policy gradient & Population-based training \\  & & G2 AC [96] & Actor–critic & Genetic algorithm \\  & DEPRL [71] & Actor–critic & Cross-entropy method \\ \hline Multi-Task Difficulties & EA enhances policy combination and synergy & EGPRL [85] & Policy gradient & Genetic programming \\ \hline \multirow{3}{*}{Policy Search} & \multirow{2}{*}{EA-integrated process for policy generation} & AGPRL [82] & Value function & Genetic programming \\  & & GPRL [40] & Policy gradient & Genetic programming \\  & & GPEDM [86] & Policy gradient & Genetic programming \\ \hline \multirow{2}{*}{Computational Efficiency} & \multirow{2}{*}{EA-integrated process for policy generation} & FiDi-RL [43] & Actor–critic & Random search \\  & & CERM-ACER [97] & Actor–critic & Cross-entropy method \\ \cline{1-1}  & & CGP [98] & Actor–critic & Cross-entropy method \\ \hline \end{tabular}
\end{table} 
Table 2: Issues encountered by reinforcement learning and corresponding EA solutions. combination for the proposed method. Ref. [92] presented Polyak-averaging coefficient updates, as shown below:  
\[ \begin{split}\theta^{Q^{\prime}}&\leftarrow\tau\theta^ {Q}+(1-\tau)\theta^{Q^{\prime}},\\ \theta^{\mu^{\prime}}&\leftarrow\tau\theta^{\mu}+(1- \tau)\theta^{\mu^{\prime}},\end{split}  \tag{13}\]  
where \(\theta^{Q^{\prime}}\) and \(\theta^{\mu^{\prime}}\) are the parameters of the target Q-network and target policy network, respectively. \(\theta^{Q}\) and \(\theta^{\mu}\) denote the parameters of the current Q-network and policy network, and \(\tau\) is the Polyak-averaging coefficient, a factor usually close to but greater than zero, used to blend the current and target network parameters. In addition to dynamic parameter adjustment, NS-MERL [93] relies on novelty search to guide agents to explore a broader state space rather than just optimizing a fixed set of parameters. In addition, NS-MERL combines gradient optimization and evolutionary search to balance individual-level exploration with team-level optimization, improving the learning efficiency and coordination ability in complex tasks.