Evolutionary Policy Optimization

## Abstract

Abstract On-policy reinforcement learning (RL) algorithms are widely used for their strong asymptotic performance and training stability, but they struggle to scale with larger batch sizes, as additional parallel environments yield redundant data due to limited policy-induced diversity. In contrast, Evolutionary Algorithms (EAs) scale naturally and encourage exploration via randomized population-based search, but are often sample-inefficient. We propose Evolutionary Policy Optimization (EPO), a hybrid algorithm that combines the scalability and diversity of EAs with the performance and stability of policy gradients. EPO maintains a population of agents conditioned on latent variables, shares actor-critic network parameters for coherence and memory efficiency, and aggregates diverse experiences into a master agent. Across tasks in dexterous manipulation, legged locomotion, and classic control, EPO outperforms state-of-the-art baselines in sample efficiency, asymptotic performance, and scalability. For visualizations of the learned policies, please visit: https://yifansu1301.github.io/EPO/ .

## 1 Introduction

Reinforcement learning (RL) has become a powerful paradigm for training autonomous decision-making agents across a wide range of domains, including games [40, 2, 47, 30], robotics [19, 1, 22, 17], and large-language-model alignment [33, 13, 46].
Like all trial-and-error methods, RL is inherently sample-inefficient, and the problem is particularly acute for *on-policy* algorithms, which discard past experience after each update.

Despite this drawback, on-policy model-free methods—commonly referred to as *policy gradients* [38, 30]—are widely adopted in real-world applications [40, 2, 19, 1, 33]. Their popularity stems from their ability to achieve higher *asymptotic* returns and *stable* performance, making them the method of choice in data-rich settings such as simulations and game engines [26], where large batches are easy to generate. Unfortunately, recent works [41, 34] show that policy-gradient algorithms do *not scale* well with larger batch sizes: because data are collected from the current policy, adding more parallel environments does not guarantee greater diversity. Instead, the data distribution quickly converges, so simply running more environments on additional GPUs yields highly correlated trajectories rather than useful variety.

Evolutionary algorithms (EAs) contribute the complementary strength of explicit *diversity* through random perturbations, making them natural partners for RL. Their combination—evolutionary RL (EvoRL) [31, 36]—maintains a population of policies and refines them with evolutionary operators. Yet EvoRL has not been widely adopted. Classic EvoRL relies on gradient-free evolutionary strategies (ES) [20, 3, 48], whose simplest variants reduce to random search [27]: they parallelize well but remain extremely sample-inefficient. Methods such as CEM-RL [35] introduce off-policy updates to improve efficiency but degrade asymptotic performance owing to distribution shift. Population-Based Training (PBT) [34, 18] ensembles multiple on-policy agents, yet each policy evolves in isolation, wasting the opportunity to amortize experience across the population.

Can we design an algorithm that (i) scales with the number of parallel environments, (ii) learns efficiently from shared experience, and (iii) preserves the high asymptotic returns and stability of on-policy methods? We answer in the affirmative. We introduce Evolutionary Policy Optimization (EPO), a novel policy-gradient algorithm that merges the strengths of EA and policy gradients: EA supplies diverse experience during massively parallel training, and experience from all agents is amortized to train a master agent via the split-and-aggregate policy-gradient (SAPG) formulation [41], which performs off-policy updates with importance sampling to fold follower data into the master’s on-policy set.

EPO maintains a population of agents alongside a master agent, all sharing the same network weights (analogous to gene-expression rules). This design prevents parameter bloat as the population scales, enabling efficient training and storage of large networks. Each agent is conditioned on a unique latent variable (“gene”), ensuring behavioral diversity while preserving coherence. Periodically, Darwinian selection [8] is applied: low performers are removed, and elite agents are retained. These elites undergo crossover and mutation, injecting controlled diversity without excessive divergence. All agents are updated synchronously with policy gradients using shared reward signals. In every iteration, EPO aggregates experience from all agents into the master via SAPG updates [41], enabling the master to learn from diverse, high-quality data and fully exploit massive parallelism.

We evaluate EPO on several challenging and realistic environments spanning manipulation (Isaac Gym Allegro-Kuka [26]), locomotion (ANYmal Walking [16], Unitree A1 Parkour [4, 49]), and classic control benchmarks (DeepMind Control Suite [45]). Across this suite, EPO significantly outperforms prior state-of-the-art RL methods. Moreover, our scaling-law analysis shows that EPO effectively scales with increased computational resources, making it a promising approach for large-scale RL training.

## 2 Related Work

#### On-Policy Reinforcement Learning

On-policy approaches [43] update the policy using data collected by the current policy, ensuring that the training data remains closely aligned with the policy’s behavior. This strategy often leads to higher asymptotic rewards and stable performance, but at the cost of sample inefficiency, as past experiences from older policies are discarded. Despite this drawback, on-policy methods are the preferred choice in settings where data is abundant, such as simulations and game engines, where large training batches can be efficiently generated. In practice, on-policy RL has proven highly effective for training autonomous decision-making agents in games [40, 2, 47, 30], robotics [19, 1, 22, 17], and more recently, in developing reasoning capabilities in large language models (LLMs) [33, 13, 46]. Two widely used algorithms in this category are Trust Region Policy Optimization (TRPO) [37] and Proximal Policy Optimization (PPO) [38], both designed to stabilize learning by constraining the magnitude of policy updates.

#### Off-Policy Reinforcement Learning

Unlike on-policy algorithms, off-policy methods allow for policy updates using data collected from different or older policies, enabling more efficient reuse of past experiences. Classic approaches such as Q-learning and its deep variant Deep Q-Network (DQN) [29] rely on a replay buffer to store state-action transitions for more stable learning. More recent continuous control algorithms, such as Deep Deterministic Policy Gradient (DDPG) [25, 39], Twin Delayed DDPG (TD3) [11], and Soft Actor-Critic (SAC) [14], extend these ideas within an actor-critic framework, further improving stability and performance. While off-policy methods are typically more sample-efficient than on-policy methods, they often suffer from distribution shift and overestimation of action values, which can hinder stable learning. To effectively amortize experience across all policies in a population while ensuring stable updates, we adopt the Split and Aggregate Policy Gradient (SAPG) formulation [41], which performs off-policy updates via importance sampling, enabling the aggregation of experience from individual follower policies into the on-policy data of a master policy.

#### Evolutionary Reinforcement Learning

Another related line of work is Evolutionary Reinforcement Learning (EvoRL) [31], which integrates population-based search with policy optimization to explore a broad range of candidate solutions. Historically, evolutionary computation (EC) has been employed to optimize neural network weights [44, 5, 6], architectures [12], and hyperparameters [18, 42]. For instance, OpenAI ES [7] applied a streamlined evolution strategy to Atari games. More recently, researchers have integrated gradient-based updates alongside evolutionary operations (e.g., mutation and crossover), improving scalability and leveraging the representational power of deep neural networks [35, 21]. For example, [20] extends Deep Deterministic Policy Gradient (DDPG) by introducing evolutionary operations on a population of policies, combining the exploratory benefits of evolution with gradient-based optimization for more robust learning. However, these methods remain vulnerable to distribution shift, which undermines their asymptotic performance. In this work, we propose to combine the strengths of evolutionary search with on-policy updates, enabling higher asymptotic rewards while maintaining high sample efficiency.

## 3 Preliminaries

In this paper, we propose Evolutionary Policy Optimization (EPO), a novel reinforcement learning algorithm built on the well-established on-policy RL method Proximal Policy Optimization (PPO) and Genetic Algorithm (GA). Below, we provide a brief overview of these foundational methods.

#### Reinforcement Learning

A standard reinforcement learning setting is formalized as a Markov Decision Process (MDP) and
consists of an agent interacting with an environment $E$ over a number of discrete time steps. At each
time step $t$, the agent receives a state $s_{t}$ and maps it to an action $a_{t}$ using its policy $\pi_{\theta}$. The agent
receives a scalar reward $r_{t}$ and moves to the next state $s_{t+1}$. The process continues until the agent
reaches a terminal state marking the end of an episode. The return $R_{t}=\sum_{k=0}^{\infty}\gamma^{k}r_{t+k}$ is the total accumulated return from time step $t$ with discount factor $\gamma\in(0,1]$. The goal of the agent is to maximize the expected return.

#### PPO

Proximal Policy Optimization is widely used in reinforcement learning applications. It is an actor-critic based policy gradient algorithm, where the key idea underlying policy gradients is to increase the probabilities of actions that lead to higher return and decrease the probabilities of actions that lead to lower return, iteratively refining the policy to achieve optimal performance.

Let $\pi_{\theta}$ denote a policy with parameters $\theta$, and $J(\pi_{\theta})$ denote the expected finite-horizon undiscounted return of the policy. $J(\pi_{\theta})$ is updated by:

$$ $\nabla_{\theta}J(\pi_{\theta})=\mathop{\mathbb{E}}\limits_{\begin{subarray}{c} s\sim d^{\pi_{\theta}}\\ a\sim\pi_{\theta}(\cdot\mid s)\end{subarray}}\left[\nabla_{\theta}\log\pi_{ \theta}(a\mid s)\,A^{\pi_{\theta}}(s,a)\right],$ (1) $$

where $A^{\pi_{\theta}}(s,a)$ is an advantage function that estimates the contribution of the transition to the gradient. A common choice is $A^{\pi_{\theta}}(s,a)=Q^{\pi_{\theta}}(s,a)-V^{\pi_{\theta}}(s)$, where $Q^{\pi_{\theta}}(s,a)$ and $V^{\pi_{\theta}}(s)$ are the estimated action-value and value functions, respectively. This form of update is termed an actor-critic update. Since we want the gradient of the error with respect to the current policy, only data from the current policy (on-policy) can be utilized. But these updates can be unstable because gradient estimates are high variance and the loss landscape is complex. An update step that is too large can degrade policy performance. Proximal Policy Optimization (PPO) modifies Eq. [1](https://arxiv.org/html/2503.19037v2#S3.E1) to restrict updates to remain within an approximate trust region where improvement is guaranteed:

$$ $L_{on}(\pi_{\theta})=\mathop{\mathbb{E}}\limits_{\pi_{\text{old}}}\left[\min \left(r_{t}(\pi_{\theta})A^{\pi_{old}}_{t},\,\text{clip}\left(r_{t}(\pi_{ \theta}),1-\epsilon,1+\epsilon\right)A^{\pi_{old}}_{t}\right)\right],$ (2) $$

where $r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}\mid s_{t})}{\pi_{\text{old}}(a_{t}\mid s
_{t})}$, $\epsilon$ is a clipping hyperparameter, and $\pi_{\text{old}}$ is the policy collecting the on-policy data. The clipping operation ensures that the updated $\pi_{\theta}$ stays close to $\pi_{\text{old}}$. Empirically, given large numbers of samples, PPO achieves high performance and is stable and robust to hyperparameters.

#### Genetic Algorithm (GA)

is one of the most well-known Evolutionary Algorithms (EAs). It relies on three key operators: selection, mutation, and crossover. A core concept in genetic algorithms is the population, where each individual represents a potential solution evaluated using a fitness function. The classic genetic algorithm emphasizes crossover as the primary mechanism for exploration [15] and is usually applied to deal with the problem of hyper-parameter tuning in reinforcement learning.

## 4 Approach

The core idea behind Evolutionary Policy Optimization (EPO) is to integrate the population-based exploration of Genetic Algorithms (GAs) with the powerful policy gradient. While our framework specifically combines GA with Proximal Policy Optimization (PPO), it can be extended to any on-policy RL algorithm utilizing an actor-critic architecture.

EPO (As shown in Figure [1](https://arxiv.org/html/2503.19037v2#alg1)) begins by initializing a population of agents, where each agent is represented by a unique latent embedding (gene), while all agents share a common actor-critic network (gene-expression rules). Among these agents, one is designated as the “master” agent. Each agent then interacts with the environment for a full episode, and its fitness value (defined differently for each task) is obtained during that episode. A selection operator then determines which agents survive to the next generation based on their fitness scores. The embeddings of the surviving agents undergo mutation and crossover to form the next generation, while a small subset of top-performing agents (“elites”) is preserved unchanged to maintain stability.

To further enhance learning efficiency, we adopted a hybrid policy update strategy [41]. The master agent updates its parameters using both (i) on-policy data collected in the most recent episode and (ii) off-policy data sampled from the trajectories of the entire population. Meanwhile, each non-master agent is updated using only its own on-policy data. This hybrid learning process ensures both efficient exploration and sample reuse, making EPO well-suited for large-scale reinforcement learning. In the following sections, we describe each component of EPO in detail.

### 4.1 Genetic Algorithm for Agent Selection

The design philosophy of our algorithm is twofold: (1) to generate diverse yet bounded data for training the master agent and (2) to ensure that non-master agents are updated efficiently, enabling them to produce useful yet diverse data. Drawing inspiration from biological evolution, we propose initializing a population of agents ($\{\pi_{k}\}_{k=1}^{K}$), where each agent is associated with a unique latent embedding ($\{\phi_{k}\}_{k=1}^{K}$). All agents share a common actor-critic network, parameterized by $\theta$ and $\psi$, respectively. The actor is conditioned on both the current state and its latent embedding to generate actions, while the value function is similarly conditioned to estimate value functions. Prior works have also leveraged latent-conditioned policies to encourage diverse behaviors [9, 10]. Among these agents, one is designated as the master agent—for simplicity, we define the master agent as $\pi_{1}(\theta,\psi,\phi_{1})$, though the general framework allows for alternative assignments.

To update non-master agents, we employ a genetic algorithm (GA) that modifies only the latent embeddings while keeping network parameters fixed. Each agent interacts with the environment for a full episode, and gets its fitness values. A selection operator determines which agents survive to the next generation, with survival probability proportional to their relative fitness scores (i.e., cumulative reward, success rate, etc.). Specifically, we retain the top $x$ ranked non-master agents ($x\in[2,K]$) as elites, forming a set $X$, which is preserved unchanged and carried over to the next generation set $Y$. We then randomly select two agents from $X$ to undergo crossover and mutation. While multiple crossover methods exist (e.g., k-point crossover, uniform crossover), we use a simple averaging method for efficiency. Given two selected latent embeddings, $\phi_{i},\phi_{j}$, where $\pi_{i}(\theta,\psi,\phi_{i}),\pi_{j}(\theta,\psi,\phi_{j})\in X$, the crossover result is computed as: $\phi^{\prime}=1/2\times(\phi_{i}+\phi_{j})$. Following crossover, a mutation step is applied by adding Gaussian noise: $\phi^{\prime\prime}=\phi^{\prime}+\epsilon,\epsilon\sim\mathcal{N}(0,\sigma^{2})$. The new agent $\pi(\theta,\psi,\phi^{\prime\prime})$ is then added to $Y$ for the next generation.

It is important to note that for more complex tasks, such as dexterous manipulation, policies require longer training before they exhibit meaningful differences in performance. In such cases (e.g., when all rewards are close to zero), performing genetic updates too early may be ineffective. To address this, we execute selection, crossover, and mutation only when the agents’ performance differences become sufficiently significant. Specifically, the genetic algorithm is applied only when the difference between fitness scores exceeds a certain proportion of the median fitness score, ensuring that evolutionary updates occur only when meaningful behavioral differentiation has emerged.

$$ $\max\{f_{k}\}_{k=2}^{K}-\min\{f_{k}\}_{k=2}^{K}>\gamma\cdot\text{median}\{f_{k }\}_{k=2}^{K}$ (3) $$

, where $f_{k}$ represent the fitness score of agent $\pi_{k}$. A simpler alternative is to apply selection, crossover, and mutation at fixed intervals—e.g., every set number of environment steps.

All parameters ($\theta$, $\psi$, $\{\phi_{k}\}_{k=1}^{K}$) are updated using the following hybrid policy optimization process.

### 4.2 Hybrid-Policy Optimization

Different agents generate diverse data through varied behaviors, which can be beneficial if the master agent can effectively learn from all available data. However, since data distributions collected by different agents may vary significantly, incorporating off-policy data requires careful correction. Following [28, 41], we apply importance sampling to reweight updates from different policies. Specifically, we sample $|S_{1}|$ transitions from $\cup_{k=2}^{K}S_{k}$ collected by agents $\{\pi_{k}\}_{k=2}^{K}$ to get $S^{\prime}_{1}$. To update the master agent $\pi_{1}$ using $S^{\prime}_{1}$, we define the off-policy loss as:

$$ $\displaystyle L_{\text{off}}(\pi_{1},S^{\prime}_{1})=\frac{1}{|S^{\prime}_{1}| }\mathop{\mathbb{E}}\limits_{(s,a)\sim S^{\prime}_{1}}[\min(r_{\pi_{1}}(s,a)A^ {\pi_{1},\text{old}}(s,a),\quad\operatorname{clip}(r_{\pi_{1}}(s,a),\mu(1- \epsilon),\mu(1+\epsilon))A^{\pi_{1},\text{old}}(s,a))]$ (4) $$

where $r_{\pi_{1}}(s,a)$ is the importance sampling ratio, given by $r_{\pi_{1}}(s,a)=\frac{\pi_{1}(s,a)}{\pi_{k}(s,a)}$ and $\mu$ is an off-policy correction term, given by $\mu=\frac{\pi_{1,old}(s,a)}{\pi_{k}(s,a)}$. The final policy loss combines this off-policy loss with the on-policy loss given by Equation [2](https://arxiv.org/html/2503.19037v2#S3.E2):

$$ $L(\pi_{1})=L_{on}(\pi_{1})+\lambda\cdot L_{off}(\pi_{1},S^{\prime}_{1})$ (5) $$

#### Critic Update with Hybrid Data

The target value for the critic is computed using $n$-step returns (where $n=3$) for on-policy data:

$$ $\hat{V}_{\text{on},\pi_{k}}^{\text{target}}(s_{t})=\sum_{m=t}^{t+2}\gamma^{m-t }r_{m}+\gamma^{3}V_{\pi_{k},\text{old}}(s_{t+3})$ (6) $$

The critic loss for on-policy data is then:

$$ $L^{\text{critic}}_{\text{on}}(\pi_{k})=\mathop{\mathbb{E}}\limits_{(s,a)\sim \pi_{k}}\left[\left(V_{\pi_{k}}(s)-\hat{V}_{\text{on},\pi_{k}}^{\text{target}} (s)\right)^{2}\right]$ (7) $$

For off-policy data, however, multi-step returns are not directly applicable. Instead, we approximate the $1$-step return:

$$ $\hat{V}_{\text{off},\pi_{1}}^{\text{target}}(s^{\prime}_{t})=r_{t}+\gamma V_{ \pi_{1},\text{old}}(s^{\prime}_{t+1})$ (8) $$

The critic loss for off-policy data is then:

$$ $L^{\text{critic}}_{\text{off}}(\pi_{1},S^{\prime}_{1})=\frac{1}{|S^{\prime}_{1 }|}\mathop{\mathbb{E}}\limits_{(s,a)\sim S^{\prime}_{1}}\left[\left(V_{\pi_{1} }(s)-\hat{V}_{\text{off},\pi_{1}}^{\text{target}}(s)\right)^{2}\right]$ (9) $$

The final critic loss is a weighted combination of the on-policy and off-policy losses:

$$ $L^{\text{critic}}(\pi_{1})=L^{\text{critic}}_{\text{on}}(\pi_{1})+\lambda\cdot L ^{\text{critic}}_{\text{off}}(\pi_{1},S^{\prime}_{1})$ (10) $$

#### Updates for Non-Master Agents

All agents, except for the master agent $\pi_{1}$, are updated exclusively using on-policy losses, as defined in Equations [2](https://arxiv.org/html/2503.19037v2#S3.E2) and [6](https://arxiv.org/html/2503.19037v2#S4.E6). This ensures that while the master agent leverages additional experience from off-policy data, the remaining agents maintain stability by learning from their own trajectories, thereby preserving diversity within the population.

Figure: Algorithm 1 Evolutionary Policy Optimization

## 5 Experiments

Figure: Figure 2: We evaluate our algorithm on seven challenging environments that span a diverse set of tasks, including manipulation [34], locomotion [4], and classic control benchmarks [45]
Refer to caption: extracted/6530194/figures/experiment_setting.png

We evaluate our method on eight challenging and realistic environments, including manipulation [34], locomotion [16, 4], and classic control benchmarks [45], comparing it against state-of-the-art (SOTA) approaches. Additionally, we conduct ablation studies to assess the contribution of each component. To demonstrate scalability, we analyze our method’s performance with increasing computational resources. Finally, we evaluated computational cost versus performance gain as EPO scales with agent numbers (Supplementary Material). Each aspect is discussed in detail.

### 5.1 Experimental Setup

Tasks We evaluate our algorithm across eight challenging environments covering a diverse range of tasks, including manipulation [34]—Single-Arm Regrasping (M:SG), Single-Arm Throwing (M:ST), Two-Arm Regrasping (M:TG), Two-Arm Reorientation (M:TR), and Two-Arm Reorientation with Obstacle (M:TO); locomotion—ANYmal Walking (L:A)[16] and Unitree A1 Parkour (L:U)[4]; and classic control—Humanoid from the DeepMind Control Suite (D)[45]. A visual overview of these tasks is shown in Figure[2](https://arxiv.org/html/2503.19037v2#S5.F2); additional details are provided in the Appendix.

Following prior works [34, 41], we use the number of successes per episode as the performance metric for manipulation tasks. For L:A and D, we report the cumulative rewards as defined in their original benchmarks. For L:U, we use the average terrain difficulty level achieved by all A1 agents during an episode, which serves as a proxy for success rate.

Figure: Figure 3: Performance curves of EPO compared to SAC, PQL, PPO, SAPG, PBT, and CEM-RL baselines. EPO demonstrates superior sample efficiency and higher asymptotic performance, particularly on challenging tasks.
Refer to caption: extracted/6530194/figures/main_result.png

Baselines We evaluate our approach against state-of-the-art RL methods tailored for large-scale training. The comparisons include off-policy algorithms—SAC [14] and Parallel Q-Learning [24]; on-policy methods—PPO [38]; hybrid-policy methods—SAPG [41]; and population-based EvoRL methods—PBT [34] and CEM-RL [35]. We use the original implementations of each method without introducing algorithm-level modifications. Please refer to the supplementary material for additional details.

For a fair comparison, all algorithms are trained using the same number of environments ($N=24{,}576$) and identical network architectures. For methods incorporating a population-based component (SAPG, PBT, CEM-RL), we standardize the population size to 64. Since the impact of the number of agents on final performance can vary, we also report results using each method’s original agent configuration for completeness (e.g., PBT: 8 agents; SAPG: 6 agents). Each experiment is run with 5 different random seeds, and we report the mean and standard error in the plots. Further ablation studies on the effect of population size are presented in Section [5.3](https://arxiv.org/html/2503.19037v2#S5.SS3).

Table: Table 1: Performance of EPO compared to SAC, PQL, PPO, SAPG, PBT, and CEM-RL across a diverse set of tasks. In manipulation tasks, PPO, SAPG, and PBT demonstrate learning capabilities in simple scenarios; however, only EPO achieves significant learning progress on complex tasks. For locomotion tasks, where PPO is widely regarded as a strong baseline and the default method, EPO still achieves superior performance by a large margin on the more challenging problems.

Figure: Figure 4: Ablation and scaling results for EPO. (a) Ablation of agent number ($K=8,16,32,64$) with a fixed total number of environments ($N=24{,}576$). As the number of agents increases, EPO’s performance improves. (b) Training and scaling curves with increasing environments. With a fixed number of environments per agent (384), total performance improves as environments scale.
Refer to caption: extracted/6530194/figures/ablation_scale.png

### 5.2 Results and Analysis

As shown in Figure [3](https://arxiv.org/html/2503.19037v2#S5.F3) and Table [1](https://arxiv.org/html/2503.19037v2#S5.T1), our algorithm consistently and significantly outperforms all baselines. For more visualization, please refer to [https://yifansu1301.github.io/EPO/](https://yifansu1301.github.io/EPO/).

In manipulation tasks, while methods like PPO, SAPG, and PBT perform reasonably on simpler tasks (e.g., single-arm regrasping and throwing), EPO still achieves higher success rates. For more complex tasks requiring two-arm coordination, baselines fail to learn meaningful behavior even after $1\times 10^{10}$ steps, whereas EPO achieves strong performance. For example, EPO reaches 35.8 episode successes in two-arm reorientation using $1\times 10^{10}$ steps, surpassing SAPG’s 28.58 after $5\times 10^{10}$ steps in the original SAPG paper [41], highlighting EPO’s superior sample efficiency. A similar trend holds in locomotion. On flat-ground ANYmal walking, baselines match EPO, but in more complex parkour tasks requiring rich exploration, EPO significantly outperforms them. In the DeepMind Control Suite (Humanoid), all methods perform reasonably, yet EPO still yields marginally higher returns.

Comparing SAPG (5th column) with SAPG 6 (6th column) reveals that hybrid-policy-only updates do not benefit from large-scale simulation and may even degrade performance, as increasing the number of agents introduces a higher proportion of poorly performing policies. In contrast, comparing SAPG (5th column) with EPO (last column) highlights the critical role of the genetic algorithm (Table [1](https://arxiv.org/html/2503.19037v2#S5.T1)), especially in more challenging tasks like two-arm manipulation. By filtering out underperforming agents and learning only from high-quality data, the GA ensures stable and effective training. This demonstrates that the genetic algorithm is not a minor enhancement, but a fundamental component of the hybrid-policy update.

Notably, EPO demonstrates a lower performance variance across different random seeds compared to baseline methods, indicating strong robustness to hyperparameter variations, including random seed initialization. This is especially significant given the well-known sensitivity of reinforcement learning algorithms to hyperparameter settings. EPO’s robustness makes it a compelling choice for training RL models with minimal hyperparameter tuning, addressing a major challenge in deploying RL in real-world applications.

Compared to EvoRL baselines, EPO demonstrates superior learning efficiency and asymptotic performance. While PBT performs well on simpler tasks, its effectiveness diminishes in more complex scenarios. This is largely due to PBT’s requirement to train each policy nearly to convergence before selecting optimal hyperparameters, along with its inability to leverage data from past agents—resulting in inefficient use of experience. Consequently, PBT does not benefit from larger population sizes, as evidenced by the similar performance between PBT and PBT 8. In contrast, CEM-RL relies heavily on off-policy updates, which becomes a limitation in difficult tasks where most agents perform poorly. This leads to the accumulation of low-quality data and ultimately degrades asymptotic performance across a wide range of tasks. EPO overcomes these limitations by employing a hybrid-policy update mechanism while retaining the genetic algorithm for agent selection, particularly in complex environments.

### 5.3 Ablations

In this ablation study, we investigate the impact of population size while keeping the total computational budget fixed, using the single-arm reorientation task—where the agent must grasp an object and reorient it to a target 6-DoF pose—as the testbed. Unless otherwise specified, all subsequent analyses in this section are conducted within this environment. Specifically, we fix the number of environments to $N=24{,}576$ and vary the population size from $K=8$ to $K=64$ (Figure [4](https://arxiv.org/html/2503.19037v2#S5.F4)(a)). The results indicate that our algorithm benefits from a larger population. While this may seem intuitive, it is not a guaranteed outcome—larger populations introduce greater behavioral diversity, which can also increase the prevalence of poorly performing agents and result in noisier data. To address this, EPO applies Genetic Algorithms (GA) at the latent space level to periodically eliminate underperforming agents, ensuring that only high-quality data contributes to learning. Moreover, since all agents share a common actor-critic network, we maintain controlled diversity while avoiding unbounded policy divergence—an essential property for stable and scalable training.

We also analyzed the impact of different crossover strategies beyond simple averaging. Specifically, we experimented with two alternative approaches—uniform crossover and fitness-weighted averaging. In uniform crossover, each element of the latent vector is randomly selected from one of the two parents with equal probability. In the fitness-weighted average approach, the parents’ latent vectors are combined proportionally to their fitness scores. Our method achieved a 36.7% success rate, compared to 32.6% for uniform crossover and 37.0% for fitness-weighted averaging, indicating that performance is relatively robust to the choice of crossover strategy. And for simplicity, we keep the averaging strategy.

### 5.4 Scaling Law

Here, we examine the scalability of EPO with increased computational resources. Specifically, we evaluate EPO’s scalability with larger training batches generated by parallel simulations by fixing the number of environments per agent and increasing the number of agents from $K=16$ to $K=128$, thereby scaling the total number of environments from $N=6,144$ to $N=49,152$. It is worth noting that even at $K=16$, the batch size of $N=6,144$ is already significantly large. As shown in Figure [4](https://arxiv.org/html/2503.19037v2#S5.F4)(b), EPO continues to improve with increasing training data, even when the number of environments reaches $N=49,152$, demonstrating that our algorithm overcomes the limitations of existing on-policy RL methods, which typically saturate beyond a certain batch size. This highlights EPO’s strong potential for data-rich environments, such as simulations and game engines, where large training batches can be easily generated and leveraged effectively.

It is worth noting that our method focuses on scaling with training data, which is orthogonal to efforts that scale performance through larger neural network architectures. This makes it naturally compatible with existing regularization techniques designed for training large networks [32, 23]. Additionally, the shared actor-critic architecture enhances memory efficiency and inherently supports scaling to larger network sizes.

## 6 Conclusion

In conclusion, we present Evolutionary Policy Optimization (EPO), a novel policy gradient algorithm that integrates evolutionary algorithms with policy optimization. We demonstrate that EPO significantly outperforms state-of-the-art baselines across several challenging RL benchmarks while also scaling effectively with increased computational resources. Our approach enables large-scale RL training while maintaining high asymptotic performance, and we hope it inspires future research in advancing scalable reinforcement learning.

## References

- Akkaya et al. [2019]
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al.
Solving rubik’s cube with a robot hand.
*arXiv preprint arXiv:1910.07113*, 2019.
- Berner et al. [2019]
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al.
Dota 2 with large scale deep reinforcement learning.
*arXiv preprint arXiv:1912.06680*, 2019.
- Chen et al. [2019]
Zefeng Chen, Yuren Zhou, Xiaoyu He, and Siyu Jiang.
A restart-based rank-1 evolution strategy for reinforcement learning.
In *IJCAI*, pages 2130–2136, 2019.
- Cheng et al. [2024]
Xuxin Cheng, Kexin Shi, Ananye Agarwal, and Deepak Pathak.
Extreme parkour with legged robots.
In *2024 IEEE International Conference on Robotics and Automation (ICRA)*, pages 11443–11450. IEEE, 2024.
- Choromanski et al. [2018]
Krzysztof Choromanski, Mark Rowland, Vikas Sindhwani, Richard Turner, and Adrian Weller.
Structured evolution with compact architectures for scalable policy optimization.
In *International Conference on Machine Learning*, pages 970–978. PMLR, 2018.
- Choromanski et al. [2019]
Krzysztof M Choromanski, Aldo Pacchiano, Jack Parker-Holder, Yunhao Tang, and Vikas Sindhwani.
From complexity to simplicity: Adaptive es-active subspaces for blackbox optimization.
*Advances in Neural Information Processing Systems*, 32, 2019.
- Chrabaszcz et al. [2018]
Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter.
Back to basics: benchmarking canonical evolution strategies for playing atari.
In *Proceedings of the 27th International Joint Conference on Artificial Intelligence*, pages 1419–1426, 2018.
- Darwin [2023]
Charles Darwin.
Origin of the species.
In *British Politics and the environment in the long nineteenth century*, pages 47–55. Routledge, 2023.
- Ebert et al. [2021]
Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine.
Bridge data: Boosting generalization of robotic skills with cross-domain datasets.
*arXiv preprint arXiv:2109.13396*, 2021.
- Eysenbach et al. [2018]
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine.
Diversity is all you need: Learning skills without a reward function.
*arXiv preprint arXiv:1802.06070*, 2018.
- Fujimoto et al. [2018]
Scott Fujimoto, Herke Hoof, and David Meger.
Addressing function approximation error in actor-critic methods.
In *International conference on machine learning*, pages 1587–1596. PMLR, 2018.
- Gaier and Ha [2019]
Adam Gaier and David Ha.
Weight agnostic neural networks.
*Advances in neural information processing systems*, 32, 2019.
- Guo et al. [2025]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Haarnoja et al. [2018]
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.
Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.
In *International conference on machine learning*, pages 1861–1870. PMLR, 2018.
- Hoffmeister and Bäck [1990]
Frank Hoffmeister and Thomas Bäck.
Genetic algorithms and evolution strategies: Similarities and differences.
In *International conference on parallel problem solving from nature*, pages 455–469. Springer, 1990.
- Hutter et al. [2016]
Marco Hutter, Christian Gehring, Dominic Jud, Andreas Lauber, C Dario Bellicoso, Vassilios Tsounis, Jemin Hwangbo, Karen Bodie, Peter Fankhauser, Michael Bloesch, et al.
Anymal-a highly mobile and dynamic quadrupedal robot.
In *2016 IEEE/RSJ international conference on intelligent robots and systems (IROS)*, pages 38–44. IEEE, 2016.
- Hwangbo et al. [2019]
Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter.
Learning agile and dynamic motor skills for legged robots.
*Science Robotics*, 4(26):eaau5872, 2019.
- Jaderberg et al. [2017]
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al.
Population based training of neural networks.
*arXiv preprint arXiv:1711.09846*, 2017.
- Kalashnikov et al. [2018]
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al.
Scalable deep reinforcement learning for vision-based robotic manipulation.
In *Conference on robot learning*, pages 651–673. PMLR, 2018.
- Khadka and Tumer [2018]
Shauharda Khadka and Kagan Tumer.
Evolution-guided policy gradient in reinforcement learning.
*Advances in Neural Information Processing Systems*, 31, 2018.
- Khadka et al. [2019]
Shauharda Khadka, Somdeb Majumdar, Tarek Nassar, Zach Dwiel, Evren Tumer, Santiago Miret, Yinyin Liu, and Kagan Tumer.
Collaborative evolutionary reinforcement learning.
In *International conference on machine learning*, pages 3341–3350. PMLR, 2019.
- Kumar et al. [2021]
Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik.
Rma: Rapid motor adaptation for legged robots.
*Robotics: Science and Systems XVII*, 2021.
- Lee et al. [2024]
Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R Wurman, Jaegul Choo, Peter Stone, and Takuma Seno.
Simba: Simplicity bias for scaling up parameters in deep reinforcement learning.
*arXiv preprint arXiv:2410.09754*, 2024.
- Li et al. [2023]
Zechu Li, Tao Chen, Zhang-Wei Hong, Anurag Ajay, and Pulkit Agrawal.
Parallel $q$-learning: Scaling off-policy reinforcement learning under massively parallel simulation.
In *International Conference on Machine Learning*, pages 19440–19459. PMLR, 2023.
- Lillicrap [2015]
TP Lillicrap.
Continuous control with deep reinforcement learning.
*arXiv preprint arXiv:1509.02971*, 2015.
- [26]
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al.
Isaac gym: High performance gpu based physics simulation for robot learning.
In *Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)*.
- Mania et al. [2018]
Horia Mania, Aurelia Guy, and Benjamin Recht.
Simple random search provides a competitive approach to reinforcement learning.
*arXiv preprint arXiv:1803.07055*, 2018.
- Meng et al. [2023]
Wenjia Meng, Qian Zheng, Gang Pan, and Yilong Yin.
Off-policy proximal policy optimization.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 37, pages 9162–9170, 2023.
- Mnih [2013]
Volodymyr Mnih.
Playing atari with deep reinforcement learning.
*arXiv preprint arXiv:1312.5602*, 2013.
- Mnih [2016]
Volodymyr Mnih.
Asynchronous methods for deep reinforcement learning.
*arXiv preprint arXiv:1602.01783*, 2016.
- Moriarty et al. [1999]
David E Moriarty, Alan C Schultz, and John J Grefenstette.
Evolutionary algorithms for reinforcement learning.
*Journal of Artificial Intelligence Research*, 11:241–276, 1999.
- Nauman et al. [2024]
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłoś, and Marek Cygan.
Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control.
*arXiv preprint arXiv:2405.16158*, 2024.
- Ouyang et al. [2022]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
Training language models to follow instructions with human feedback.
*Advances in neural information processing systems*, 35:27730–27744, 2022.
- Petrenko et al. [2023]
Aleksei Petrenko, Arthur Allshire, Gavriel State, Ankur Handa, and Viktor Makoviychuk.
Dexpbt: Scaling up dexterous manipulation for hand-arm systems with population based training.
*arXiv preprint arXiv:2305.12127*, 2023.
- Pourchot and Sigaud [2019]
Aloïs Pourchot and Olivier Sigaud.
Cem-rl: Combining evolutionary and gradient-based methods for policy search.
In *7th International Conference on Learning Representations, ICLR 2019*, 2019.
- Salimans et al. [2017]
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever.
Evolution strategies as a scalable alternative to reinforcement learning.
*arXiv preprint arXiv:1703.03864*, 2017.
- Schulman et al. [2015]
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz.
Trust region policy optimization.
In Francis Bach and David Blei, editors, *Proceedings of the 32nd International Conference on Machine Learning*, volume 37 of *Proceedings of Machine Learning Research*, pages 1889–1897, Lille, France, 07–09 Jul 2015. PMLR.
URL [https://proceedings.mlr.press/v37/schulman15.html](https://proceedings.mlr.press/v37/schulman15.html).
- Schulman et al. [2017]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
*arXiv preprint arXiv:1707.06347*, 2017.
- Silver et al. [2014]
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller.
Deterministic policy gradient algorithms.
In *International conference on machine learning*, pages 387–395. Pmlr, 2014.
- Silver et al. [2016]
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.
Mastering the game of go with deep neural networks and tree search.
*nature*, 529(7587):484–489, 2016.
- Singla et al. [2024]
Jayesh Singla, Ananye Agarwal, and Deepak Pathak.
Sapg: split and aggregate policy gradients.
*arXiv preprint arXiv:2407.20230*, 2024.
- Snoek et al. [2012]
Jasper Snoek, Hugo Larochelle, and Ryan P Adams.
Practical bayesian optimization of machine learning algorithms.
*Advances in neural information processing systems*, 25, 2012.
- Sutton et al. [1999]
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour.
Policy gradient methods for reinforcement learning with function approximation.
*Advances in neural information processing systems*, 12, 1999.
- Tang et al. [2020]
Yunhao Tang, Krzysztof Choromanski, and Alp Kucukelbir.
Variance reduction for evolution strategies via structured control variates.
In *International Conference on Artificial Intelligence and Statistics*, pages 646–656. PMLR, 2020.
- Tassa et al. [2018]
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al.
Deepmind control suite.
*arXiv preprint arXiv:1801.00690*, 2018.
- Team et al. [2023]
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al.
Gemini: a family of highly capable multimodal models.
*arXiv preprint arXiv:2312.11805*, 2023.
- Vinyals et al. [2019]
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al.
Grandmaster level in starcraft ii using multi-agent reinforcement learning.
*nature*, 575(7782):350–354, 2019.
- Zheng and Cheng [2023]
Bowen Zheng and Ran Cheng.
Rethinking population-assisted off-policy reinforcement learning.
In *Proceedings of the Genetic and Evolutionary Computation Conference*, pages 624–632, 2023.
- Zhuang et al. [2023]
Ziwen Zhuang, Zipeng Fu, Jianren Wang, Christopher Atkeson, Sören Schwertfeger, Chelsea Finn, and Hang Zhao.
Robot parkour learning.
In *Conference on Robot Learning (CoRL)*, 2023.

## Appendix A Computational Cost

We also evaluate the computational cost of EPO compared with other baselines that involve a population-based component (SAPG, PBT, CEM-RL). Since EPO requires calculating fitness score of all agents and determining whether iteration is necessary, it slightly increases the computational cost over PBT and SAPG. However, simply increasing the number of agents in EPO does not have a significant impact on memory usage or inference time. This is because the additional cost mainly comes from adding a small number of latent embeddings, which does not drastically affect the overall resource usage. All results are obtained on task M:SO with 24,576 environments and $5e9$ environment steps.

## Appendix B Experiment Details

In this section, we present more details of our experiments.

### B.1 Task and Evaluation Details

- •
Manipulation: Single-Arm Regrasping (M:SG): The agent must lift a cube from the table and hold it near a randomly generated target point (shown as a white sphere in Figure [2](https://arxiv.org/html/2503.19037v2#S5.F2)) within 30 steps. A success is defined as the center distance between the cube and the target point being within a predefined threshold.
- •
Manipulation: Single-Arm Throwing (M:ST): The agent need pick up the cube and throw it to a bucket at a random position. A succcess is defined as if the cube is throwed into the bucket.
- •
Manipulation: Single-Arm Reorientation (M:SO - used for ablations): The agent must pick up an object and reorient it to a specified 6-DoF pose including both position and orientation. A success is defined as both the positional and orientation differences between the object and the target pose being within a threshold.
- •
Manipulation: Two-Arm Regrasping (M:TG): The goal is the same as in one-arm regrasping. However, the generated goal pose is far from the initial pose, requiring two Kuka arms to collaborate to transfer (e.g. throwing) the object to reach target poses in different regions.
- •
Manipulation: Two-Arm Reorientation (M:TR): The goal is the same as in one-arm reorientation. However, the target pose is placed far from the initial pose, necessitating coordinated actions between the two Kuka arms.
- •
Manipulation: Two-Arm Reorientation with Obstacle (M:TO): The task shares the same objective as two-arm reorientation, but with an obstacle in the center of the table. The agent must learn to pass the cube through the obstacle in order to successfully align it with the target orientation.
- •
Locomotion: ANYmal (L:A): A quadruped robot is tasked with following velocity commands across flat terrain.
- •
Locomotion: Unitree A1 Parkour (L:U): In this task, the robot dog needs to learn how to traverse complex terrains, such as hurdles, gaps, rough terrain, etc. The robot dog will advance to the next, more difficult terrain only after it can successfully navigate the current one. Therefore, we consider the average terrain difficulty level that the robot dogs are on as the success rate. A value of 1 means that, on average, all robot dogs can complete the terrain at the first difficulty level.
- •
DeepMind Control Suite: Humanoid (D): A 21-joint humanoid must run forward at a target velocity of 10 m/s without falling.

### B.2 Hyperparameters and Network

#### Manipulation Tasks

We employ a Gaussian policy in which the mean is generated by a single-layer LSTM with 768 hidden units. The input observation is first processed by an MLP with hidden layers of sizes 768, 512, and 256, using ELU activation, before being fed into the LSTM.

| Hyperparameter | Value |
| --- | --- |
| Discount factor | $0.99$ |
| Learning rate | $1e-4$ |
| KL threshold for LR update | $0.016$ |
| Num Envs | $24,576$ |
| Mini-batch size | $4\cdot$ num_envs |
| Grad norm | 1.0 |
| Clipping factor | 0.1 |
| Critic coefficient | 4.0 |
| Horizon length | 16 |
| LSTM Sequence length | 16 |
| Bounds loss coefficient | 0.00001 |
| Mini Epochs | 2 |
| $\gamma$ in Eq.(3) | 0.3 |
| $\lambda$ in Eq.(5) | 1.0 |
| $x$ in Line 13 of Algo.1 | num_agents$(K)-2$ |

#### Locomotion Task

We adopt a Gaussian policy in which the mean is produced by a MLP with hidden layer sizes of 512, 256, and 128, using ELU activation functions.

| Hyperparameter | Value |
| --- | --- |
| Discount factor | $0.99$ |
| Learning rate | $1e-4$ |
| KL threshold for LR update | $0.016$ |
| Num Envs | $24,576$ |
| Mini-batch size | $4\cdot$ num_envs |
| Grad norm | 1.0 |
| Clipping factor | 0.1 |
| Critic coefficient | 4.0 |
| Horizon length | 16 |
| Bounds loss coefficient | 0.00001 |
| Mini Epochs | 2 |
| $\gamma$ in Eq.(3) | 0.5 |
| $\lambda$ in Eq.(5) | 1.0 |
| $x$ in Line 13 of Algo.1 | num_agents$(K)-2$ |

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 2
presentation: 2
questions: 1. For crossover operation, the authors mentioned “we use a simple averaging method for efficiency”. This averaging strategy does not fit biological intuition. An experimental comparison on different choices of crossover operators should be included.
2. How is the mutation noise decided and how do different choices influence the results?
3. The authors mentioned “policies require longer training before they exhibit meaningful differences in performance … we execute selection, crossover, and mutation only when the agents’ performance differences become sufficiently significant” near Line 249. Since genetic operations are not applied until the differences are significant enough, where do the differences come from? This is somewhat counter-intuitive because it is waiting for sufficient diversity/differences to apply the genetic algorithm which aims to enhance diversity.
4. Can the authors provide the analysis on the differences or diversity among policies/genes/the trajectories generated by different policies? In my opinion, this should be a very critical part of the experiments.
soundness: 1
strengths: - The presentation of the proposed method is almost clear.
summary: This paper proposes an approach of combining Evolutionary Algorithms and Reinforcement Learning, called EPO. EPO maintains a population of policies, consisting of a master policy that learns in an off-policy manner and other policies are optimized via EA. The policies in the population share the policy and critic networks, and each policy owns its unique gene embedding. The proposed method is empirically evaluated in 8 tasks, including dexterous manipulation, legged locomotion, and classic control.
weaknesses: - The motivation of this work does not make sense to me. I understand the value of combining EA and RL. But the authors motivate this point by mentioning that on-policy RL does not scale well with larger batch sizes. However, off-policy RL is not discussed in this logic chain.
- Combining EA and RL has been well studied and a comprehensive survey is provided in [7], as well as other related topics like RL with parallel/population environments. However, important related literature is largely ignored in this paper.
    - RL with vectorized environments
        - [1] Simplifying Deep Temporal Difference Learning. https://arxiv.org/abs/2407.04811
        - [2] FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control. h[ttps://arxiv.org/abs/2505.22642](https://arxiv.org/abs/2505.22642)
    - Evolutionary RL
        - [3] Policy optimization by genetic distillation. ICLR 2018
        - [4] EvoRainbow: Combining Improvements in Evolutionary Reinforcement Learning for Policy Search. ICML 2024.
        - [5] Proximal distilled evolutionary reinforcement learning. AAAI 2020.
        - [6] ERL-Re$^2$: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representation. ICLR 2023
        - [7] Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms. *IEEE Transactions on Evolutionary Computation (TEC)*
- Line 156 is problematic. Infinite horizon with a discount factor of 1 leads to numerical divergence
- Concerns on the experiments:
    - For performance comparison, important baseline methods that use evolutionary RL or population-based RL, such as ERL-Re2 [6], PDERL [5], EvoRainbow [4], FastTD3 [2], should be included as necessary baselines. The authors claim “EPO outperforms state-of-the-art baselines in sample efficiency, asymptotic performance, and scalability”. From my perspective, this is an overclaim with no convincing support.
    - The “ablation study” provided in Figure 4 is not an ablation. The ablation should be comparing EPO with its variants like EPO - off-policy training, EPO - mutation, EPO - crossover, etc.
    - The scaling provided in Figure 4 does not make sense to me, because more samples for used when increasing the number of agents or environments and the convergence performance of the curves is not presented. Thus, I cannot rule out the possibility that “32 agents” converges the same/learns better than “64 agents” when using the same number of interaction samples.
    - Moreover, why is “128 agents“ only shown in Figure 4(b) but missing in Figure 4(a)?
    - One very important point is about diversity. The authors mentioned multiple times that the recipe of EPO (or generally evolutionary RL) is the diversity provided by the population. However, I did not see any in-depth analysis on the differences or diversity among policies/genes/the trajectories generated by different policies. This is very important to the evaluation of the proposed method.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: - Could the authors provide clear definitions of their contributions over prior work and perform surgical ablations that toggle each core idea of their algorithm?
- Other questions are raised under Weaknesses.
soundness: 2
strengths: - The idea of maintaining a shared actor-critic with a latent "gene" per agent is interesting to reduce memory efficiency. The increased "coherence" between the agents would come at the cost of diversity; however, it's an interesting and meaningful design knob.
- The empirical results are on challenging high-dimensional tasks and show a strong task performance.
- The idea of combining population-based exploration with standard policy gradient is interesting and well-motivated by prior work.
- Scaling with a larger number of agents shows a clear trend.
summary: The paper proposes Evolutionary Policy Optimization (EPO): a shared actor–critic approach with agent diversity induced via a latent “gene” per agent and applying evolutionary operators (selection/crossover/mutation) in the latent space. A master policy is trained with a split-and-aggregate (SAPG) approach with importance-weighted off-policy correction that uses data from follower agents. Results on challenging continuous-control tasks show strong returns and good scaling versus PPO/SAC/PBT/CEM-RL.
weaknesses: - The paper's key contributions are unclear from the writing and experiments. The individual components like latent conditioning, shared weights, and evolutionary algorithms are known, so it's not clear where the novelty lies, and what isolated novel contributions drive gains. The paper does not explicitly mention the contributions, and there are no ablations of individual components of the supposed contributions.
- Section 4.2's off-policy loss needs an algorithmic or theoretical justification that it doesn't break any assumptions of the underlying algorithms and would preserve its convergence properties.
- It's great to see gains with parallelism and scaling; however, some experiments in a limited data regime would also clarify the limitations and capabilities of this method.
- The differential use of multi-step returns is also not justified empirically. It's tough to understand what the real reasons are that drive the gains.
- "EPO demonstrates a lower performance variance across different random seeds compared to baseline methods" - from the figure, this claim does not seem justified. Are there any quantitative results that the variance over seeds and hyperparameter variations is necessarily reduced with EPO? And what is the motivation for such variations to be reduced?

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: - I noticed that many baselines in the paper were run for 1 billion steps, for off-policy algorithms, how long did this actually take in terms of wall-clock time and compute resources?  
- Under fixed compute budget and fixed step count, please provide comparisons for returns with \(n \in \{1,3,5\}\) and \(\lambda \in \{0.8, 0.9, 0.95\}\); what are the differences in performance?
soundness: 2
strengths: - The writing is clear and the method is easy to follow.
- The approach retains performance gains when increasing population size and parallel environment count, aligning with the common practice of scaling RL by boosting throughput.
summary: The paper proposes Evolutionary Policy Optimization (EPO): it maintains a population of agents, each controlled by a latent embedding, while all agents share a common actor–critic network. A designated master agent aggregates experience from the entire population and performs off-policy updates; the remaining agents conduct on-policy PPO updates. Evolutionary operations (selection, crossover, mutation) are applied only to the latent embeddings, not to the network parameters. Experiments span manipulation, legged/humanoid locomotion, and classic control tasks, showing that EPO outperforms several baselines in sample efficiency, final performance, and scalability with parallel environments.
weaknesses: - The paper suffers from insufficient baseline coverage and ambiguous positioning: there exist many recent hybrid EA-RL[1-4] methods (e.g., PDERL[5], PGPS[7], CSPS[6], ERL-Re²[8], EvoRainbow[9]). Comparing to older methods such as PBT and CEM-RL (pre-2018) does not convincingly demonstrate effectiveness. The lack of comprehensive literature mapping undermines readers’ ability to assess the core contribution.

- Most of the design components (on-policy/off-policy mix, population+latent embeddings) have appeared in previous work (e.g., CSPS used on/off-policy combination). The authors should clearly articulate how their method differs from CSPS and other ERL methods, and highlight the novel aspects.

- The experiments lack comparison on established benchmarks with other advanced hybrid algorithms (e.g., MetaWorld, MuJoCo, DMC).

- Some fine-grained implementation details may have large effects on results (e.g., using 3-step TD returns while comparison methods may not). The authors should provide ablations to rule out that performance gains come from hyper-parameter tuning.

- The authors should explicitly distinguish between two types of scaling: (A) increasing parallel throughput (N ↑) and (B) increasing model parameters.

- The evolutionary trigger using range/median is heuristic; a sensitivity scan over γ and a comparison with fixed-interval trigger or trend-based trigge are needed, especially in sparse reward environments.

- Missing technical details: recommend consolidating list of elite count, crossover method, mutation noise, latent dimension & injection location (actor/critic symmetric or not), replay capacity/sampling strategy, IS clipping bounds, etc.; also provide scripts and random seeds.

---
[1]. Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms. IEEE TEVC 2024.

[2]. Reinforcement learning versus evolutionary computation: A survey on hybrid algorithms. Swarm Evol. Comput. 2019.

[3]. Evolutionary reinforcement learning: A survey.

[4]. A survey on evolutionary reinforcement learning algorithms. Neurocomputing, 2023.

[5]. Proximal distilled evolutionary reinforcement learning, AAAI 2020.

[6]. Cooperative heterogeneous deep reinforcement learning, NeurIPS 2020.

[7]. PGPS: Coupling policy gradient with population-based search.

[8]. ERL-Re^2: Efficient evolutionary reinforcement learning with shared state representation and individual policy representation. ICLR 2023.

[9]. EvoRainbow: Combining improvements in evolutionary reinforcement learning for policy search. ICML 2024.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: ### General questions
- What is the dimension of the latent embeddings (genes)? How sensitive is performance to this size?
- The paper notes prior work on latent-conditioned policies for diversity / skills (e.g., Ebert et al., 2021; Eysenbach et al., 2018). Could the authors clarify how EPO’s use of latent embeddings (“genes”) differs from these earlier approaches like DIAYN, which also condition policies on latent variables to induce behavioural diversity? Specifically, what new insight does your GA-based evolution add?

### Figure 3 and Table 1
In the experiments, the total number of environments is fixed at 24 576 and split evenly among agents (e.g., SAPG 6 → 4096 envs / agent, SAPG 64 and EPO 64 → 384 envs / agent). Could the authors clarify whether EPO also follows this same split? If so, how do they ensure that the observed performance gains stem from algorithmic scalability rather than simply from the diversity induced by the latent-conditioned population under reduced per-agent rollouts?

### $5.3
The total number of environments (24 576) is fixed while varying population size (K = 8–64), meaning only the split per agent changes. Since EPO claims to overcome the scalability cap when increasing total parallel environments, why is N held constant here? Doesn’t this setup test population diversity rather than true scalability with additional rollouts?

### $5.4
Since EPO’s main claim is improved scalability with increasing parallel environments, §5.4’s limited scaling experiment (up to 49k envs, no detailed comparison) feels insufficient. Could the authors expand the scaling analysis to better support this central claim?
soundness: 2
strengths: - Clear motivation and solid integration of known techniques (PPO + GA + SAPG).
- Strong empirical gains, especially in complex manipulation tasks.
- Elegant, memory-efficient design using shared weights and latent conditioning.
- Results demonstrate robustness across seeds and domains.
summary: The paper proposes Evolutionary Policy Optimization (EPO), a hybrid reinforcement learning method combining Genetic Algorithms (GAs) and on-policy gradients (PPO) via Split-and-Aggregate Policy Gradients (SAPG). Agents share a single actor–critic network but differ through latent embeddings (“genes”), evolved by GA to maintain diversity. A master policy aggregates all agents’ experience through importance-weighted updates.
Experiments on 8 tasks (manipulation, locomotion, control) show that EPO outperforms PPO, SAPG, PBT, and CEM-RL, and appears to scale better with parallel environments.
weaknesses: See the questions mainly but here is summary;
- Limited novelty: Most components are borrowed from prior work (CEM-RL, SAPG, DIAYN).
- Scaling evidence insufficient: §5.4’s scaling test (up to 49 k envs) lacks baseline comparisons and deeper analysis.
- Ablation mismatch: §5.3 fixes total environments, testing population diversity rather than true scalability.


Decision: Reject