AEAP: A Reinforcement Learning Actor Ensemble Algorithm with Adaptive Pruning
Authors: WEI ZHANG
Guni Sharon
Venue: Accepted by TMLR ()

Abstract: Actor ensemble reinforcement learning methods have shown promising performance on dense-reward continuous control tasks. However, they exhibit three primary limitations: (1) diversity collapse when using a shared replay buffer, often necessitating carefully tuned regularization terms; 
(2) computational overhead from maintaining multiple actors; and (3) analytically intractable policy gradients when using stochastic policies in ensembles, requiring approximations that may compromise performance. To address this third limitation, we restrict the ensemble to deterministic policies and propose Actor Ensemble with Adaptive Pruning (AEAP), a multi-actor deterministic policy gradient algorithm that tackles the remaining limitations through a two-stage approach. First, to alleviate diversity collapse, AEAP employs dual-randomized actor selection that decorrelates exploration and learning by randomly choosing different actors for both environment interaction and policy update. This approach also removes reliance on explicit regularization. Second, when convergence to homogeneous policies still occurs over time, computational efficiency is further achieved through adaptive dual-criterion pruning, which progressively removes underperforming or redundant actors based on critic-estimated value and action-space similarity. Although AEAP introduces four additional hyperparameters compared to TD3 (a baseline single-actor deterministic policy gradient algorithm), we provide two domain-agnostic parameter configurations that perform robustly across environments without requiring tuning. 
AEAP achieves superior or competitive asymptotic performance compared to baselines across six dense-reward MuJoCo tasks. On sparse-reward Fetch benchmarks, AEAP outperforms deterministic policy gradient methods but falls short of SAC (a baseline stochastic policy gradient algorithm) on one of three tasks. When compared to fixed-size multi-actor baselines, AEAP reduces wall-clock time without sacrificing performance, establishing it as an efficient and reliable actor ensemble variant.

Under review as submission to TMLR
AEAP: A Reinforcement Learning Actor Ensemble Algorithm with Adaptive Pruning
Anonymous authors Paper under double-blind review
Abstract
Actor ensemble reinforcement learning methods have shown promising performance on dense- reward continuous control tasks. However, they exhibit three primary limitations: (1) diversity collapse when using shared replay buffer, often necessitating carefully tuned regularization terms; (2) computational overhead from maintaining multiple actors; and (3) computationally inefficient policy gradients when using stochastic policies in ensembles due to high-variance estimates, requiring approximations that may compromise performance. To address this third limitation, we restrict the ensemble to deterministic policies and propose Actor Ensemble with Adaptive Pruning (AEAP), a multi-actor deterministic policy gradient algorithm that tackles the remaining limitations through a two-stage approach. First, to alleviate diversity collapse, AEAP employs ital-randomizzd actor selection that decorrelates exploration and learning by randomly choosing different actors for both environment interaction and policy update. This approach also removes reliance on explicit regularization. Second, when convergence to homogeneous policies still occurs over time, computational efficiency further achieved through adaptive dual-criterion pruning, which progressively removes underperforming or redundant actors based on critic-estimatin value and action-space similarity. Although AFAP introduces four additional hyperparameters compared to TD3 (a baseline single- actor deterministic policy gradient algorithm), provide two domain-agnostic parameter configurations that perform robustly across environments without requiring tuning. AEAP achieves superior or competitive asymptotic performance compared to baselines across six dense-reward MuJoCo tasks. On sparse-reward Fetch benchmarks, AEAP outperforms deterministic policy gradient methods but falls short of baseline stochastic policy gradient algorithms. When compared to fixed-size multi-actor baselines, AEAP reduces wall-clock time without sacrificing performance, establishing it efficient and reliable actor ensemble variant.
1 Introduction
Deep reinforcement learning (RL) has demonstrated significant potential across diverse continuous control domains, such as video games (Mnih et al., 2015; Silver et al., 2017), robotic manipulation (Clegg et al., 2018; Peng et al., 2018), and traffic optimization (Ault et al., 2020; Ault & Sharon, 2021). A large body of reinforeement learning algorithms (Fujimoto et al., 2018; Haarnoja et al., 2018) rely on a single agent to explore the environment via trial and error. However, single-actor exploration augmentation (Burda et al., 2018a; Ostrovski et al., 2017) often stalls narrow regions high-dimensional action spaces, loading poor sample efficiency or even suboptimal convergence (Yang et al., 2022).
Actor ensemble (i..,, multiple actors policies) methods have emerged as promising direction for enhancing exploration by maintaining diverse set of actors (Peng et al., 2020; Ren et al., 2021). Despite their empirical success, three critical limitations persist. First, shared replay bulfers tend to accelerate diversity collapse, where diversity refers to the entropy in action distributions across actors (Fujimoto et al., 2018). To alleviate this, explicit regularization strategics such as entropy bonuses or divergence penaltics can be employed but prove difficult tune (Sheikh et al., 2022; Masood & Doshi- Velez, 2019). Insufficient regularization yields homogeneous policies whereas excessive regularization destabilizes learning (Zahavy et al., 2023).

---

Under review as submission to TMLR
Second, computational overhead scales linearly with ensemble size (Chen et al., 2021). Third, when treating stochastic ensemble policies as components of a probabilistic mixture, computing policy gradients becomes computationally inefficient due to high-variance gradient estimates that grow with the number of mixture components, requiring surrogate approximations that can compromise performance (Ren et al., 2021).
Motivated by these challenges and by recent evidence highlighting the benefits of deterministic policy gradients for training Gaussian-mixture actors (Dey & Sharon, 2024), we propose Actor Ensemble with Adaptive Pruning (AEAP), multi-actor deterministic policy gradient algorithm.
State Exploration Rate
0.6
AEAP comprises two key components: (i) dual-randomized actor sclection where one random actor interacts with the environment while another random actor receives gradient updates; and (ii) a duul- stualpguning mechanism based on both performancee-stimated values and action-space distances.
0.3
Dual-an
DARD
TGS
0.1
0.2
0.3
0.4
0.5
Timesteps (1e6)
We demonstrate that dual-randomized actor selection maintains diversity by inducing high variance during exploration, without the need for explicit regularization. To validate this mechanism, we systematically compare four selection strategies in 1-D bandit setting. After establishing the benefits in this simplified setting, we extend our analysis to realistic high- dimensional action spaces, and measure diversity through two metrics: (i)
Figure 1: Dual-randomized ex- hibits similar new state explo- ration rates as existing regular- ized method on Walker2d.
Principal Component Analysis (PCA) (Jolliffe, 1986) decomposition of action outputs during training; and (ii) new state exploration rate using RSNorm (Lee et al., 2025) to normalize and discretize states (i.e., tracking newly encountered discretized states over time)- We compare against TD3 (Fujimoto et al., 2018) and recent deterministic ensemble method Gaussian Mixture Deterministic Policy Gradient (GAMID) (Dey & Sharon, 2024), which includes explicit diversity regularization. As illustrated in Figure 1, dual-randomized selection maintains diversity at similar levels to existing regularized methods. (Further details are provided in Section 4.2).
Two observations molivate our following dual-pruning strategy: (i) high variance in updates inevitably produces unstable actors, and persistent. low-performing ac- tors would impede learning convergence, necessitating performance-based pruning, and (ii) computational waste stems from homogeneous policies or actor dominance, requiring redundancy-based pruning. We demonstrate that this pruning mechanism readily combines with existing ensemble methods (Figure (a)) and independently improves performance.
Walker2d-45
Average Return (1)
Average Return
55.00-
AEAP (N-7)
SimBa
PMOE
5000
GAMID
AEAP (N-3)
Prunnng
4500
PMOE
SAC
GAMID
4000
GAMID+FFuning
0.5
1.0
1.5
2.0
4260
4600
5000
Time (hr)
Figure 2: (a) Performance improvement in existing ensemble meth- ods (GAMID/ (PM))) after integrating pruning on Walker2d. (b) AEAP achieves high computational efficiency without sacrificing performance on Walker2d.
To evaluate the performance of the com- bination, we provide comprchensive em- pirical evaluations using two domain-agnostic parameter configurations: a conservative pruning approach for stable training, and an aggressive pruning approach for efficient. training. Wall-clock time measurements confirm that AEAP achieves better computational efficiency than fixed-ensemble approaches without sac- rificing performance (Figure (b)).. On dense-reward MuJoCo (Todorov et al., 2012) tasks, both settings achieve competitive performance that matches or exceeds baseline algorithms. On sparse-reward Fetch tasks, both outperform deterministic policy gradient baselines but underperform stochastic policy gradient methods Soft Actor-Crice Critic (SAC) (Haarnoja et al., 2018) and SimBa (Lee et al., 2025), with aggressive pruning demonstrating superior performance relative conservative pruning. By integrating dual-randomized actor selection with adaptive dual-criterion pruning, our work addresses the tension between exploration benefits and computational efficiency, offering a practical, efficient, and reliable alternative to fixed-size actor ensembles.
2

---

Under review as submission to TMLR
2
Related Work
We review the most relevant actor ensemble methods hercin, and relegate detailed discussion of singlc-actor exploration and critic ensembles to Appendix A. Actor ensemble methods can enhance exploration by maintaining multiple policies, but face fundamental trade-offs between actor diversity and computational efficiency. We categorize these approaches by their coordination mechanisms.
Distributed RL. A natural extension to enhance exploration is employing multiple actors concurrently. Distributed RL frameworks maintain multiple homogencous actors generating parallel trajectories, with a central learner aggregating experiences for policy updates (Mnih et al., 2016; Espeholt al., 2018; Horgan et al., 2018). Kapturowski et al. (2019) augment this paradigm with recurrent networks and burn-in replay, while Espeholt et al. (2020) centralise inference on GPUs for higher throughput. However, these approaches demand substantial computational resources and introduce significant synchronization overhcad between learners and actors, they require access to an emulator, which is often impractical in real-world settings.
Shared Replay Buffer. A different approach maintains a shared replay buffer across actors, assuming all actors contribute throughout training. These methods diverge in how they maintain actor diversity. Value-based selection approaches guide actor specialization with heuristies. Previous work partitions the state space and train specialized actors on subtasks (Ghosh et al., 2018) and aggregates outputs from multiple policies to stabilize training (Barth-Maron et al., 2018; Chen & Peng, 2019; Januszewski et al., 2021; Li et al., 2023). Zhang et al. (2018b) combine actor ensembles with tree search for refined action selection. Yet these methods still suffer from diversity collapse as actors often converge when sharing experiences.
Explicit regularization methods add penalties (Peng al., 2020; Zahavy et al., 2023), pairwise KL divergence maximization (Sheikh et al., 2022), and mixture entropy optimization (Baram et al., 2021; Ren al., 2021; Peng et al., 2019). Gamid (Dey & Sharon, 2024) extend this line by applying deterministic policy gradients to Gaussian mixtures, achieving strong performance in continuous control tasks. However, these approaches remain sensitive to hyperparametter tuning and can still exhibit single-actor dominance despite explicit diversity mechanisms, Our work extends Gamid by introducing dual-randomized actor selection and adaptive pruning to climinate redundant actors while preserving exploration benefits.
terms to the objective. These include behavioral
Population-based based RL. A nutation-based approach generates new actors by mutating high-performing ones, inherently promoting diversity. This approach is exemplified by several works (Conti et al., 2018; Doncieux & Mouret, 2013; Lehman & Stanley, 2011), but requires careful tuning of evolutionary parameters and often suffers from sample inefficiency in continuous control tasks.
Pruning in RL. Works on pruning in RL primarily target network compression to enhance computational efficiency and improve generalization (Song et al., 2019; Zhang et al., 2018a). Studies demonstrate that magnitudo- based and structured pruning can significantly compress deep RL. networks while preserving or enhancing performance (Livne & Cohen, 2020; Graesser al., 2022; Obando-Ceron al., 2024; Park et al., 2024). Our method differs by disabling entire actor networks, addressing ensemble-level redundancy rather than within-network sparsity.
Remark 1. We clarify that target network approaches (van Hasselt et al., 2015; Fujimoto et al., 2018), while related, fundamentally differ from ceoomble-assed techniques. Target networks stabilize training by decoupling rapidly changing networks from their bootstrapped targets, thus making updates more predictable and reducing oscillations. In contrast, ensemble-based approaches explicitly promole diversity or exploration through maintaining multiple distinct policies or critics.
3 Preliminaries
Deep Reinforcement Learning. The goal of reinforcement learning is to find a policy that maximizes the expected cumulative discounted return over a long horizon for a given Markov Decision Process (MDP) defined by the tuple (S, A,P, R,y,, where S is the state space, A is the action space. The transition dynamics are captured by P: S A + ASS), where A(S) denotes the space of probability distributions over S. The reward function is given by S A IR, and 7 (0,1) is the discount factor (Sutton & Barto, 1998).
3

---

Under review as submission to TMLR
A policy can be either stochastic S + A(A); mapping states to distributions over actions, or de- terministic S - A, mapping states directly to actions. The action-value function under policy is defined as: (s( Ef ar) 6o = s, du = a where the expectation is over trajectories T generated by following policy я. The optimal action-value function, Q*(s, a) max, Q" a), satisfics the Bellman optimality equation (Bellman, 1957): (*( = R(s, a) + [max Q*('', ("''''')) a')]- In vast body of value-based deep reinforcement learning, the Q-function is often approximated using neural networks with parameters 0, denoted as Qo (Mnih et al., 2015). The network is trained by minimizing the temporal difference error over transitions r,,,,r,s) sampled from a replay buffer D (Sutton, 1988):
I(0)= = ((s....)))))) (r- + max Qur (s'ss'a') - Qo(s,a))² a))² where O' denotes target network parameters that are periodically updated to stabilize training.
Deterministic Policy Gradient. The Deterministic Policy Gradient (DPG) theorem (Silver et al., 2014) provides framework for optimizing deterministic policies Mo S + A. For parameterized deterministic policy Pa with parameters Ø, the objective is to maximize the expected return: J(Ø) = Ea [Ro]
Eump' (Qt(s, (s)))))))] where p is the state distribution under policy p. The deterministic policy gradient is given by: (o) = Eargge (o)""ss,))l==odos V [s)]
Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2019) implements DPG using deep neural networks for approximating both the policy Hs and critic Qo. The critic is trained by minimizing the temporal difference error: [(0)- F(a,))))") (v- Qo(s, a))²
y=r+~Q0" (8', (or((')) Twin Delayed Deep Deterministic Policy Gradient (TD3) (Fujimoto et al., 2018) addresses overestimation bias in DDPG through three modifications: (1) clipped Double Q-learning, using twin critics Qe, where the minimum is used for value estimation, (2) delayed policy updates where the actor is updated less frequently than critics (typically every d iterations), and (3) target policy smoothing, adds clipped noise to target actions to regularize the value function:
(s', (s') + +e), E~ e~ clip(W(0.)). y = - min Q (s', 112
0' - a0 + (1-a)0' o' - aá + - a)
where is the soft update rate. These modifications often lead to more stable learning and improved performance in continuous control tasks.
Gaussian Mixture Policies. While standard policy gradient methods optimize single Gaussian pol- icy =[ass)- = (()); E). Gaussian mixture models (GMMs) (Alspach & Sorenson, 1972) offer richer wy (s))))((; (a; Σ,) where ws((s) are state-dependent mixing weights satisfying Σ, wi(s) = 1. However, computing exact policy gradients for stochastic mixtures requires intractable marginalization over component assignments. Gamid (Dey & Sharon, 2024) circumvents this by treating each component mean μ4 as a deterministic policy, enabling direct application of deterministic policy gradients to GMMs.
representational capacity through multiple components: T(als) = E
4 Algorithm
In this section, we present Actor Ensemble with Adaptive Pruning (AEAP) and analyze its two key components: fual--raddmized actor selection and adaptive dual-criterion pruning.
4.1 AEAP
AEAP maintains dynamic ensemble of active actors that evolves throughout training, as shown in Algorithm 1. At each ileration, an actor is randomly selected from the active ensembles, perturbing its actions with the Gaussian noise for enviroument interaction, and collecting transitions for the shared replay buffer. For policy updato, a different actor randomly selected from the active ensembles and updated via deterministic policy gradients. Every iterations, the pruning mechanism evaluates all actors against our dual criteria, climinating those that are either underperforming (low Q-values) or redundant. (similar behaviors).
4

---

Under review as submission to TMLR
Algorithm 1 AEAP: Actor Ensemble with Adaptive Actor Pruning
AEAP Hyperparameters: (1) Initial number of actors N; (2) Distance pruning threshold Ç:
2:
(3) Critic pruning ratio threshold E; (4) Pruning frequency x
TD3 Hyperparameters: Exploration variance Σ; Target update rate t; Policy update frequency d
4: Initializations:
Replay buffer D; Actor networks {d:);=
Critic networks 01, O2 and targets 88, 022; Active actor set Sective + [I,...,NN
6: for = to T' do + TD3-style Training with Stochastic Actor Selection
Select actor ~ Uniform(Sactia Execute observe s', done, and store (s, r, 8', done) in D
and sample action + T.. (s) + N(0
8:
Sample mini-batch B from D
10:
Compute largets: j Iniform(Snctivv)) a' t Tay (clpp)))
1 K ++> min; 1.2 (o: (d',a')
11:
Update all critics: 0, f arg ming, [8] [Q( (s, y)²
12:
if % d= then
13:
Updale actor randomlyy: ΦR f- arg max [8
Eses Qo, (8, To, (8)) k Uniform(S
14:
Soft-update critic targets: 8 + =0, + (1 T) end if
15:
16:
if t % == 0 and |Sactivel > 1 then + AEAP Actor Pruning Mcchanism
17:
Compute pairwise distances: Di [B] EeEs || T (s) TT, (s) for all i # j
18:
Compute actor values: Qi = [8 E.e8 min Otarg TØ, (s)) for all E Sactive
19:
if ]z E Sactive such that Q₂ < & max; Qi then + Performance pruning Remove actor with the lowest Q-value
20:
21:
else if max Dij < Ç then-> Redundancy pruning Remove actor with lower Q-value from the closest pair
22:
23:
end if
24:
end if
25:
end for
4.2 Exploration Benefits of Mull-Ranommized Actor Selection
Empirical studies (Ren et al., 2021; Dey & Sharon, 2024) and theo-
80% Threshold
Thrashold
retical analyses (Calinon et al., 2012; Chan al., 2022) demonstrate that actor ensembles enhance exploration, particularly where the state-action space remains largely unexplored by single-actor meth- ods. Theoretical analyses examine exploration gains through entropy measures and probability ratios, highlighting that certain regions of the action space have substantially lower probabilities of being sampled by a single Gaussian actor than by an ensemble modeled as Gaussian mixture. We further substantiate these insights through intuitive visual analyses, centered on two primary observations:
TD3 (o=
(a= 0.5)
1.0
1.0
0.8
Expaired Verance
0.5-
0.5
Princball Component
Prinipall
D44
TD3-0-01 0.1
TC8: d- 0.5
Single-actor Exploration is Limited. Merely increasing the explo- ration variance (a) of a single actor, as commonly practiced in TD3, is insufficient to guarantee robust exploration, particularly in high- dimensional action spaces. We examine this exploration limitations of TD3 using the Humanoid environment from MuJoCo (Todorov et al., 2012), which features a challenging 17-dimensional action space. Figure 3 shows Principal Component Analysis (PCA) (Jolliffe, 1986) of actions generated by TD3 during the initial 50,000 training steps with varying exploration variance o E (0.1, 0.5} and the distribution heatmaps for the first two action dimensions. The explained variance ratios reveal that TD3 concentrates exploration predominantly along few dimensions. While raising a from 0.1 to 0.5 partially addresses this limitation, the concentration effect persists, presumably due to the action clipping that constrains noise augmented actions to valid bounds. Appendix B.2 provides additional
mm
04 Dim
Dm
Figure 3: (a) PCA of TD3 actions with varying exploration variance on Hu- manoid. (b) TD3 action distribution
heatmaps on Humanoid two di- mensions).
5

---

Under review as submission to TMLR
Ador Postions& Gradient Efects (Supe 10c0)
Ader Postions & Gradent Effects (Stegle ton0
Adtor Postions Goadiett Efrcce (Stepe 10320)
Action Visitation (3K Steps)
Dual Fandom
Randomie
164
227
500
684
801
189
106
Greedy Fondom-
214
313
992
708
381
181
Greedy+ All
178
157
191
462
770
102
215
151
Randkur-All
GuosppRannom
Croody-til
1.
Figure 4: (a) The background gray curve shows the bimodal reward landscape in 1-dimensional action space. Colored dots represent actor positions at different training steps, with arrows indicating gradient-directed movement (From -1 to 1). Methods with random updates (Dual-Random, Steedy+Ranndod update only one actor per step, with dual-randomized selection exhibiting multimodal behavior. (b) Cumulative action space visitation frequency over 3K steps.
results across multiple seeds, even with high exploration noise (o 1.0), TD3 exhibits clustering near the action space boundaries, limiting effective exploration in high-dimensional action spaces.
Remark 2. We provide additional similar analysis for the Hopper domain, a simpler environment with a 3-D action space, in Appendix B. These results indicate that increasing exploration noise in lower-dimensional settings can indeed broaden action coverage more effectively, highlighting the importance of dimensionality considerations when designing exploration strategies.
Dual-randomized Actor Selection Maintains Diversity. Inspired by dropout techniques in Hiraoka et al. (2022) that randomly deactivate Q-heads to prevent single critic dominance, we extend this randomization concept to actor selection: we randomly choose one actor interacts with the environment and another random one receives gradient. updates. We evaluate four configurations by crossing two design choices: (1) how actors are selected for environment interaction - either randomly through performance- prrformance-based greedy selection, and (2) which actors receive policy updates either single randomly selected actor all actors simultaneously. This yields four variants allowing us to isolate the impact of randomization at each stage, demonstrating that dual-randomized actor selection offers superior exploration advantages.
Using the continuous bandit problem from Huang et al. (2023), which features a deterministic re- ward function with two modes in a 1-D action space. We employ four actors with identical network architectures and orthogonal initializations. (Full experiment setting is reported in Appendix B.3). Figure (a tracks current actor positions and corrosponding gradient dynamics over lime, revealing that configurations where all actors receive updales strongly converge toward a single local optimum. Conversely, dual-randomized selection exhibits multi-modal behavior capable of overcoming local optima.
Dual-Ran sesm
GAMID 18%
We altribute this to two factors: (i) random actor selection promotes diverse environment interactions, particularly early in training when the critic remains immature, thereby providing richer for critic learning, similar to supervised learning where balanced samples accelerate learning; and (fi) updating single random actor introduces delayed, high-variance learning dynamics, giving actors that discover narrow high-reward regions a high probability of remaining there and avoiding premature convergence. As illustrated in Figure 4.(b), dual- randomized selection achieves the broadest. action space coverage among all strategics. Cumulative action selection probabilitics in Appendix B.3 confirm consistent patterns.
50 Percentage
100
State Exploration
0.9
0.6
0.3
Dual-tan
0.3 0.4
Timesteps (1e6)
Figure 5: (b) PCA decom- position on Humanoid. (b) New state exploration on Humanoid.
This capability can also extends to high-dimensionall action spaces. We demon- strate this using GAMID (Dey & Sharon, 2024), multi-actor algorithm with explicit divergence regularization. Figure 5.(a) compares PCA decomposition of TD3, rand-izndd selection, and GAMID on Humanoid, showing the first five principal components where dual-randomized exhibits the most uniform distribution across dimensions. Figure 1b(b) demonstrates that diversified actions increase new state discovery, measured using Running Statistics Normalization (RSNorm) from (Lee et al., 2025) to standardize and discretize observations for
6

---

Under review as submission to TMLR
Table 1:
Actor selection ratios and final mean performance(I) for Gamid under varying e values, Actor labels are assigned by selection frequency, with Actor 0 being the most frequently selected.
(a) WALKER2D-V5
(b) HUMANOID-V5
Actor 0
Actor 1
Actor 2
R
Actor 0
Actor 1
Actor 2
Actor
R
I/N
72.22%
15.82%
11.96%
4,213
64.18%
16.94%
11.33%
7.55%
4,938
1/(2N)
84.23%
7.15%
8.61%
4,337
78.26%
9.81%
7.42%
4.51%
5,016
(10N)
96.47%
2.23%
1.29%
4,581
93.84%
2.94%
1.96%
1.26%
5,219
No Pruning
Performance Pruning (+P)
Redundancy Pruning (+R)
Both Pruning [+PR]
Average Return (1)
Average Retum (1)
PMOE
GAMID
Stept' 10000
1.0
Dust-Random
Feedamdergyy Phuned
+P
+P
+R
+R
0.5
+PR
+PR
4200
4800
4200
4800
Figure 6: (a) Integration of pruming strategies leads to performance improvements for both GAMID and PMOE in Ant, with the highest gains achieved when both redundancy-based and performance- frreemance-based criteria are combined. (b) Comparison of cumulative action selection probabilities between randomizdd actor selection and its performanco-basoe pruning extension.
tracking exploration rates (details in Appendix B.4). Dual-randomized exhibits similar new state exploration rales as existing regularized method Humanoid.
However, this high variance has double-edged effect: while enabling escape from local optima, unstable gradients may trap some actors in suboptimal regions between peaks, as shown in Figure 4.(b). This high visilation frequency between peaks arises because immature critic functions could generale scatlered gradients, trapping actors in intermediate areas. We address this challenge in the following section.
4.3 Actor Pruning: From Exploration to Exploitation
We address two central questions regarding actor pruning strategy: (1) why pruning actors is preferable to retaining all actors throughout training, and (2) how actors should be pruned effectively.
Pruning is preferable. Two computational inefficiencies motivate pruning. First, as shown in Section 4.2, unstable gradients produce underperforming actors that impede learning convergence, Second, even with explicit regularization, training dynamics drift toward actor dominance where certain policies dominate sampling, wasting computation seldom-used actors.
We demonstrate the second issue using GAMID, which selects actors for environment interaction via an E-greedy rule. Table 1 reports the empirical selection ratios and final mean performance for each actor under three values in Walker2d-v5 and Humanoid-v5. Smaller & yields better returns but also steeper concentration of interaction counts on one actor. plausible explanation that, maintaining several actors benefits exploration in the early phase when stale action visitation is sparse. However, as exploration becomes less valuable relative to exploitation, higher enforces more random selections when the algorithm should exploit the superior actor, and thus lcads to worse performance. This observation confirms that the computational overhead of maintaining all actors becomes increasingly unjustified, as remaining actors contribute little beyond redundant. updates, motivating our adaptive pruning approach. Figure 6.(a) shows performance improvement in GAMID after integrating redundancy- based pruning.
Pruning Strategy. We introduce dual pruning based on performance- values and action-space distances to maintain exploration diversity while ensuring computational efficiency:
7

---

Under review as submission to TMLR
Performance-Basee Pruning: Actors whose mean Q-value falls below predefined ratio of the best performer are removed.
Kedundaucy-Based Pruning: actor with lower Q-value is removed.
When maximum pairwise action distance falls below threshold, the
We detail hyperparameter choices for pruning and their rationale. The performance threshold ç 0.85 eliminates actors performing below 85% of the best performer, value that empirically performs well across environments while avoiding overly aggressive pruning- A potential concern is that overestimation bias might incorrectly prune actors with better Q-value estimates. We address this in Appendix B.5 and find that several mitigating factors make this situation rare in practice. To ensure adequate training before pruning decisions, we use a pruning frequency of x = 10, 000 steps. The distance pruning threshold is defined as: { f= c. v (A) Omax, where A is the action space with maximum action magnitude Omax per dimonsion, and e scaling constant. This directly reflects the maximal possible action-space distance.
We identify two particularly effective configurations:
1. Conservative setting: (N = 3,c 1.0) Small initial ensemble coupled with a conservative threshold (C = 1.0- 100==1.--iiiimi (im((A) 0mmx)) typically retains one or two trained actors for stability.
2. Aggressive setting: (N = c
Large initial ensemble alongside a higher threshold (C = 1...v /dim amaxxx) rapidly pruning to single dominant policy after leveraging exploration capabilities. This is particularly beneficial in sparse-reward environments.
We demonstrate the effectiveness of performance- based pruning using the previous bandit example. Figure 6.(b) shows that starting from three actors, with performance ratio threshold of 0.85 and pruning frequency of 1, 000 steps. Actors are progressively removed when their average Q-value falls below 85% of the best performer. This adaptive mechanism successfully guides the ensemble to the global optimum by removing suboptimal actors while preserving the best.
Figure (.()) demonstrates that integrating either single or combined pruning mechanisms into GAMID and another ensemble method Probabilistic Mixture-of-Experls SAC (PMOE) et. al., 2021), 1, yields performance in the Ant environment, which has nigher-dimensional action space. Additional comparisons in Humanoid and HalfChectah are provided in Appendix B.6. Further ablation studies on AEAP, presented in Section 5.2, also analyze the two pruning strategies in detail and confirm that both contribute to performance improvements.
Remark 3. We briefly provide a hcuristic justification for our preference of the maximum-distance criterion over mean-distance criterion. Mean pairwise distances are known to concentrate in high- sigh-dimensional spaces, making them mumerically indistinct and insensitive to meaningful behavioral differences, especially cases involving small variance (Beyer et al., 1999). Reinforcement learning typically employs small variance in exploration, where subtle yet significant differences in actor behaviors are often undetectable by mean-based metrics. Consequently, adopt criterion based purely the geometric properties of the action space, independent of variance, to more reliably identify genuine behavioral rodundancy among actors.
5 Experiments
We aim to evaluate four aspects of AEAP: (1) performance competitiveness against established single-actor and multi-actor baselines across dense and sparse reward enviromments, (2) computational efficiency gains through adaptive pruning compared to fixed-size ensembles, (3) robustness of our proposed hyperparameter configurations across diverse continuous control tasks, and (4) effectiveness of individual and combined pruning strategies.
5.1 Baseline Comparisons and Computational Efficiency
Baselines. We evaluate AEAP against established baselines on contimuous control benchmarks from MuJoCo (Todorov et al., 2012) and Fetch (Plappert et al., 2018a) domains. Our comparisons include
8

---

Under review as submission to TMLR
Hopper-v5
Walker2d-v5
Ant-v5
HalfCheetthhv55
Average Retumm
3600
AEAP (N-3)
AEAP (N-7)
5500
AEAP (N=7)
SimBa
6000
AEAP (N-7)
SimBa
13000
AEAP (N-3)
SimBa
3300
TD3 SAC
GAMID
SimBa
4500
5000
AEAP (N-3) GAMID
SAC
(NAI/GAMID
PMOE
5000
AEAP (N=3)
TD3
GAMID
12000
11000
AEAP (N-7)
3000
PMOE
4000
TD3
1000
SAC
PMOE
10000
TD3
SAC
GAMID
PMOE
0.5
1.0
1.5
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
2.5
0.5
1.5
Humanoid-vš v5
FeichSlide-v4
FelchPickAndPlacee v4
5600
SimBa
1.0
SimBa
1.0-
Average Return
AEAP (N-7)
SAC
AEAP (N-3/7)
SAC
Sim8a
AEAP (N-7)
SAC AEAP (N-7)
SimBa
5200
AEAP (N=3)
TD3
GAMID
0.9
TD3
0.7-
TDS
AEAP (N
SAC
GAMID
0.7-
TD3
AEAP (N-3)
GAMID
PMOE
0.8
GAMID
0.5
0.5-
4800
1.0
1.5
2.0
2.5
0.7
0.5
1.0
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
Time (hr)
Time (hr)
Time (hr]
Time (hr)
Figure 7: Performance-effiiinny trade-off MuJoCo contimuous-control benchmarks (average episode return) and sparse-reward Fetch domain (success rate). While AEAP does not achieve the highest performance on every task, it consistently attains competitive performance across domains while requiring less wall-clock time than the recent thate-of-the-art SimBa and other fixed-size actor ensembles, making it an efficient alternative when resources or training time are constrained. Wall-clock measurements represent mean runtime over seven trials (1M steps each) on NVIDIA RTX 4090 GPU and Intel 19-13900K CPU.
single-actor methods TD3 (Fujimoto al., 2018) and SAC (Haarnoja et al., 2018), for multi-actor ensembles, use Probabilistic Mixture-of- Experts SAC (PMOE) (Ren et al., 2021) and Gaussian Mixture Deterministic Policy Gradient (GAMID-PG) (Dey & Sharon, 2024). For completeness, we also include the most recent state-of-the-art deep RL architecture, SimBa (Tee et al., 2025), and observe that we achieve comparable post-training performance on over half of the MuJoCo domains with lower running time.
Implementations. We adopt the aggressive setting mentioned in Section 4.1 by default, implementations for TD3 and SAC utilize Stable-baselines3 (Raffin et al., 2021), whereas PMOF, GAMID-PG and SimBa leverage author-provided code, adhering strictly to recommended hyperparameters from original literature (detailed in Appendix C). Consistent with prior work (Ibarz et al., 2021), methods were augmented with Hindsight Experience Replay (HER) (Andrychowicz et al., 2018) for sparse reward tasks within the Fetch domains. All experiments are conducted over random trials.
Performance. The post-training performance are summarized in Table 2, with corresponding learning curves illustrated in Appendix B.7. Figure 7 presents the results, where the x-axis represents computation time and the y-axis denotes performance. Points in the upper-left corner indicate higher performance and compute efficiency. On dense-reward Mujoco tasks, AEAP consistently attains competitive performance across domains while requiring less wall-clock time than recent. state-of-the-art methods such as SimBa and other fixed-size actor ensembles or simpler baselines like SAC, even though it does not achieve the highest performance on every individual task. When resources or training time are constrained, AEAP offers a compelling performance--ffiiinnng trade-off for practitioners prioritizing faster experimentation or deployment. cycles. On sparso-reward Fetch tasks, AEAP substantially outperforms other deterministic methods while approaching the performance of SAC and SAC-based SimBa. The underperformance on FetchPickAndPlaa likely stems from its multi-stage exploration challenge: SAC with entropy regularization keeps sampling diverse actions long enough to discover that sequence, whereas the deterministic actors in AEAP narrow their exploration once pruning begins, reducing the chance of stumbling onto the critical manipulation pattern.
Efficiency. Figure 7 demonstrates AEAP's efficiency gains through adaptive pruning. Despite starting with actors, AEAP achieves substantially shorter runtime compared to fixed-size actor ensembles such as Gamid with competitive performance. This demonstrates the effectiveness of AEAP as a computationally efficient. and reliable alternative to conventional multi-actor reinforcement learning algorithms.
9

---

Under review as submission to TMLR
Table 2: Mean performance and the 1-standard deviation on continuous control benchmarks. The best and second-best performing RL algorithms have been highlighted.
TD3
SAC
GAMID
PMOE
SimBa
AEAP(N=3) AEAP(N-7)
MuJoCo (v5)
HallCheetah
10,097
10,493
10,285
253
10,156 4 312
12,732
13,110
11,121
1: 429
Hopper
3,499 +: 225
3,487
= 11 3,367
+ 196
3,124
= 189
3,486
:: 370
3,587
+: 365
3,649
= 358
Walker2d
3,998
4,514
4,987
4,378
254
5,261
5,057
:: 275
5,209
Ant
4,630
= 475
4,098
+ 213
4,436
+ 292
4,089
= 387
5,288
= 195
5,476
+: 205
5,632
199
Humanoid
5,004
129
5,389
5,254
167
4,923
298
5,583
328
5,375
+ 211
5,429
125
Swimmer
243 128
348
265 = 117
314
+ 87
349 = 24
347 +
351
Fetch (v4)
Push
0.9 + 0.35
0.99 0.1
0.8 0.4
0.99 = 0.1
0.99 = 0.11
0.99 + 0.1
Slide
0.62 0.15
0.78
0.34
0.66 0.19
0.91 0.24
0.74
= 0.26
0.85
0.25
PickAndPlace
0.65 +: 0.18
0.99 0.1
0.81 0.27
0.99 +: 0.05
0.77 + 0.13
0.78 4: 0.21
Robustness. We conduct additional experiments on Gym-v4 (Appendix B.7) to further validate the robustness of our default hyperparameter choices. These additional experiments confirm the reliability and effectiveness of our proposed configurations for AEAP under various experimental conditions.
5.2 Ablation Studies on Main Hyperparameters
Figure presents ablation results the sensitivity of initial actor count N {3,5, 7) and the distance pruming threshold coefficient E {0.5, 1.0, 1.5} using the Walker2d and Humanoid representative intermediate and high dimensional environments. The first row shows training curves and the second row tracks how the number of actors evolves throughout training. For direct comparison with single-actor methods, we include a black line indicating the mean performance of TD3. Several key observations are revealed:
Small 0.5: Minimal pruning occurs (first column in 8a and 8b), with actor counts remaining near initial values throughout training and training curves intertwine without clear differentiation. This matches our expectations: without explicit regularization, randomly selecting among multiple actors resembles running multiple independent TD3 instances. While actors show no performance stratification, initial learning accelerates compared to standard TD3 before event converging to baseline performance, consistent with our exploration analysis in Section 4.2.
Moderate 1.0: Actor counts reduce to approximately half the initial ensemble size (second column in 8a and 8b), with clear performance stratification emerging. Smaller ensembles (N = 3 N 5) tend to yield superior performance than the larger one (N 7) because the dual-randomized update scheme gives each actor fewer gradient steps in larger ensembles, slightly hampering individual learning. Nevertheless, even with reduced update frequency, pruned ensembles outperform both unpruned ensembles and single-actor baselines, demonstrating the value of selective retention over fixed-size approaches.
Large 1.5: Aggressive pruming rapidly converges to 1 1 actors after initial exploration (third column in 8a and 8b). Larger ensembles (e.g., N = or N 7) achieve superior final performance by fully exploiting multi-actor exploration before consolidation, while smaller ensembles N = suffer from insufficient early diversity due to the premature aggressive pruning.
Pruming dynamics: Larger ensembles consistently prune faster across all values. Despite orthogonal initialization, more actors naturally lead to greater action-space overlap, triggering distance-basea pruming more frequently.
It important to note that the heightened sensitivity observed in Figure results from deliberately testing boundary conditions with exaggerated pruning thresholds rather than reflecting realistic deployment scenarios.
10

---

Under review as submission to TMLR
[-0.5 (mm]44 daao
<=1.0: dIMIAT dros
Z- (dmm)))) Aeos
6- 0.5.5. dimL.) Roae
Cm3 3.0 (imLA)) Maa C-
BImLAT -e
Avnragg Reward
Avcr Roward
2000
3000
3020
3000
3000
Stops/1e0)
Stsps/1e8)
Stops) ted)
Steperfe6i
Stopsite0)
(a) Walker2d-v5
(b) Humanoid-v5
Figure 8: Ablation studies on main hyperparameters. Top rows show training performance (average reward) and bottom rows track ensemble size evolution (average number actors) across differrnt initial actor counts and distance thresholds Ç. With small ñ, minimal pruning occurs and performance remains close to TD3 baseline. Moderate enables effective pruning with clear performance improvements, while aggressive Ç rapidly reduces ensemble size, with larger initial ensembles achieving superior final performance.
web Parformance Pruning
Wel Distance Pruningg
All
AEAP
TD3
Hopper-v5
Walker2d-v5
Humanold-v5
6000
6000
Average Number of Actores
Average Reward
3000
3000
3000
0.0
0.5
1.0
0.5 (1e6)
1.0
0.0
0.5
1.0
0.0
0.5
0.0
1.0
0.5
1.0
Steps (1e6)
(1e6)
Stepe
Steps (1e6)
Steps (1e6)
Figure 9: Ablation studies on pruming mechanisms, showing that combined pruning achieves the highest performance.
We provide hyperparameter selection guidelines below that are designed be domain-agnostic. Similar trends are observed across additional domains, as detailed in Appendix B.8.
Hyperparameter selection guidelines. Hyperparameter selection can be distilled into two scenarios: (1) adopt the conservative setting when sustained behavioural diversity is valuable, as in non-stationary, multi-objective, or risk-sensitive domains where multiple distinct policies hedge against, regime shifts; and (2) choose the aggressive setting when the aim is to produce one high-performin policy quickly, as benchmark control suites or fixed industrial tasks, leveraging broad carly exploration followed by rapid, computo-cfficiett convergence.
5.3 Ablation Studies on Pruning Mechanisms
We ablate the two pruning mechanisms independently using the aggressive configuration (N 7, ( 1.5 (im((A) (max) across three environments of increasing action dimensionality: Hopper (3-D), Walker2d (6-D), Humanoid (17-D). Figure 9 reveals that combined pruning achieves the highest performance and most efficient actor reduction, validating that both mechanisms are essential and complementary. Distancc-based pruning triggered more frequently than performance-batt based pruning across all environments, indicating that action-space similarity more common than performance differences during training.
11

---

Under review as submission to TMLR
6 Discussion and Limitations
This paper presents Actor Ensemble with Adaptive Pruning (AEAP); novel approach that addresses the fundamental tension between exploration diversity and computational efficiency in ensemble-based reinforcement learning. Our key contributions include: (1) demonstrating that dual-randomized actor selection naturally maintains behavioral diversity without explicit regularization terms, (2) developing an adaptive dual-criterion pruning mechanism that evaluates actors based on both critic-estimated performance and action-space similarity, enabling the ensemble harness early diversity for exploration while progressively eliminating redundant or underperforming actors for computational efficiency, and (3) showing that AEAP achieves lower computational overhead compared to fixed-size ensembles like Gamid (Dey & Sharon, 2024) and PMOE (Ren et al., 2021) while maintaining superior performance, particularly in sparse-reward environments where exploration is crucial. By automatically adjusting ensemble size based on the learning phase, maintaining diversity during exploration and consolidating during exploitation, AEAP provides a practical framework for deploying ensemble methods without the computational burden of maintaining redundant actors throughout training.
Although our experiments illustrate the flexibility of this approach, several limitations warrant further investigation. First, AFAP introduces four additional hyperparametens compared to TD3. Two of these, performance threshold = 0.85 and pruning frequency = 10,000, perform robustly across all tested domains without requiring per-task tuning. However, Section 5.2 reveals that the scaling factor and initial ensemble size N exhibit. sensitivity under extr settings. Though our ablation studies identify two robust configurations, conservative pruning (N=3, c=.00 and aggressive pruning (N.7, that transfer reliably across all nine domains, we acknowledge that optimal hyperparameter selection may benefit from domain-specific adaptation in certain scenarios.
Second, formal convergence guarantees on how pruning affects convergence rates and sample complexity remain an open problem. Gal & Ghahramani (2016) prove dropout approximates variational inference, and Osband et al. (2016) demonstrate bootstrapped ensembles implement Thompson sampling by maintaining diverse posterior samples over value functions. Our random actor selection similarly prevents deterministic co-adaptation, preserving epistemic uncertainty throughout training. Additionally, delayed updates from randomly selecting actors for backpropagation align with asynchronous optimization theory, where Zinkevich et al. (2010) show parallel actors with gradient delays achieve linear speedup, and Osband et al. (2018) prove asynchronous SGD retains convergence when delays remain bounded. Each actor in AEAP creates inherent delays that facilitate escape from sharp local minima, observed in our bandit experiments (Figure 4). However, existing analyses assume tabular settings or convex objectives, whereas AEAP operates in high- dimensional continuous control with non-convex policy networks and adaptive ensemble sizes. Extending formal guarantees to this setting remains a challenging problem.
We believe this work nevertheless opens many exciting directions for future research. First, theoretical analysis of convergence guarantees under adaptive ensemble sizes would strengthen the foundation of this approach and guide principled hyperparameter selection. Second, investigating adaptive pruning criteria that incorporate uncertainty estimates model-based predictions could further improve efficiency. We hope this work inspires further exploration of adaptive ensemble methods that balance exploration, exploitation, and computational efficiency in deep reinforcement learning.
References
D. Alspach and H. Sorenson. Nonlinear bayesian estimation using gaussian sum approximations. IFFF Transactions on Automatic Control, 17(4):439-448, 1972. doi: 10.100/TACC.992.11000:
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew,
Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay, 2018. URL https: www.aii.wgg.ass////ooa 114999955....
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqu: Variance reduction and stabilization for deep reinforcement learning, 2017. URL https://arxi:: org/abs//611.....
httpp::/aaxiv.rrr/a////111...1.1 019299.
12

---

Under review as submission to TMLR
James Ault and Guni Sharon. Reinforcement learning benchmarks for traffic signal control. In Thirty-fifth Conference on Newral Information Processing Systems Dalasels and Benchmarks Track (Round 1); 2021. URL nttps://openrevier httppp//pppaareeeeeeeee/r//trtts mtr.://ooorrrmiiiitcgchoooor..
James Ault, Josiah P. Hanna, and Guni Sharon. Learning an interpretable traffic signal control policy, 2020.
URL attps://arxae htttps//arraiv..o//bb////// wwtpw///1///......////// 11023.
Nir Baram, Guy Tennenholtz, and Shie Mannor. Maximum entropy reinforcement learning with mixture policies, 2021. URL https://arxi:: wttp:iiiw..ggab////.....33 abs//10//22......//// 0ttp//2213.11166.... 10176.
Gabriel Barth-Maron, Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients, 2018. URL attps://arxiv hbgg/bbs//e htte//bb/1800....0 1800.086111
Richard Bellman. Dynamic Programming. Dover Publications, 1957. ISBN 9780486428093.
Kevin S. Beyer, Jonathan Goldstein, Raghu Ramakrishnan, and Uri Shaft. When is "nearest neighbor" meaningful? In C. Beeri and P. Bruneman (eds.), Proceeding of the 7th International Conference Database Theory, volume 1540 of Lecture Notes in Computer Science, pp. 217-235. Springer, 1999. URL http://wwrr springerlingg wtteerliin.coo////// ink aap71:=044p44cmmee@@2...
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity-driven learning, 2018a. URL https: https:s/aaxxvv.orggg. wwt.:////////.....o.//// 04355.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation, 2018b. URL https://arxix: org/ tabbs/8100 12894.
Sylvain Calinon, Affan Pervez, and Darwin G. Caldwell. Multi-optima exploration with adaptive gaussian mixture model. In 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL), pp. 1-6, 2012. doi: 10.1109/Devv LD000//DDDL.m222264000000000
Alan Chan, Hugo Silva, Sungsu Lim, Tadashi Kozuno, A. Rupam Mahmood, and Martha White. Greedification operators for policy optimization: Investigating forward and reverse kl divergences, 2022. URL https: ///axxii.rrg/aa////22/ 08285..
Gang Chen and Yiming Peng. Off-policy actor-critic in an ensemble: Achieving maximum general entropy and effective environment exploration in deep reinforcement learning, 2019. URL https://arxiv.org org/
abbs/190205553 05551.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-lcarning: Learning fast. without a model, 2021. URL https://aaaat https:////avxxxooa/a/aae httg:/abs/2110.05990 http:://aarrrr.oo//////01..111
Alexander Clegg, Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. Learning to dress: synthesizing human dressing motion via deep reinforcement learning. ACM Trans. Graph., 37(6), December 2018. ISSN 0730-0301. doi: 10.1145/3272127.227750 38.50448. URL nttps://doi.or: or./oo..//..//.................. 11..114/33272127 .2755448
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Improving exploration evolution strategies for deep reinforcement learning via a population of novelty-seeking agents, 2018. URL https://arxiv.org http://aawiv...r//////1//.111 httpg//b///11..eeee// 06560.
Sheelabhadra Dey and Guni Sharon. Comparing deterministic and soft policy gradients for optimizing gaussian mixture actors. Transuctions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openrevveer net/
httpe///ppprrrrrreeeeeetttt///// met//Eoummiddq$pppu000
Sheelabhadra Dey, James Ault, and Guni Sharon. Continual optimistic initialization for value-based reinforcement learning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, AAMAS '24, pp. 453 583-462, Richland, SC, 2024. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9798400704864.
13

---

Under review as submission to TMLR
Stephane Doncieux and Jean-Baptiste Mouret. Behavioral diversity with multiple behavioral distances. In 2013 TEEE Congress on Evolutionary Compulation, pp. 1427-1434, 2013. doi: 00010///CCC.2113655777777
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Tain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1407-1416. PMLR, 10-15 Jul 2018. URL https: attps://proot http////coceeecnss///ee/e// mlr. w.a...rressv///// espeholt18a. html.
Lasse Espcholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, and Marcin Michalski. Scod rl: Scalable and efficient deep-rl with accelerated central inference, 2020. URL https://arxiv: http:///aaawww..gab/////eee wtt./bs//1///////o// 06591.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration, 2019. URL
abs/aabss...////////////////aaee 10295.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods, 2018. URL https://arrae http::////rdv..o//////////00/ argg/bb//800..oo 110//1102.000 09477.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning, 2016. URL
www.o//ggggg.///////// 02142.
Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divide-and-conquer reinforcement learning, 2018. URL https://arxiv, wtt/abs/171... 09874.
Laura Graesser, Utku Evei, Erich Elsen, and Pablo Samuel Castro. The state of sparse training in deep reinforcement learning, 2022. URL https://arxiv, httpa://arravv.rr///aa//2000 arg/abs//20...o... 10369.
Tuomas Haarnoja, Haoran Tang, Picter Abbecl, and Sergey Levine. Reinforcement learning with deep energy-based policies, 2017. URL httpp:://aaxxi.orr/bbbb... 155.0/00011700..16.5..
Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018. URL https://arxiv. wwwpiiivvorr/bbba wrw/ 01290.
Takuya Hiraoka, Takahisa Imagawa, Taisci Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka. Dropout q-functions for doubly efficient reinforcement learning, 2022. URL
02034.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Mattco Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay, 2018. URL https://arxiv.
Zhiao Huang, Litian Liang, Zhan Ling, Xuanlin Li, Chuang Gan, and Hao Su. Reparameterized policy learning for multimodal trajectory optimization, 2023. URL httpp////arri..o//aa///////
Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine. How train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research, 40(4 5):698 721, January 2021. ISSN 1741-3176. doi: 10.1177/007783:19990975558 URL 10tg/11.... 1177 11770/78117777778889998888886
http:// httt://dxxdae
Piotr Januszewski, Mateusz Olko, Michal Królikowski, Jakub Šwiętkowski, Marcin Andrychowicz, Lukasz Kuciński, and Piotr Miloś. Contimuous control with ensemble deep deterministic policy gradients, 2021.
URL https://arxt wwwpriivorggab////e otg//bbs/111..5855
I.T. Jolliffe. Principal Component Analysis. Springer Verlag, 1986.
Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, and Remi Munos. Recurrent experience replay in distributed reinforcement learning. In International Conference on Leorning Representations, 2019. URL. http::///pppreeeeeeeeee/rr/rrrre lett/forumm.ddrr11ff/...
14

---

Under review as submission to TMLR
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learning: Controlling the estimation bias of q-learning. In International Conference on Learning Represenlations, 2020. URL
https://openrevier, wtwpeeiiiwweeeeoonnn/ee. ntt:forunt itt
Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R. Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning, 2025. URL httpp://aaxxvv.ooo http//bb/////////////// 09754.
Kimin Loe, Michael Laskin, Aravind Srinivas, and Picter Abbeel. Sunrisc: A simple unified framework for ensemble learning in deep reinforcement learning, 2021. URL https://arxiv. httpaaaaiivrrioaaaaee/ee//aa/eee
.04988.
Joel Lehman and Kenneth O. Stanley. Abandoning objectives: Evolution through the search for novelly alone. Evolutionary Computation, (0(2):189-22) 2011. doi: 10.1162/EVCO 00025. __0055...
Lin Li, Yuze Li, Wei Wei, Yujia Zhang. and Jiyc Liang. Multi-actor mechanism for actor-critic reinforcement learning. Information Sciences, 647:119494, 2023. ISSN 0020-0255. doi: https://dbi.orr//00.0000 www.://00............o/o.. 119494. URL hwtp://wwww.cneeneeeeeret... http:///ccmeeatttccaaa//////////
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wienstra. Continuous control with deep reinforcement learning, 2019. URL https:
Dor Livne and Kobi Cohen. Pops: Policy pruning and shrinking for deep reinforcement learning. IEFE Journal of Selected Topics in Signal Processing, 14(4):789 801, May 2020. ISSN 1941-0484. doi: 10.1109/ 2967566. URL http://dxs doi. htgg/10.109////// 2020. 2967566.
Muhammad A. Masood and Finale Velei-Velez. Diversity-inducing policy gradient: Using maximum mean discrepancy to find a set of diverse policies, 2019. URL https: attps://arxiv. www.aawwww..w..aa.a.....e.oooeee http//bb/190..0000
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540))5529 533, February 2015. ISSN 00280836. URL http: // /x. doi oogio.rgg/.... (008//atuue14200
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukeuoglu. Asynchronous methods for deep reinforcement learning, 2016. URL https://arxiv. http://axivvooor///a// org/abs/ htrg/abs/16022 hetp///1...o00 1678..1783. 01783.
Johan Obando-Ceron, Aaron Courville, and Pablo Samuel Castro. In value-based deep reinforcement learning, a pruned network is a good network, 2024. URL https://arxiv org/ abs//42.. 12417
lan Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn, 2016. URL https://arxiv wwwpa//..rrgaaa/////ee/ htt.b////6.......... 04621.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning, 2018. URL https:
htse//bb//1806/ 03335.
Georg Ostrovski, Mare G. Bellemare, Aaron van den Oord, and Remi Munos. Count-basod exploration with neural density models, 2017. URL https: /tttp:://rxxv.... http://awwwwx.x./aaa/// org////////3/.... 011111011..10.....111
Seongmin Park, Hyungmin Kim, Hyunhak Kim, and Jungwook Choi. Pruning with scaled policy constraints for light reinforcement learning. IEEE Access, 1-33055-360663 2024. doi: 10.1109/ACCESS5.20 2024. 3367002.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. -uriosity-driven exploration by self-supervised prediction, 2017. URL http///iv.orggbb///////aaaa
15

---

Under review as submission to TMLR
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimie: example-guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics, 37::))1-14, July 2018. ISSN 1557-7368. doi: 00.1145/3197517.22013111 URL http:///tt..oi..o/11.0..... 0tte///1.11///1977550 3201311.
Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. Mcp: Learning composable hierarchical control with multiplicative compositional policies, 2019. URL 09808.
Zhenghao Peng, Hao Sun, and Bolei Zhou. Non-local policy optimization via diversity-regularizdd collaborative exploration, 2020. URL httpp////aaaii..///bb/////////// 07770
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schncider, Josh Tobin, Macick Chociej, Peter Welinder, Vikash Kumar, and Wojeiech Zaremba. Multi-goal reinforcement learning: Challenging robolics environmeuts and request for research. CoRR, abs/1802.094644 2018a. URL http://arxiv wwtp://rr//ab//.....////
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Picter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration, 2018b. URL
https://arxiv: wwt./o/////////////////ee//ee//t aa//aa/s//.....//// 01905.
Antonin Rafiin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-basclnee3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1 8, 2021. URL http: httt:://jmrr orgoooo///oo pppppppaa2//p2///
html.
Jie Ren, Yewen Li, Zihan Ding, Wei Pan, and Hao Dong. Probabilistic mixt ofe-oo-eexsssss for efficient deep reinforcement learning, 2021. URL
Hassam Sheikh, Mariano Phiclipp, and Ladislau Boloni. Maximizing ensemble diversity in deep reinforcement learning. In International Conference on Learning Represenlations, 2022. URL forum?id=hjo -xppDp2..
net/
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic Policy Gradient Algorithms. In JCML, Beijing, China, June 2014. URL https://inria, hal. htt.iiiaaaaa..cnneeeee....a science/ ha1-00938992.
David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm, 2017. URL http:://axvvvvvvo..eee 1111b/11111....
Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning, 2019. URL https://arxiv..rg//
02975.
Bradly C. Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models, 2015. URL
00814.
Richard S. Sutton. Learning to predict by the methods of temporal differences. Mach. Learn., 3(1):9 3(1):9444, August 1988. ISSN 0885-6125. doi: 10.1023/A:102223333317999 URL https://doi.. org/10. arg///.1023//A. 1023/A: 1022633531479.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, Cambridge, MA, 1998.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Exploration: study of count-based exploration for deep reinforcement learning, 2017. URI. httpp: wttp:/iwwwrw//////./.o// 1.7117.
16

---

Under review as submission to TMLR
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 (EEE/RS) International Conference on Intelligent Robols and Systems, pp- 5026-5033, 2012. doi:
URL SA..00//RROS.222.666606.0
Hado Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning, 2015.
06461.
Zhengyu Yang, Kan Ren, Xufang Luo, Minghuan Liu, Weiqing Liu, Jiang Bian, Weinan Zhang, and Dongsheng Li. Towards applicable reinforcement learning: Improving the generalization and sample efficiency with policy ensemble, 2022. URL http:////aaiiv.orrr///////////// 092884
Tom Zahavy, Yannick Schroecker, Feryal Behbahani, Kate Baunli, Sebastian Flennerhag, Shaobo Hou, and Satinder Singh. Discovering policies with domino: Diversity optimization maintaining near optimality, 2023. URL https://arxiv.org///
13521.
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning, 2018a. URL https://arxiv.oot wwt..r/////...00000000 06893.
Shangtong Zhang, Hao Chen, and Hengshuai Yao. Ace: An actor ensemble algorithm for continuous control with tree search, 2018b. URL http:///aaaxivv..aa//////aee////
Martin Zinkevich, Markus Weimer, Lihong Li, and Alex Smola. Parallelized stochastic gradient descent. J. Lafferty, C. Williams, J. Shawe-Taylon R. Zemel, and A. Culotta (eds.). Advances in Neurol Information Processing Systems, volume 23. Curran Associates, Inc., 2010. URL httpp::/procceedeeee wtw.eeedinnnnnuuiis...... cc/ paper_f
pdf.
17

---

Under review as submission to TMLR
A Additional Related Work
Exploration in Single-Agent RL. Exploration allows agents to visit under-cxplored regions of the enviromment by deviating from optimal policies. Many approaches induce exploration by introducing stochasticity, either by adding noise directly to the output actions (Fujimoto et al., 2018; Mnih et al., 2015; Lillicrap et al., 2019) or to the policy parameters (Plappert et al., 2018b; Fortunato et al., 2019). Other strategies explicitly optimize for exploration, such as maximizing action entropy (Haarnoja et al., 2017: 2018) maintaining visitation maps that molivate visits infrequently encountered states (Tang et al., 2017;
Ostrovski ct al., 2017). Intrinsic rewards (Burda. et al., 2018b; Stadie et al., 2015; Dey et al., 2024) and curiosity-driven objectives (Burda al., 2018a; Pathak et al., 2017) have also been utilized. However, these methods fundamentally rely on modifying or augmenting the agent's current policy, thus inherently limiting exploration scope based on previously learned behaviors.
Critic Ensembles RL. Another prominent line of research maintains multiple critic networks (value functions) to guide exploration more effectively. Osband et al. (2016) trains multiple Q-heads approximate Thompson sampling and Anschel et al. (2017) reduces variance by averaging historical predictions. Osband et al. (2018) injects fixed bias to preserve long-tern uncertainty, Lan et al. (2020) mitigates overestimation via conservative ensemble targets. Lee et al. (2021) adds uncertainty bonus to the ensemble for improved exploration. Chen et al. (2021) extends these ideas to continuous control by updating a ten-critic ensemble with random subsampling, achicving state-of-tho-art sample efficiency.
B Additional Experimental Results
This section provides comprehensive experimental results to support the findings presented in the main lext.
B.1 Exploration Effectiveness in Low-Dimensional Environments for TD3
Humanoid-v5 environment (17-D action space)
(b) Hopper-vō environment: (3-D action space)
Figure 10: Action distribution heatmaps showing coverage across the first two action dimensions for TD3 with varying exploration noise levels.
Figure 10.(b) presents action distribution heatmaps for TD3 in the Hopper environment. Unlike the high- dimensional Humanoid environment, increasing exploration noise in Hopper demonstrably improves action
18

---

Under review as submission to TMLR
0-10
90% Thandook
90% Traakee
Egrihed Valarre R,30
04-
06
AEAP 0.12
AEAP
AEAP 1.01
Expariee Vironco Rato
Prenqui Componett
(a) Seed: 3847291
TD3 0.13
TD3 (a-m0 0.5)
TDS 1.0)
Enrendd Naturce Su20
98
04-
02-
15
AEAP (o 0.11
AEAP (0:11 0.5)
MEAP = 1.00
Expaned Wirten RMo
5 10
15
6
10
Precqui Componet
Percuul Component
Precpel Component
(b) Seed: 6152038
Figure 11: PCA analysis in Humanoid-v5 environment across different random seeds.
coverage actoss the visualized dimensions. This observation corroborates the dimenssinnllis-deeeeddet nature of exploration offectiveness in single-actor methods.
B.2 Exploration Effectiveness in High- High-Dimensional Environments for TD3
We provide additional empirical evidence supporting our claim from Section 4.2 that incrcasing exploration variance in single-actor methods is insufficient for robust exploration in high-dimensional action spaces.
Figures 11 presents PCA projections of TD3's 17-dimensional action outputs in the Humanoid environment across another two random sceds, with exploration noise E (0.1, 1.0). These results demonstrate the consistency of TD3's limited exploration patterns across different initializations, complementing the analysis presented in Figure 3 of the main text.
19

---

Under review as submission to TMLR
B.3 1-D Continuous Bandit Experimental Setup
The environment features a 1-dimensional action space a E [-1,1] 1] with deterministic bimodal reward function. The reward landscape contains a narrow global optimum at = -0.7 (reward = 1.0) and a wide local optimum at a = 0.3 (reward = 0.5), as illustrated in Figure 12. This design creates an cuploration-exploitataae tradc-off where the global peak is challenging to discover due to its narrow width, while the local peak is easily found but yields suboptimal rewards.
In the all-actor update configuration, all actors in the ensemble receive gradient updates simultaneously using the same batch of experiences, contrasting with the singlo-actor update where only one randomly chosen actor is updated per step. Updates occur al every limestep with batch size of 32.
Reward
1.0
0.5
0.0
-1
Figure 12: 1-D Continuous Bandit environment
Figure 13 tracks cumulative action selection probabilities over time for three actors, revealing distinct convergence patterns.
Slip# 2000
Blepd-4000
Slepd 7000
Slepd 10000
Random Setection Group
Randon Randam Single
Random Random Single
1.0
Random Random Singa
Random Rardom Sinfle
Random All
Random All
Random All
Random +Al
0.55
0.5
0.5
0.5
0.0
Gresdy Swardor Group
Smedy Random Single Gready All
1.0
Gready Random Single Croady All
1.0
Grundy Random Singa Cready Al
Gmedy Rardam Singl Gready AAl
0.5
0.6
0.5
0.6
0.0
Figure 13: Cumulative action selection probabilities under four actor- management schemnes with three actors. Random-Single maintains diversity but also trapped between optimal peaks.
B.4 Running Statistics Normalization (RSNorm)
We adopt this normalization technique from (Lee et al., 2025), which demonstrates its superior performance compared to alternative normalization methods. RSNorm standardizes input observations by tracking the running mean and variance of each input dimension during training, preventing features with disproportionately large values from dominating the learning process.
20

---

Under review as submission to TMLR
Given an input observation or Rd at timestep t, we update the running observation mean μe R variance of E Rd as follows:
and Ne = = de. of of= (1
(1)
where 6, = - and d, denotes the dimension of the observation. Once jt and of are computed, each input observation or is normalized
o, RSNNrm((o)) 0: Pr
(2)
Vož +
where E Rd is the normalized output, and is small constant for numerical stability.
In case, after applying RSNorm, np. digitize to discretize the state space. For Walker2d, we discretize the 17-dimensional observation space into 10 bins, and for Humanoid, we discretize the 347-dimensional observation space inlo 50 bins.
B.5 Overestimation Bias in Pruning Decisions
A potential concern with performance-based pruming is whether overestimation bias in Q-networks could lead to incorrect pruning decisions, where actors with better value estimates are mistakenly removed due to critic inaccuracies.
We do not. observe this situation in experiments, probably due to several mitigating factors. First, we inherit. double Q-learning mechanism, which addresses overestimation bias by taking the minimum of two critic estimates. Second, only actors that consistently underperform get identified for pruning. Third, during the initial freeze period where both actors and crities have high variance, no pruning occurs. After this period, our pruning frequency is every 10,000 steps, which is conservative and allows critics to mature.
average Q-values over entire batches, which naturally smooths out extreme outliers
A particularly influential parameter may be batch size, as smaller batches introduce higher variance in Q-value estimation, potentially causing the pruning issues mentioned above. The following table shows experiments across different batch sizes yield similar performance across several domains (5 runs):
Environment
Batch Size
256
512
1024
Walker2d-v5 4,209-1467
5,181+227
5,173:194
Humanoid-v5
1,429++125
5,552+198
5,341+186
Ant-v5
5,632-1999
5,462+191
5,648+103
Table 3: Performance consistency across batch sizes (5 runs)
B.6
Integration of Pruning Strategies with GAMID and PMOE
Figure 14 shows that integrating pruning strategies improves performance for both GAMID and PMOE in HalfChoctah and Humanoid. The largest gains occur when both rodundancy-based and performanco-based criteria are combined.
B.7 Learning Curves of AEAP on Gym-v4 and Gym-v5
Figure 16 demonstrates the performance of AEAP on MuJoCo-v5 environments under the aggresive parameter configuration: N = actors with distance threshold Ç = 1.5 - ddim(AA Gmax-
Figure 16 demonstrates the performance of AEAP on MuJoCo-v4 environments under the conservative parameter configuration: N = 3 actors with distance threshold ( 1.0 /dm((A) 6max- The results confirm
21

---

Under review as submission to TMLR
No Pruning
Performance Pruning (+P)
Redundancy Pruning (+R)
Both Pruniing (+PR)
Average Return (t)
Average Retum (t)
PMOE
GAMID
+P
+P
+R
+R
PR
+PR
4900
5200
5500
4900
5200
5500
(a)
Humanoid-vō environment (17-D action space)
No Pruning
Performance Pruning (+P)
Redundancy Pruning (+R)
Both Pruninng (+PR)
Average Return (t)
Average Retum (t)
PMOE
GAMID
+P
+P
+R
+R
PR
+PR
4200
4800
4200
4800
(b) HalfCheetah-vä environment (6-D action space)
Figure 14: Integration of pruning strategies with GAMID and PMOE different Mujoco tasks.
that AEAP consistently outperforms matches baseline methods across different Gym environment versions, establishing the robustness of our approach to environmental variations.
B.8 Additional Ablation Studies Main Hyperparameters
Following standard practice in (Haarnoja et al., 2018), we provide comprehensive sensitivity analyses across all MuJoCo environments. Figures 17 through 19 extend the hyperparameter analysis from Section 5.2, presenting detailed ablation studies on initial actor count N and distance pruning threshold for each environment. These results validate the robustness our proposed configurations and confirm the parameter selection guidelines across diverse continuous control tasks.
C Hyperparameters And Compute
We used the luned hyperparameter values for SAC and TD3, that have been provided in RL Baselines3 Zoo which is built on top of Raffin et al. (2021). For Gamid-PG, we used hyperparameters recommended in Dey & Sharon (2024). For SimBa, we used recommended hyperparameters from Lec al. (2025) with minor adjustments.
All reported experiments were distributed between 3 machines; (1) machine with 64 32-core AMD Ryzen Threadripper PRO 5975WX CPUs, cach clocked at 4.3 GHz with 250 GB RAM with 2 NVIDIA GeForce RTX 3090 24 GB GPUs (2) a machine with an Intel(R) Core(TM) 19-13900K CPU with 24 cores (8 performance + 16 efficiency cores), base clock at 3.0 GHz (boost up to 5.8 GHz) with 32 GB RAM and an NVIDIA GeForce RTX 4090 24 GB GPU.
22

---

Under review as submission to TMLR
AEAP
TD3
GAMID-PG
SAC
PMOE
SimBa
Swimmer-v5
Hopper-v5
Walker2d-vb +v5
400
4000
6000
Averogg Reward
800
8000
4000
3000
200
2000
2000
100
1000
0.8
0.8
Ant-v5
HalfChoeiah.v5
Humaanoi-vv5
6000
12000
Avenge Revard
10000
4000
7500
3000
2020
2000
1000
1000
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
3.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
08
1.
FetchPush-v4
FetchSllie-vil
FetchPiakAndPlace-v
1.0
1.0
1.0
0.8
0.8
0.6
0.4
0.4
0.4
Steps (166)
Staps (1e6)
Steps (1o6)
Figure 15: Training performance on MuJoCo-v5 environments. Shaded regions represent standard deviation
across 7 independent trials. Curves are smoothed using a 100-step moving window for clarity.
TD3
SNC
Hospar-a4
Walker2s-od
400
Average Rewerd
300
3000
4000
3000
200
2000
2000
100
1000
0.2
1.0
0.6
0.0
08
HalChhebshhed
Humenoiddoo
6000
Avarage Reward Roward
4000
10000
5000
8000
3000
4000
3000
2000
4000
2000
2000
1000
(1ap)
Slaps (1a6)
Slapa (1e6)
Figure 16: Training performance on MuJoCo-v4 environments. Shaded regions represent standard deviation
across independent trials. Curves are smoothed using a 100-step moving window for clarity.
23

---

Under review as submission to TMLR
N
TD3
{ 0.5 ddi(.A)) amax
= 1.0 - dim(AA) amax
< 1.5 vdim(A) -mma
6000
6000
6000
Average Reward
3000
3000
3000
Average Number of Actors
7
5
5
1
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5 Steps(1e6)
1.0
Steps(1e6)
Steps(1e6)
Figure 17: Hyperparameter ablation results for Hopper-v5 environment.
=
TD3
= ddim(A) amax
{ 1.0 /dim(A) amax
v.00-dim(A)))
{= 1 /dim(A) amax
6000
6000
6000
Average Reward
3000
3000
3000
0
Average Number of Actors
7
5
5
1
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5 Steps(1e6)
1.0
Steps(1e6)
Steps(1e6)
Figure 18: Hyperparameter ablation results for Ant-v5 environment.
24

---

Under review as submission to TMLR
N
N =
N =
TD3
Ç = 0.5 dim(.A) amax
1.0-v dim(.A)
< 1.5-v vdi((A) -amxx
Average Reward
8000
6000
6000
3000
3000
3000
Average Number of Actors
5
5
3
1.0
0.5
0.0
Steps(1e6)
Steps(1e6)
Steps(1e6)
Figure 19: Hyperparameter ablation results for HalfCheetah-v5 environment.
Table 4: SimBa hyporparamoters.
Hyperparameter
Value
Critic block type
SimBa Residual
Critic num blocks
2
Critic hidden dim
512
Critic learning rate
1c-4
Target critic momentum (7)
5e-3 + 1e-3
Actor block type
SimBa Residual
Actor num blocks
Actor hidden dim
128
Actor learning rate
1e-4
3e-4
Initial temperature (ao)
le-2
Temperature learning rale Target entropy (H²).
le-4
|A|/2
Batch size
256
Optimizer
AdamW
Optimizer momentum 82)
(0.9, 0.999)
Weight decay (A)
le-2
Discount (7)
Heuristic
Replay ratio
2 -
HumanoidBench: True
Clipped Double Q
Other Envs: False
25


Reviews:
[{"review_id": "xyrUn6oI4b", "reviewer": "Reviewer_ogz1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new RL algorithm Actor Ensemble with Adaptive Pruning (AEAP), an ensemble-based method that shows strong performance while also being computationally efficient. AEAP combines randomized actor selection with adaptive pruning to maintain diversity and improve efficiency. Extensive experiments and analysis are provided.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provide a quite extensive list of analysis and ablations to help support the claims. \n\n- Figure 4 is interesting and shows Dual-random variant has multimodal distribution even at a later training steps. Figure 3 and 5 also help support the diversity claims. \n- Ablations such as in Figure 6 show the effect of each component. Figure 8 shows it is relatively robust to hyperparameters. \n- Figure 7 shows the proposed method is quite strong considering both performance and computation efficiency. \n\nDiscussions in the paper are quite objective and covered strength of the method but also places where it does not perform the top. Overall I find the claims to be fairly convincing.", "audience_yn": "Yes", "audience_explanation": "Ensemble-based RL algorithms have been important in improving the performance of RL algorithms, though they can bring a big computation overhead. Studying how to reduce such overhead while maintaining or further improving performance is a fairly important topic. This paper can be especially interesting to people who work in deep reinforcement learning.", "requested_changes": "- What are x-axis and y-axis of Figure 4? Please clarify\n- It is slightly confusing what the x-axis is in Figure 8 and 9, can you add a bit more caption", "broader_impact_concerns": "N/A", "additional_comments": "Overall a well-written paper, I appreciate the extensive analysis and experiments. \n\nDownsides: the additional complexity is not ideal, the performance improvement alone is also not that strong compared to the baselines. Though a combination of performance and efficiency are good. \n\nMinor issue: can always benefit from even more experiments on harder benchmarks. Though I acknowledge the current experiments are quite extensive. \n\nQuestion: if we do not consider computation efficiency and focus on sample efficiency, can the performance of the proposed method be further improved in terms of sample efficiency? \n\nQuestion: how well fine-tuned are the baseline methods on some of the fetch environments given they are not typically benchmarked on these? (I believe)\n\nQuestion: \"Actors are progressively removed when their average Q-value falls below 85% of the best performer\" interesting way to prune them. Now, what happens if say the Q networks are getting overestimation problems? Could there be cases when the lower Q value ones actually have better estimate of their values and pruning them might be a mistake? \n\nQuestion: When all training is finished, do you keep a single actor?", "extra_scores": {}, "extra_text": {}}, {"review_id": "U5kFakPPoO", "reviewer": "Reviewer_nf1p", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new reinforcement learning algorithm, AEAP (Actor  Ensemble with Adaptive Pruning), which addresses the challenge of intractable policy  gradients often encountered in ensemble methods using stochastic policies. By  adopting deterministic policies, the proposed method improves gradient tractability  and introduces two key innovations: (1) Dual-Randomized Actor Selection: This mechanism decouples exploration and  learning by employing different random actor selections for environment interaction  and policy updates. It effectively maintains policy diversity without requiring explicit  regularization terms. (2) Actor Pruning: Actors are progressively pruned based on two complementary criteria, namely estimated value from the critic and action-space similarity, allowing the framework to eliminate underperforming or redundant actors dynamically. Extensive experiments on both MuJoCo and Fetch benchmark tasks demonstrate AEAP's competitive performance and computational efficiency compared to the existing baselines. \n\n**Strengths** \n\n1. The proposed algorithm is conceptually simple and easy to follow. By presenting  experimental analysis to motivate the design choices, the paper illustrates how each modification contributes to the performance improvement. \n\n2. The experiments are quite thorough, demonstrating that AEAP consistently  outperforms other deterministic methods (TD3, Gamid-PG) across most environments. Regarding training efficiency, AEAP also surpasses other multi-actor ensemble approaches. Additionally,  the paper includes ablation studies on the number of actors and the pruning threshold, further validating the effectiveness of the proposed design. \n\n3. As an extension of Gamid, AEAP addresses the limitations of actor ensembles in  sparse reward tasks, where performance often degrades. Moreover, it achieves this  with lower computational overhead. \n\n**Weaknesses** \n\n1. Although the paper states that the motivation is to address the difficulty of tuning \nregularization parameters, the proposed method introduces several new pruning related hyperparameters. While two general configurations are provided, the  results still show sensitivity to different tasks and environments. Therefore, the  approach appears less attractive as a solution for improving algorithmic stability. \n\n2. The paper focuses primarily on practical implementation improvements, but lacks  theoretical grounding to support the proposed method. Specifically, it remains somewhat unclear why dual-randomized actor selection can in general encourage exploration. I can see that the proposed approach appears to work well in the 1D toy example (cf. Figure 2), but this result does not imply that this holds for RL problems in general. More justifications are needed to support that this design is indeed principled. \n\n3. Based on Table 2, it seems that the performance of AEAP is statistically about the same as the vanilla SAC in most of the tasks (e.g., Hopper, Humanoid, Halfcheetah, Swimmer, and the three Fetch tasks). Therefore, this makes AEAP not that appealing as simple SAC can already achieve similar total returns. \n\n4. Some of the discussions on actor ensemble need further explanations:\n- Page 2, first paragraph: It is stated that “when treating stochastic ensemble policies as components of a probabilistic mixture, computing exact policy gradients becomes analytically intractable, requiring surrogate approximations that can compromise performance.” However, in typical RL, one does not need to compute the policy gradient exactly through an analytical form. Instead, a stochastic estimate of the gradient is sufficient. For example, even if the action distribution follows a mixture Gaussian, an estimate of the policy gradient can still be calculated as long as the score function $\\nabla \\log\\pi(a\\rvert s)$ can be calculated numerically. Therefore, I do not see why this is actually a challenge.\n- Page 4, third paragraph of Section 4.2: It is stated that “A plausible explanation is that, maintaining several actors benefits exploration in the early phase when state–action visitation is sparse. However, as exploration becomes less valuable relative to exploitation, higher $\\epsilon$ enforces more random selections when the algorithm should exploit the superior actor, and thus leads to worse performance.” As Table 1 only shows the final action selection ratios and final mean performance under different $\\epsilon$, the result does not provide much information about the training dynamics. Hence, it is unclear why Table 1 suggests that pruning is helpful at a later training stage.", "claims_and_evidence_yn": "Yes", "claims_explanation": "For both proposed enhancements, the authors provide empirical evidence  demonstrating that the method effectively enhances exploration without requiring  additional regularization terms, while also reducing computational cost. AEAP also  clearly outperforms other ensemble RL methods in sparse reward settings. However, as noted in the weaknesses, it remains unconvincing that replacing regularization with pruning-based hyperparameters leads to a more stable or generally robust solution.", "audience_yn": "Yes", "audience_explanation": "The ideas of dual-randomized actor selection and pruning mechanisms are likely to be of interest to TMLR’s audience, particularly those researching ensemble-based RL or focused on improving exploration, despite that AEAP does not always achieve the best performance across all environments.", "requested_changes": "1. Including theoretically-grounded explanation or analysis and additional empirical evidence demonstrating that the proposed pruning strategy is more robust than regularization-based approaches. \n\n2. Currently, the paper primarily highlights how AEAP improves the exploration  issue in TD3. It would be beneficial to provide further explanation on why AEAP outperforms other actor ensemble methods specifically in sparse reward environments. \n\n3. To make sure that the improvement of AEAP over the baselines is indeed statistically significant across tasks, some reliable metrics (e.g., interquartile mean) can also be provided to strengthen the empirical results.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "utEpqcfTmo", "reviewer": "Reviewer_vsrG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel algorithm for actor ensembles in continuous control reinforcement learning. The core contributions are the introduction of a \"dual-randomized\" strategy, in which a random actor is chosen to execute actions in the environment, and independently an actor is chose which receives a policy update. In addition, the paper proposes actor pruning as actors either collapse to the same policy over time, or lag behind in performance.\n\nThe paper presents a mostly empirical, and thorough, evaluations of the proposed method without theoretical investigation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claimed contributions seem mostly supported by the empirical evidence. Overall, the ideas are sound, however, their impact on performance as well as supporting evidence for the claims is somewhat thin.\n\nThis is mostly due to three issues:\n* The paper proposes two orthogonal improvements to actor-ensemble training, which do not seem to be related in an intuitive manner.\n* The proposed method for ensuring diversity is heuristic in nature.\n* The paper evaluates baselines and other methods with strongly outdated architectures.\n\nThe first issue is hopefully somewhat self explanatory. I think that it would be useful to discuss pruning when combined with other strategies for obtaining diverse expert actors, such as those presented by the baselines. The idea seems to be readily combineable with other.\n\nThe second issue arises because the mechanisms which underlie the dual-actor selection method are not discussed or evaluated in detail. While a theoretical investigation would be great, but is potentially out-of-scope for the paper, I think it would greatly benefit the community if the work sought to clarify why the proposed strategy leads to multimodal policies. In the context of replay-buffer based off-policy learning, all actors will randomly see all data at some point, and so it is not quite clear whether the proposed strategy does indeed lead to stable diversity, or simply delays the updates of all actors.\n\nThe third issue arises as the authors seem to only test their method with outdated architectures for TD3 and SAC. A lot of recent work [1,2,3,4] has focused on producing more reliable learning in actor-critic methods, and it should be evaluated if the strategies proposed here robustly generalize to state-of-the-art approaches. This is highly relevant as the much more stable value learning of these approaches might necessitate explicit regularization to ensure actor diversity.\n\nWhile it has sadly been a standard in RL to base advancements on highly outdated algorithms and architectures such as SAC and TD3 in their original formulation, I have decided to become more strict in admonishing this common practice in my reviews. I ask the AC to intervene in case they believe this is take into consideration that this is a harsher standard than other reviewers might apply.\n\n[1] Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control, Nauman et al., Neurips 2024, https://sites.google.com/view/bro-agent/\n[2] SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning, Lee et al., https://openreview.net/forum?id=jXLiDKsuDo\n[3] Dissecting Deep RL with High Update Ratios: Combatting Value Divergence, Hussing et al., https://openreview.net/forum?id=ofwv9VYp3h\n[4] CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity, Bhatt et al., https://openreview.net/forum?id=PczQtTsTIX", "audience_yn": "Yes", "audience_explanation": "The paper addresses an important question, effectively training ensembles in off-policy actor critic settings.", "requested_changes": "The ordering of the sections in the paper is slightly suboptimal. It would be preferable to have a fully specified algorithm (Section 4.3) before showing experiments (Section 4.1). Otherwise the reader is left wondering what exactly is being evaluated.\n\nFigure 5 is deeply confusing, I think it would be much clearer to show performance over time, or time to reach some set reward in a standard bar plot, than merely describing the time to finish the experiments. In addition, the complete lack of scale on the graph makes it seem arbitrary and like a poor choice of visualization.\n\nMinor typos:\n\n4.3 Actor Ensmble with Adaptive Pruning > Actor Ensemble", "broader_impact_concerns": "n/a", "additional_comments": "n/a", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)