Rethinking Policy Diversity in Ensemble Policy Gradient in Large-Scale Reinforcement Learning
Authors: Naoki Shitanda
Motoki Omura
Tatsuya Harada
Takayuki Osa
Venue: ICLR 2026 Poster (2026)
Keywords: Distributed Reinforcement Learning
Agent Ensemble Learning
Agent Diversity
Exploration Efficiency
Primary area: reinforcement learning
TL;DR: Appropriately controlled policy diversity improves the learning efficiency of ensemble RL in large-scale environments.

Abstract: Scaling reinforcement learning to tens of thousands of parallel environments requires overcoming the limited exploration capacity of a single policy. Ensemble-based policy gradient methods, which employ multiple policies to collect diverse samples, have recently been proposed to promote exploration. However, merely broadening the exploration space does not always enhance learning capability, since excessive exploration can reduce exploration quality or compromise training stability. In this work, we theoretically analyze the impact of inter-policy diversity on learning efficiency in policy ensembles, and propose Coupled Policy Optimization which regulates diversity through KL constraints between policies. The proposed method enables effective exploration and outperforms strong baselines such as SAPG, PBT, and PPO across multiple tasks, including challenging dexterous manipulation, in terms of both sample efficiency and final performance. Furthermore, analysis of policy diversity and effective sample size during training reveals that follower policies naturally distribute around the leader, demonstrating the emergence of structured and efficient exploratory behavior. Our results indicate that diverse exploration under appropriate regulation is key to achieving stable and sample-efficient learning in ensemble policy gradient methods. Project page at https://naoki04.github.io/paper-cpo/ .

Published as a conference paper at ICLR 2026
RETHINKING POLICY DIVERSITY IN ENSEMBLE POL- ICY GRADIENT IN LARGE-SCALE REINFORCEMENT LEARNING
Naoki Shitanda 12, Motoki Omura', Tatsuya Harada?.,, Takayuki Osa"
(shitanda, omura, harada] @mi u-t.koooooo........ ac. ip takayuki www.rikare....... ip
The University of Tokyo, Tokyo, Japan
2 RIKENN Center for Advanced Intelligence Project, Tokyo, Japan
ABSTRACT
Scaling reinforcement learning to tens of thousands of parallel environments re- quires overcoming the limited exploration capacity of single policy. Ensemble- based policy gradient methods, which employ multiple policies to collect diverse samples, have recently been proposed to promote exploration. However, merely broadening the exploration space does not always enhance learning capability, since excessive exploration can reduce exploration quality or compromise train- ing stability. In this work, we theoretically analyze the impact of inter-policy diversity learning efficiency in policy ensembles, and propose Coupled Policy Optimization which regulates diversity through KL constraints between policies. The proposed method enables effective exploration and outperforms strong base- lines such as SAPG, PBT, and PPO across multiple tasks, including challenging dexterous manipulation, in terms of both sample efficiency and final performance. Furthermore, analysis of policy diversity and effective sample size during training reveals that follower policies naturally distribute around the leader, demonstrating the emergence of structured and efficient exploratory behavior. Our results indicate that diverse exploration under appropriate regulation key to achieving stable and sample-efficient learning in ensemble policy gradient methods. Project page at httpp:://aokii/ github. ww/paper-cpppeeeeeeeeeeeee /
INTRODUCTION
With the advent of GPU-based massively parallel physics simulators such as Isaac Gym (Makoviychuk et al. 202I) and Genesis (Genesis Authors 2024), it has become feasible to collect data from over tens of thousands of environments simultancously for robot deep reinforcement learning (RL). Given the inherently trial-and-error nature of RL., such parallelism has the potential to dramatically improve learning efficiency for high-dimensional and complex tasks, such as dexterous hand manipulation. However, recent work (Singla al.) (024)) has reported that simply increasing the amount of data does not necessarily lead to improved learning efficiency in on-policy methods like PPO (Schulman et al.) 2017). This result suggests that simply using single policy in massively parallelized environments does not sufficiently diversify exploration and thus cannot significantly improve learning efficiency.
To address these challenges, agent ensemble approaches have been proposed collect diverse samples. Recent work (Singla et al., (2024) introduced seaderr-followe framework shown in Fig. Fig. in which one leader agent and multiple followers are each assigned to separate blocks of parallel environments. Each follower performs independent on-policy learning, while the leader aggregates off-policy samples from followers using importance sampling (IS). Unlike other agent ensemble methods (Aleksei Petrenko 2023 Liet 2023a), this enables the use of all collected data without discarding any samples, thereby facilitating diverse exploration. Their approach has demonstrated significantly improved learning performance
non-aggrgg methods like DexPBT (Aleksei Petrenko [202)). as well as over off-policy methods such as PQL (Li et al.) 2023b). However, it remains an open question whether greater inter-policy diversity necessarily translates into better performance.

---

Published as a conference paper at ICLR 2026
(a) Agents Ensemble RL
(b) Excessive Diversity Harms Training Stability
(c) Coupled Policy Optimization
(Policy Space)
(Policy Space)
Leader Agent
F2
Off Off-Policy Sampling
Misa ligned follower's samples may reduce effective sample size increase bias by clipping in PPO
F3
cause loss monotonic improvement
Adversarial
KL Constraints
leader's off-policy update.
Reward
Follower Agents
Figure I: Appropriately controlled policy diversity improves the learning efficiency of ensemble RL in large-scale environments. (a) The leader-follower approach is an agent ensemble method that aggregates samples from multiple followers into leader policy. (b) Misalignment between policies may causes decline in sample efficiency and training stability. (c) Our method introduces KI. divergence constraints keep followers distributed around the leader, as well as adversarial reward to prevent policies overconcentration.
In this work, we theoretically and empirically investigate the impact of inter-policy diversity on ensemble policy gradient methods, showing that excessive diversity harm both training stability and sample efficiency as shown in Fig. [b). To address this issue, we propose Coupled Policy Optimization (CPO), novel method that introduces KL divergence constraint during follower policy updates in the leader-follower framework, thereby promoting diverse yet well-structured exploration around the leader (Fig. TT)). In addition, prevent overconcentration among policies, we incorporate an adversarial reward that discriminates agent identity from state- state-action pairs, ensuring balanced and effective diversity.
Extensive experiments on dexterous manipulation, gripper- based manipulation and locomotion tasks demonstrate that our method outperforms strong baselines such as SAPG, DexPBT, and PPO in both sample efficiency and final performance. In addition, we confirm that the KL constraint drives the IS ratios closer one, which increases the effective sample size (ESS) and mitigates the clipping bias in PPO, thereby improving the effective sample efficiency and training stability. Furthermore, analysis of the ensemble policies reveals that SAPG suffers from severe policy misalignment, where some follower policies diverge significantly from the leader, hindering leaning ability. In contrast, CPO naturally induces a stable and well-structured policy formation, with followers distributed around the leader in balanced manner.
To summarize, our main contributions are as follows:
We provide a theoretical analysis showing that excessive inter-policy diversity in ensemble policy gradient methods degrades training stability and sample efficiency.
We propose CPO, leader-follower framework that introduces KI divergence straint and adversarial reward during follower updates to enable effective and stable exploration in policy space. The proposed method outperforms strong baselines including SAPG, DexPBT, and PPO across challenging robotic tasks.
We empirically verify that the KL constraint keeps IS ratios close to one leader's off- policy policy update, leading to improved sample efficiency.
Through inter-police KL divergence analysis, we show that CPO naturally induces structured policy formation which follower policies are consistently distributed around the leader policy, avoiding the policy misalignment observed in prior method.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
2.1 DISTRIBUTED REINFORCEMENT LEARNING
Deep reinforcement learning (RL) relies on tnda--and-error, and increased data collection through massively parallel environments directly contributes to performance improvement. Early work focused on asynchronous distributed algorithms across multiple devices with hundreds thousands of environments, favoring off-policy methods (Espeholt et al.) 2018] |Horgan et al., 2018] Espehhlt (ct al. 2019). Recently, GPU-based simulators such as Isaac Gym (Makoviychuk et al.) 2021 1) have enabled tens of thousands of environments to run synchronously on a single device, reviving interest in on-policy methods that often achieve higher final performance in robotic tasks (Rudin et al. 2022
Handa et al.] 2023 Zhuang al. 2023 Li et al.; 2023b). However, naively scaling methods like PPO to such large numbers of environments yields dimimishing returns, since a single policy provides limited exploration diversity, resulting in similar trajectories (Singla et al. 2024).
2.2 AGENT ENSEMBLE PARALLELED ENVIRONMENTS
To enhance exploration diversity in massively parallel environments, ensemble methods with multiple policies have been explored. DexPBT (Aleksei Petrenko 2023). for example, trains policies with different hyperparameters in parallel but discards data from non-selected policies, reducing overall efficiency. SAPG (Singla et al., 2024) instead leverages all follower data through IS in a leader- follower framework, improving exploration diversity and training stability. Yet, the impact of inter-policy diversity has not been thoroughly examined, and excessively divergent followers may generate ofl-policy samples that destabilize the leader.
2.3 POLICY UPDATE WITH REGULARIZATION
Policy regularization is widely used in RL, typically constraining divergence either from the dataset
policy in offline RI (Fujimoto & Gu 2027 Garg al., 2023
Sikchi al.) Nair et al. 2020) or
from the old policy in online RL (Schulman et al. 2015
2017 Abdolmaleki et al. 2018 18) thereby
improving stability and efficiency. Beyond being used purely as constraints, policy regularization has also been extended to explicitly promote diversity across multiple policies, using behavioral or distributional distance metrics (Parker-Holde et al. 2020 Yao et al. 2023 Wu al. 2022]. this
work, we regularize the divergence between follower and leader policies so that followers explore near the leader policy space, collecting data informative to the leader while maintaining diversity. For this purpose, and motivated by our theoretical analysis of the leader policy's 's learning dynamics, we employ KL divergence, following prior approaches such as XQL (Garg et al. 2023) and AWAC (Nair et al. 2020).
3 PRELIMINARIES
In this paper, we theoretically show that excessive inter-policy diversity in ensemble policy gradient methods under massively parallel environments can harm training stability and sample efficiency, and we propose a method that controls the diversity between agents to promote efficient exploration. Since both our analysis and the proposed method build upon the leader-follower framework of SAPG Singla et al., 2024). we first review the formulation of a fundamental on- on-policy algorithm, PPO (Schulman et al. 2017). and then summarize the key ideas and limitations of SAPG.
3.1 REINFORCEMENT LEARNING
In RL, tasks are typically formalized as a Markov Decision Process (MDP), defined by a tuple P, Here, S is the state space, A is the action space, P( 1 az) is the state transition probability density, r(s, a) is the reward function, ? is the discount factor, and d(su) is the initial state distribution. A policy (()ss) S x A + IR is defined actions conditioned on the state. The objective of RL is to learn a policy that maximizes the expected return E. Ro [r] where R2 = Ef - 8-r a;) and T is task horizon.
probability distribution over
3

---

Published as a conference paper at ICLR 2026
3.2 PROXIMAL POLICY OPTIMIZATION (PPO)
PPO is a widely used on-policy algorithm that stabilizes updates by clipping the IS ratio with the behavior policy, All agents in this study trained with PPO with modifications. The objective is:
(0) aa [min(r())A(s,, clip(((()) - e, + +)
(I)
where r(0) [e(aaa)
=aa (a|s) is the IS ratio and is the clipping parameter. The advantage function is A(s, a) (s, a) V (s), with the action-value function (s a.)
E,, R al and the value function (s) = Ed, [R] s). Thus, A(s, a) measures how much better action a is compared to the average action under T..
3.3 SPLIT AND AGGREGATE POLICY GRADIENTS (SAPG)
SAPG is state-of-the RL method designed to enhance exploration diversity and sample efficiency in massively parallel environments. It trains multiple policies concurrently, where each follower agent collects data that is aggregated into a leader policy. The leader leverages off-policy data from followers through IS, enabling diverse exploration with parallel environments.
Specifically, the N parallel environments are divided into M blocks, and one leader policy and M M-1 follower policies are each assigned to the blocks. All agents share the same policy and value networks conditioned identification vectors. The leader policy TL, (al s) and follower policies TF. (als). where E ..., ( M-2), are updated by the objective functions in Eq. and Eq.
LSARG.L(O..). -B..arrio Lold
min (r (s) a), (0)((r 1-c, (4
[mi (Tiaa (O) (0) a), clip(r 10)1
LSAPG.F. (0) (0)= Fa [min (0) a), clip(rF, (0), 10), (0)
where j {.....: 2 denotes the index of follower agent randomly sampled each training
epoch, and the density ratios between behavior policy and the updating policy are defined as:
LL (a|s) La (als)
FL (0) = TL (a|s)
(0) = TF (als)
The (0)
F, (als)
F. And (a.s)
Here, F a) and ( a) denote the advantage functions for the i-th follower and the leader policy, respectively. Furthermore, SAPG introduces an entropy regularization term applied to all policies to encourage diversity in exploration across agents.
However, SAPG lacks an explicit mechanism to control the distance between leader and follower policies while applying entropy regularization, which may cause followers to drift significantly from the leader. As preliminary study, we conducted an ablation on the entropy regularization term in SAPG and found that, although it promotes exploration and can improve sample efficiency, also increases the follower-leader KL divergence and often leads to severe misalignment. See Appendix A.7 for details. In this paper, analyze how such excessive divergence affects learning.
4 EFFECT OF POLICY DIVERSITY ON ENSEMBLE POLICY GRADIENT
Policy diversity affects ensemble policy gradient methods in two major aspects: data coverage and training stability. While diverse exploration increases coverage and mitigates local optima, excessive diversity reduces sample density and weakens the variance-reduucion effect of parallel environments, reflecting a fundamental excessive divergence between the leader and follower policies can directly harm training stability and sample efficiency. We formalize this intuition through the following propositions.
opploration-exploitaiii
trade-off in reinforcement learning. More critically,
The expected absolute deviation of the IS ratio from is inversely related to the effective sample size (ESS); the deviation increases, the ESS decreases.
Proposition! 1.
4

---

Published as a conference paper at ICLR 2026
When the leader and follower policies diverge, the expected absolute deviation of the IS ratio for leader update with follower samples, Es Faa (a|s) increases. This deviation leads
(ala)
higher variance the ratio, thereby diminishing the ESS, which is standard metric of sample
efficiency in IS with approximation (Martino et al. 2017), where is the IS ratio, defined as follows:
WF
ESS
W/
1=
2==1 w
Intuitively, samples from misaligned follower policies contribute little to the leader's learning. thereby reducing the overall sample efficiency of the leader update. The detailed derivation is provided in Appendix A.I.I 1
Proposition 2. The L norm of the bias of the gradient estimate induced by the PPO clipping operator is upper bounded by the square root of expectation involving the IS ratio deviation.
PPO ensures learning stability by clipping the IS ratio, however, this introduces bias into the gradient estimate. As the IS deviation increases, the effect of clipping becomes more pronounced, resulting in larger bias and destabilizing the leader's 's learning. This can be shown by upper bounding the L norm of the bias as a function of the IS deviation. The detailed derivation is provided in Appendix A.1.1
These propositions show that while policy diversity improves exploration, excessive divergence between the leader and follower policies causes the IS ratio to deviate from 1, which may undermine the sample efficiency and stability of the leader update by reducing ESS and increasing the gradient estimation bias, as shown in Fig. [b). We then examine how this deviation can be suppressed.
For the leader update with follower samples, The expected absolute deviation of IS ratio from I is upper bounded by the KL. divergence between the follower and leader policies.
Proposition
Proof. From Pinsker's inequality, we have P - QI1 S V2DKu (PII)) for any two distributions P and Q. Applying this to the leader and follower policies, then:
/( | (als) (I(aas)) da 12DI,.(FF. (1
(6)
Here, using the identity to a ~ Fala
#y (ala)) FFoia [a/s)
TFaia (als))
we take the expectation with respect
TFola (a|s)
Ty (a)s)
FFa (als) L (a|s)
EawaFald
Fosa (als)
da
(7)
T Fua (a)s))
T Fosa (als)
22 FFa (-1s) [-)()
(8)
Furthermore, assuming reachability, we take the expectation over the states encountered by the follower policy. This yields Ex,a--FFa that the KL divergence increases, the IS ratio deviates further from 1.
S 22DKI, (118)[(= showing Fan (a|s)
Consequently, introducing a constraint on the KI. divergence between the leader and follower policies alleviates the IS ratio deviation, [Schulman et al. (2015) and Xie al. (2025) also argue that long as the KL divergence or IS ratio deviation between the target and behavior policies remains small, the update error due distribution shift reduced, and performance improvement is guaranteed. These motivate the need for KL-based coupling between leader and followers, to regulate policy diversity in ensemble policy gradient methods.
COUPLED POLICY OPTIMIZATION
Building upon the theoretical observation in section 4 propose CPO, method that regulates the
inter-agent distance during training. Our approach extends SAPG (Singla et al. 2024) by constraining the KL divergence between the leader and each follower policy during follower updates, enabling diverse yet meaningful exploration for the leader. Furthermore, we introduce an auxiliary adversarial reward that encourages diversity across follower policies, prevent overconcentration of agents.
5

---

Published as a conference paper at ICLR 2026
5.1 FOLLOWER'S ' POLICY UPDATE UNDER KL CONSTRAINT
We formulate the update of each follower policy as a constrained optimization problem with KL divergence constraint to the leader policy:
" (a arg max Apt (s, s.t. (TF( [|)((()))) S EKL- xF.
Following the approach of AWAC (Nair al. 2020), this problem admits closed-form parametric solution, which we then approximate with a neural network policy "F The resulting parametric objective of follower update can be written follows:
LCPO,F; (0) -a,,s
log FF (als) exp
1 (f
+ LSAPG.F. (10)
where, A is temperature parameter control the strength of KL constraint. The detailed derivation of Eq. 10 provided in Appendix A.2 Thus, the policy objective of our proposed method, (f,
can be expressed as an extension of the SAPG policy objective [sAPG((..) as:
Lcro(0) LSAPG(O. +
Σ
FCro,F., (0,
(11)
RE(O....MM22)
where B is coefficient introduced to roughly match the scale between the PPO objective and the KL-regularized loss term, which involves an exponential. The pseudocode of our method and the discussion on computational complexity are provided in Appendix A.3
5.2 ADVERSARIAL REWARD FOR FOLLOWERS DISTRIBUTION
In our method, follower policies are trained to explore within KL-bounded neighborhood around the leader to preserve the stability of the leader's off-policy PPO update. While it prevents harmful misalignment issue, also indirectly pulls followers closer one another, which can reduce the diversity of their state-action coverage within the neighborhood.
To mitigate the issue, introduce intrinsic reward to encourage sufficient separation among the policies, Although there are various ways to promote diverse exploration, e.g. RDN (Burda et al. 2018) and ICM (Pathak et al. 2017). they are not intended for scenarios where multiple policies perform rollouts in parallel and do not explicitly account for the distance between policies.
Therefore, we draw inspiration from DIAYN (Eysenbach et 2018), which explicitly encourage separation between policies. We train a discriminator De(M 81, at), parameterized by neural network with parameters Ç. to predict the index E (0..... ; }} of the policy given state-action pair and the classification loss is then used an intrinsic reward. This encourages each follower to explore distinct regions in the state-action space, such that the discriminator can identify their identity Given data buffer D, the discriminator loss and the intrinsic reward are given by:
LD(E) (s log De(y (y)ss, az)],
reth (st,, ae. y) Aadv log De st, az).
(12)
Notably, this intrinsic reward is not provided to the leader agent. When the leader is updated from the off-policy samples collected by followers, only the true environment rewards are considered.
6 EXPERIMENTS
We evaluated our method on six dexterous manipulation tasks (Andrychowicz et al. 2020 Alek
sei Petrenko) 2023), those have high dimensional action space, two gripper-based manipulation tasks, and two locomotion tasks to compare its performance against sate-of-the-ar methods under massively parallel settings. All tasks provide dense rewards and the experiments were conducted on Isaac Gym (Makoviychuk et al.) 2021) with N 24, 576 parallel environments, following the experimental setup of the prior work (Singla et al. 2024). Detailed descriptions the tasks are provided in Appendix A.6
For baselines, we selected PPO, DexPBT (Aleksei Petrenko 2023 ) and SAPG (Singla et al. 2024)
All of these methods are built upon PPO.

---

Published as a conference paper at ICLR 2026
PPO (Schulman et al., 2017): is representative policy gradient algorithm widely used across various tasks. We simply increased the number of samples collected per epoch to be equal to the product of the horizon length and the number of environments N.
DexPBT (Aleksei Petrenko, 2023): population-based parallel learning framework that divides the environments into M subsets, where M agents each with different hyperparameters train parallel. Periodically, the sewest-performing agents are removed and replaced by new agents generated through genetic algorithms, which assign updated hyperparameters for the next training phase.
SAPG (Singla al. 2024): adopts agent ensemble learning based on leader-follower network, and it represents the state-of-the- in massively parallel environments to the best our knowledge. The leader agent is updated with not only its own on-policy samples, but also off-policy samples from follower agents through IS.
For DexPBT, SAPG, and our proposed method, we set the number of parallel blocks M In SAPG and our method, the shared networks are conditioned on a one-dimensional vector E R. Hyperparameters common to both SAPG and our method, such as the entropy coefficient, were set to the same values as used in SAPG. The hyperparameters and computing environments used in all experiments are provided Appendix A.6 All experiments were conducted using five random seeds.
7 RESULTS AND ANALYSIS
We analyze the learning performance of our proposed method compared to baselines, conduct an ablation study on the strength of the KL constraint, and examine the evolution of inter-policy KL divergence during training.
7.1
TRAINING PERFORMANCE
To compare the training performance of each method, we present the learning curves on each task in Fig 2 along with the final performance on dexterous manipulation tasks after 2 x 10tH environment steps training summarized in Table Each result shows the mean and standard deviation over five random seeds.
Our proposed method consistently achieves high sample efficiency and strong final performance across all tasks especially on the manipulation tasks. In particular, while PBT fails to learn on the AllegroKuka Regrasping and Franka tasks, and SAPG struggles in the Two-Arms Reorientation task, our method demonstrates robust learning capability. Moreover, many tasks, it reaches SAPG's final performance with approximately half the number of environment steps, indicating the acquisition of efficient exploration ability. No significant improvement over SAPG is observed in the AllegroKuka Regrasping and Throw tasks, which discuss in the next section.
In locomotion tasks, due to their relative simplicity, the performance differences across algorithms are smaller. Nevertheless, PBT exhibits faster convergence, indicating that in simpler tasks, broad parallel exploration can be ad advantageous. On the other hand, our method converges slightly faster
Table 1: Performance on dexterous manipulation tasks after 10t environment steps of training. Bold indicates the method with the highest average performance for each task, as well as those not significantly different from it, as determined by t-test (p 0.05).
Task
PPO
PBT
SAPG
CPO (ours)
ShndowHand
10661 + 1050
10294 + 1728
12882 + 343
13762 + 414
AlnegroHand
10439 + 1282
13239 + 239
11989 + 817
14421 + 885
Regrasping
0.76 0.99
35.26 2.82
37.20 - 0.65
37.44 + 1.21
Reorientation
1.04 + 0.98
2.92 + 4.27
38.79 + 1.66
43.75 1 0.65
Throw
15.69 +: 3.34
19.08 1.02
22.51 +: 1.15
21.69 4 2.44
Two-Arms
Reorientation
1.41 0.80
26.43 11.12
5.11 3.41
35.30 + 2.77
7

---

Published as a conference paper at ICLR 2026
ShadowHand
AllegroHand
AllegroKuka Regrasping
Episode Rewards
15000
Episode Successes
12503
12300
28
12003
10100
2593
Episade
7500
28
5002
5250
3.00
8.00
8,00
Steps
AllegroKuka Reorientation
AllegroKuka Throw
Alllegrokuaa Two-Arms Reorientation
Episade Successas
Episade Successes
Episode successes
32
6.00
5.00
5,00
Env Steps
Env Steps
Env Steps
FranicCCublPush
FranksC.beSlack
Ep Revenns
Epioose Revends
10.0
Staps
Erw Staps
Eri Steps
Enn Steps
CPO (ours)
SAPG
PBT
PPO
Figure 2: Comparison of algorithm performance across ten robotic tasks. Learning curves across six dexterous manipulation, two gripper-based manipulation and two locomotion tasks com- paring CPO to SAPG, PBT, and PPO. CPO consistently achieves higher sample efficiency and final performance, particularly in ShadowHand, AllegroHand. AllegrokukaReorienttttioo Two-Arms Reorientation, FraskaCubePush and Stack.
than SAPG, suggesting that in leader-follower policy gradient frameworks, the stabilization and sample efficiency gains brought by KL constraints outweigh the benefits of broader data coverage through exploration diversity.
We also conducted ablation study isolate the contributions of the adversarial reward and the KL constraint, as shown in Appendix A.4
7.2 ABLATION STUDY ON KL CONSTRAINT
To assess the sensitivity to the KL. constraint hyperparameter and empirically verify the propositions in 4 we conducted an ablation study varying the KL coefficient (Ay) in the Shadow Hand and AllegroKuka Reorientation tasks. To isolate the effect of the KL constraint, we conducted experiments without the adversarial reward. Also, 3 in Eq. 11 was fixed at 0.001.
Fig. presents training curves of our method with different Ay values compared to SAPG, demon- strating that CPO was robust to a wide range of values, consistently outperforming SAPG. A practical tuning heuristic starting with a weak constraint (Ay 0.5) and gradually strengthen
ShadowHand
AllegroKuka Reorientation
Episade Rewards
15003
Episode Succesbee
12500
CPO(Af=0.05)
1:000
CPO(Af=0.1)
CPO
CPO
SAPG
8.0
0.5
0.50
0.75 1,80
Env Stops
1.50
175,
3,00
8.00
025
0.50
1.2 Env Steps
1.00
1.90
1.75
300
Figure Training Curves from the ablation study with different A
8

---

Published as a conference paper at ICLR 2026
Table 2: Mean IS Ratio Deviation and Overall ESS Rate at 5 x 10 environment steps. The reported values are computed by averaging over a window of eleven iterations.
Task
Method
Mean Ratio Deviation (L)
ESS Rate (1)
ShadowHand
SAPG
0.889
0.0223
CPO(0.5)
0.403
0.763
CPO(0.2)
0.297
0,871
CPO(0.1)
0.222
0.923
CPO(0.05)
0.187
0.941
AllegroKuka Reorientation
SAPG
0.608
0.110
CPO(0.5)
0,420
0.721
CPO(0.2)
0.276
0.888
CPO(O.1)
0.214
0,929
CPO(0.05)
0.199
0.938
Table 2 shows the mean IS ratio deviation from I and the ESS (normalized a maximum of 1) at 5 x 10" environment steps. The deviation is computed from all follower samples, and the ESS from all leader and follower samples. Consistent with Proposition in section we observed that stronger KL constraints (smaller Xy) lead to smaller deviations and higher ESS, improving sample efficiency.
7.3 KL DIVERGENCE ANALYSIS
In this section, we analyze the KI. divergence between policies during training to compare agent relationships SAPG and our method (Fig. higher-resolution results are provided Appendix(A.5), In ShadowHand and AllegroKuka Reorientation, where our method clearly outperformed SAPG, several SAPG followers misaligned significantly from the leader, producing harmful samples that hinder the leader's learning, as described in section 4 In contrast, our method maintained stable inter-agent distances, yiclding more informative samples. AllegroKuka Regrasping, where both methods achieved similar performance. SAPG followers did not show noticeable divergence, likely due to incidental alignment between SAPG's shared backbone and the task characteristics.
Interestingly, in our method the leader consistently remained the closest agent every follower (white circles in Fig. 4 suggesting that KL regularization, together with adversarial reward and entropy terms, naturally distributes followers around the leader without overconcentration. Furthermore, unlike SAPG's ablation where all agents sampled from each other, leading to excessive similarity and reduced diversity (Singla al. 2024), our approach preserves the leader-follower asymmetry: each follower learns only from its own on-policy data and the leader's off- off-policy data. This design helps maintain diversity while keeping inter-policy distances under control.
CONCLUSION AND LIMITATION
In this work, we theoretically showed that excessive inter-policy diversity in ensemble policy gradient methods under massively parallel environments can harm sample efficiency and stability by reducing effective sample size, increasing clipping bias, and weakening monotonie improvement guarantees. To address this issue, we proposed Coupled Policy Optimization, which introduces KL constraints be- tween leader and follower policies and adversarial rewards to prevent overconcentration. Experiments on dexterous manipulation, gripper-based manipulation and locomotion tasks demonstrated that CPO outperforms strong baselines such as SAPG, PBT, and PPO both sample efficiency and final performance. Ablation studies confirmed that KL constraint reduces IS-ratio deviation and improves effective sample size, while KL-divergence visualizations revealed that followers naturally distribute around the leader without misalignment, highlighting the stability and structural effectiveness of our method.
These findings suggest that in ensemble policy gradient methods under massively parallel environ- ments, is not sufficient to merely promote policy diversity; rather, appropriate control diversity is crucial for achieving both stable and sample-efficient learning.

---

Published as a conference paper at ICLR 2026
Shadow Hand
Allegro Kuka Reorientation
Allegro Kuka Regrasping
SAPG
CPO (ours)
SAPG
CPO (ours)
SAPG
CPO (ours)
5G Eanv
10G Env Steps
233
AM
5
F
150 Eav Steps
Figure 4: Comparison of the transition of KL divergence between agents with different algo- rithms. Each heatmap shows the KL divergence between the leader and follower policies during training. Row i, column j indicates the forward KL from agent é to agent j. The white circle marks the agent closest from each follower, excluding itself. SAPG often shows misaligned followers, while our method keeps them well-distributed around the leader.
limitation of our method is still rely on fixed number of policies and environments per policy. However, the effective exploration range can vary with the task and training stage. Developing algorithms that automatically adjust these parameters would be an interesting future direction, unlocking the potential of massively parallel environments, especially for tasks with high-dimensional action spaces and demanding exploration requirements.
ACKNOWLEDGEMENT
This work was partially supported by JST Moonshot R&D Grant Number JPMJPS2011, CREST Grant Number JPMJCR2015, Basic Research Grant (Super AI) of Institute for AI and Beyond of the University of Tokyo, and the HIW program of the University of Tokyo. N.S. was supported by Hayashi Rheology Memorial Utsuuoomia,,s Japan. T.O. was supported by JSPS KAKENHI Grant Number JP25K03176.
REPRODUCIBILITY STATEMENT
To facilitate reproducibility, we provide the source code of our proposed method, CPO, at the fol- lowing repository https::////. // hig.h............ com/ NNaoii (eoxx044ppppes cpo code The accompanying README highlights the files that contain the key functions used our implementation. Details of the experimental environments, well as the hyperparameters used in all experiments are listed in Appendix[A.o A.6
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum posteriori policy optimisation. arXiv preprint arXiv: (886600620) 2018.
10

---

Published as a conference paper at ICLR 2026
Gavriel State Ankur Handa Viktor Makoviychuk Aleksei Petrenko, Arthur Allshire. Dexpbt: Scaling up dexterous manipulation for hand-arm systems with population based training. In RSS, 2023.
OpenAl:
Marcin Andrychowicz. Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in nn- manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXXi:18100 12894, 2018.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference on machine learning, pp. 1407-1416. PMLR, 2018.
Lasse Espeholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, and Marcin Michalski. Seed rl: Scalable and efficient deep-rl with accelerated central inference. arXiv preprint arXiv:/910.066 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning diverse skills without a reward function. 2018.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme geleaninn: Maxent
reinforcement learning without entropy. 2023. URL
02328
Genesis Authors. Genesis: A universal and generative physics engine for robotics and beyond, December 2024. URL /tttpp:://gitubb... htw.:/hhub..con/nnnees---------e Aww.ss--ennoddeddd/eeeeeeeee
Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, et al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 5977-5984. IEEE, 2023.
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. 03 2018. doi: 10.48550/arXiv.1800 1wt.:/r0xxxi.1000033.333
Chao Li, Chen GONG, Qiang He, and Xinwen Hou. Keep various trajectories: Pro- moting exploration of ensemble policies in continuous control.
A. Oh, T. Nau-
mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.). Advances in Neu- ral Information Processing Systems, volume 36, pp. 5223-5235. Curran Associates, Inc., 2023a. URL https::
neurips cc/paper http://pprrrffiseppppeeeeeeeee ffles//apee//2233
Zechu Li, Tao Chen, Zhang- Wei Hong, Anurag Ajay, and Pulkit Agrawal. Parallel q-learning: Scaling off off-policy reinforcement learning under massively parallel simulation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Leurning Research, pp. 19440-19459, PMLR, 23-29 Jul 2023b. URL w.w.essss...pp./.e.mm.......
ml www.spess/eeo wttpsss//v222/110 ott.eeaaiina.rrrr.prssrs////////
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu-based physics simulation for robot learning. 2021.
Luca Martino, Víctor Elvira, and Francisco Louzada. Effective sample size for importance sampling based on discrepancy measures. Signal Processing, 131:386-401, 2017. ISSN 0165-1684. doi: https://o..ogg/0........gggggooo
URL /ttp http://wwwww.ccceeeieree...c com/

---

Published as a conference paper at ICLR 2026
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09:39, 2020.
Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems, 33:1 33:18050-18062, 2020.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driver exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787. PMLR, 2017.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pp. 91-100. PMLR, 2022.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning. pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning. The Twelfth International Conference on Learning Representations.
Jayesh Singla, Ananye Agarwal, and Deepak Pathak. Sapg: Split and aggregate policy gradients, In Proceedings of the 41st International Conference on Machine Learning (ICML 2024), Proceedings of Machine Learning Research, Vienna, Austria, July 2024. PMLR.
Shuang Wu, Jian Yao, Haobo Fu, Ye Tian, Chao Qian, Yaodong Yang, Qiang Fu, and Yang Wei. Quality-similar diversity via population based reinforcement learning. In The eleventh international conference on learning representations, 2022.
Zhengpeng Xie, Qiang Zhang, Fan Yang, Marco Hutter, and Renjing Xu. Simple policy optimization. In Forty-second International Conference Machine Learning. 2025.
Jian Yao, Weiming Liu, Haobo Fu, Yaodong Yang, Stephen McAleer, Qiang Fu, and Wei Yang. Policy space diversity for transsaaniiive games. Advances Neural Information Processing Systems, 36: 67771-67793, 2023.
Ziwen Zhuang, Zipeng Fu, Jianren Wang, Christopher Atkeson, Soeren Schwertfeger, Chelsea Finn, and Hang Zhao. Robot parkour learning. arXiv preprint urXiv:2309.05665 2023.
A APPENDIX
A.1 PROOFS OF PROPOSITIONS
In this section, we provide the proofs of Proposition I and Proposition 2 stated in section 4
A.1.1 PROOF OF PROPOSITION
Proof. Let NL.0a denote the number of leader (on-policy) samples and Nz. the number of follower (off-policy) samples. Assuming reachability, i.c., the support the target policy is contained that of the behavior policy, the IS ratio has unit mean: E(r] 1. Then, ESS for the leader update can be
12

---

Published as a conference paper at ICLR 2026
expressed a function of Vars aanTan (f)] as follows:
Ni Ni.onn + NN,oo
Wi
ESS
(13)
Σ Nenoo No
NL,on NL,onBs,amm Bag on (6) N N Bs of saaay Ir (0))
(14)
(Vars, [T (0)] + 1) + (Var |rr, (0) + 1)
1N + No
(15)
Var',a-Ti OL (8) Var's [L +2
Here, the variance of IS ratio for off-policy samples is lower bounded by the expected absolute deviation of from 1 as:
Eggant [II TL,o -
Ee [11 oo (0)))
(16)
Be |L, (0)2) - 21 2l |L (0) + 1,
(17)
VaraantFa off (0)) + (Ba,a-TFad [r (0)) 1)2,
(18)
Var of (f)).
(19)
A.1.2 PROOF OF PROPOSITION
Proof. Formally, the policy gradient 9ac and gradient estimated by PPO Jypo for behavior policy To.old (als) and the target policy no (a(a can expressed as:
= Bn.ulr(0) V log (o( Vppop
(20)
Oppo
(21)
where r(0) Tocala) be written as follows:
Then, the gradient estimation bias introduced by PPO clipping operator can
Bias- = Jac Ippo:
E Eg au V log (a))
(22)
Here, 11 clipped denotes the indicator function that takes the value 1 if the PPO objective is in the clipped regime, and otherwise. Therefore, the squared L norm of this bias can be bounded using Jensen's 's inequality:
Va log no (als) a)) Lelippea] I|=
Bias
E. and
log m (a|s)]] r(0)² A(s, a)² Ielipped]
F Ba da
V log (a)s)] r(0)2
2 1]--((0)
< E. ad
Vg log (a)s))) (11-r(0) + 1)2 A(s, a) 11-r(0))>+] (23)
where 1 [1-r(()] denotes the indicator function that takes the value 1 if -(()) > e, and 0 otherwise. Thus, the upper bound of the bias norm depends directly on |1 - r()), which increases as the IS ratio deviates from 1, leading training instability.
The same reasoning applies to the leader's off-policy updates using follower samples. In our framework, oo (a ss Fau (als) and oo a s) n (a]s). Therefore, the upper bound on the bias norm of the leader's gradient estimate from follower's 's samples depends on ]1 (0)
13

---

Published as a conference paper at ICLR 2026
A.2 DERIVATION OF FOLLOWER POLICY UPDATE UNDER KL CONSTRAINT
This section presents the derivation of the follower policy objective in Eq. 10 unddr the proposed KL constraint. The constrained optimization problem shown in Eq. 9 has closed-form solution, which can be obtained using the method of Lagrange multipliers, as follows:
zTh (a)a)) exp (1 (A)
T (a|s)
(24)
(F da and À is the Lagrange multiplier associated with the KL. constraint, which also serves temperature parameter controlling the strength of attraction between the leader and follower policies.
where Z = (I(as)) exp
Since the closed-form solution is expressed in non-parametric form, we approximate it using a neural network policy TF. (als). To this end, we formulate the problem of approximating the non-parametric solution with parametric model the minimization of both the forward and reverse KL divergences between them. The minimization of the forward KL divergence can be expressed as follows:
arg min (KL
म,. (als) (olss)oog log TF. (a|s)
arg min
da,
8
TR. (a(s)
arg min
/ (Tl((a)s) exp
- A3
(8) a)
(als)da, (a))),
log TogTr, T
0
1 A (3.
arg min Es
log (a 8) exp
(25)
Here, the objective function is computed as expectation with respect to the leader's off-policy samples. In contrast, the minimization of the reverse KI. divergence can be written as follows:
arg min DKI(TF,, (+1s) FT (-1s))
0
FF (als) Tp. (a|s) da,
arg min
TP. (als) log
= arg min / (als)
1 Ast(s, da, Xy
log lols)(als) log Log (als)
)
TF (F log
T TF. (a(s)))
= arg min Ee aarrF
(26)
FF (als)
In this case, the objective function is computed as the expectation with respect to the follower's on-policy samples. Here, We use separate temperature parameters 'y and X, for the forward and reverse KL terms, respectively, to ensure computational stability. By minimizing both the forward and reverse KL divergences instead of one-sided KL divergence, we can effectively utilize samples collected by both the leader and the follower.
For simplicity, we set X 0 to perform regularization solely through the forward KL term, and clipping applied for stable update. Thus, Eq. 26 reduces to LSAPG, P. (0) in Eq 3 Consequently, the follower policy's 's updated objective LCPO. (8) and Lcro, (0) is obtained as follows:
LCPO,F., (0, -Ea Ex
log TF (als) (als) exp
(1 ( (s.,a)
A A
(27)
Fc.o,FF,, (0) =T (0).
A.3 PSEUDOCODE AND COMPUTATIONAL COMPLEXITY OF THE PROPOSED METHOD
In this section, we provide the pseudocode the proposed method and discuss the computational overhead introduced by the KI. constraint and the adversarial reward. Algorithm T illustrates the
14

---

Published as a conference paper at ICLR 2026
overall procedure of CPO. For the KL constraint, the computational difference from SAPG lies in computing the follower loss L (0, Xy), where the KL divergence constraint must be evaluated once for each of the five followers. Consequently, the number of luto-diiferentiable forward backward passes involves roughly 12 components in CPO versus 7 in SAPG, 7 components in SAPG (five follower on-policy updates, one leader on-policy update, and one leader off-policy update) plus five additional components in CPO for follower's 's update from leader's samples, When the adversarial reward is used, an additional six components are introduced train the discriminator.
Nevertheless, since data collection in SAPG typically requires about twice as much time as the training updates when using N 24, 576 environments, the overall wall-clock increase in training time for CPO is modest, amounting to only about 24% with the KL constraint alone and 52% when including the adversarial reward. Considering that the proposed method reaches the final performance of S SAPG with less than half the number of environment steps, this overhead is acceptable.
Algorithm 1: Coupled Policy Optimization (CPO)
Input:
Number of environments N, number of agents M, KL coefficient Xy, adversarial weight
Andv
Output: Updated policy parameters 0, value parameters v,, discriminator parameters & Initialize shared policy parameters 0; policy embeddings Initialize value network Vi
Initialize discriminator Dę
Initialize N environments and assign to M agents leader M-1 followers) for each training iteration do
Data collection
Collect trajectories D, from all agents in parallel
Compute discriminator loss (E) using (s, o. y) E
Compute adversarial reward p Aadv log (e(v)st,a,)
// Advantage estimation
For each agent i, compute advantages A and returns Ri using Ves Sample random follower index j E 1-1) Recompute leader's 's value/advantage on follower j's data Recompute each follower's value/advantage on leader's data
// -- Policy loss aggregation
La +-
Lu x- Lg + Ln (0);
// Leader on-policy loss // Leader off-policy loss // Follower en-policy losses // Follower KL Ki-regularized losses // Entropy regularization
L x- Lr + LL
L xt Lr Σ (0):
Lu + Lr +
(0,
L t Lo L (0):
/ / -- Value loss
LV = E, V (st)s)) 12
1/ - Parameter updates 1x VoLn
Update policy 0
Update value network v t vb - V Lo
Update discriminator & 5 dn VLLD
15

---

Published as a conference paper at ICLR 2026
A.4 ABLATION STUDY ON ADVERSARIAL REWARD AND KL CONSTRAINT
In this section, we conducted an ablation study to analyze the contributions of the two key components of our method: the KL divergence constraint and the adversarial reward. We trained on two tasks, Shadow Hand and Allegro Hand, using the full CPO algorithm, as well as two ablated variants for analysis. The first variant, CPO (w/o AdR), disables the adversarial reward by setting its scaling factor to zero (Nadv = 0). The second variant, CPO (w/o KLC), removes the KL divergence constraint by setting the coefficient for the solution of the forward KL. minimization problem Eq. [1] to zero (33 0). The resulting learning curves are shown in Fig S while the discriminator losses are
plotted in Fig Additionally, the transitions of inter-policy KL divergence are visualized as color
maps in Fig.
Shadow Hand
Allegro Hand
Episode Rewards
12500
Episode Rewards
15000
10000
CRD (ouns
CPD (wi AdR)
5000
5000
CPG (wi (UC)
2500
8.00
0.5
0.50
0.75
125 1.50 Env Steps
1,00
1755
2:00
8.00 0.25 0.50
0.75 1.00
1.25 150 1.75 2,00 Env Steps
"X10""
Figure 5: Effects of KL constraint and adversarial reward on performance. Learning curves on ShadowHand and AllegroHand tasks for three variants: full CPO (red), CPO without adversarial reward (blue), and CPO without KI. constraint (green).
Shadow Hand
Allegro Hand
Discriminaton Loss
3,791
Discriminator
CPO (ours) cro (wis Addtl
1.790
CPO (wiz RLC)
1.789
0.00 0.25 0.50
0.5 1.00
125 150 Env Steps
1.75
2.00
0.00
0.25
0.50
0.75
1.00 1255 1.50 Env Steps
1.75
2.00
Figure 6: Discriminator loss under different settings. Discriminator loss during training on the ShadowHand and AllegroHand tasks for three variants: full CPO (red), CPO without adversarial reward (blue), and CPO without KL constraint (green).
As shown in Fig(51 removing the KL constraint (CPO (wo/KLC)) leads degradation in training performance. This suggests that, without proper regulation of policy distances, the followers explore in directions that deviate from the leader, reducing sample efficiency and training stability. This observation is further supported by the inter-policy KL divergence maps in Fig 7 where follower policies under CPO (wo/KLC) are visibly misaligned and drift far from the leader policy.
In contrast, removing the adversarial reward (CPO (w/o AdR)) results in only a marginal difference in training performance compared the full CPO, although tends to reduce the variance introduced by the adversarial reward across random seeds. As shown in Fig 6 the discriminator loss converges the upper bound of random classification (In 1.792). indicating difficulty in distinguishing the policies regardless of the adversarial reward. In preliminary experiments, increasing the scaling factor of the adversarial reward Aadv without performance tuning made the discriminator easily distinguish between policies. In the current experiment, however, tuned Xadv for optimal performance. As shown in Fig. 5 and Fig. 7 this results in follower policies remaining near the leader, suggesting that such alignment promotes stable and efficient learning.
Interestingly, Fig 7 shows that even without the adversarial reward, each follower's closest policy, in terms of KL divergence, is consistently the leader. This implies that the intended role of the adversarial reward, preventing overconcentration of followers, was already achieved through the KI. constraint and entropy regularization alone. The performance improvement observed with the
16

---

Published as a conference paper at ICLR 2026
Shadow Hand
Allegro Hand
CPO
CPO (wio AdR)
CPO (w/o KLC)
CPO
CPO (w/o AdR)
CPO (w/o KLC)
12
50 Earv
12
3/1
12
15
1
Eav Steps
12
F2
S0
15
12
15
50 Lisv Step
1M
Figure 7: Comparison of the transition KL divergence between agents with different settings. Each heatmap shows the KL divergence between the leader and follower policies during training. Row i, column j indicates the forward KL from agent i to agent j. The white circle marks the agent closest from each follower, excluding itself.
adversarial reward may stem from the uniform penalty it imposes, which encourages optimistic behaviors in the policies due to the relatively high estimated value of unexplored states. The actual impact this regularization appears to vary depending on the task.
A.5 TRANSITION OF INTER-POLICY KL. DIVERGENCE AT HIGHER TIME- RESO-UISONN
Visualizations of the transition of inter-policy KI, divergence across environment steps during training available on project page:
wttp:h/ww..ooppppooeeeaea cpo/
A.6
TRAINING ENVIRONMENTS AND HYPERPARAMETERS
This section provides details on the experimental environments, task description and training hyper- parameters.
A.6.1
EXPERIMENTAL. ENVIRONMENTS
We conduct our experiments using an internal GPU cluster and a large-scale academic computing facility equipped with NVIDIA A100 GPUs Due to differences network environments and CPU configurations, it is difficult to make a fair comparison of training time across tasks and algorithms. However, each condition is trained for approximately one to four days train through 20G environment steps.
A.6.2 TASK DESCRIPTION
Dexterous Manipulation Tasks: For relatively simple dexterous manipulation tasks, adopted in- hn-h reorientation with two types of multi- foltt--fingeee hands: ShadowHand (24 DoF) (Andrychowicz et al. 2020) and AllegroHand (16 DoF). The observation space consists of joint positions and velocities, as well as object orientation and angular velocity. We used an MLP-based policy network for these tasks and set the horizon length to 8.
17

---

Published as a conference paper at ICLR 2026
For more complex dexterous manipulation tasks, we adopted the Regrasping, Reorientation, and Throw tasks in the Allegro- Kuka environment (Aleksei Petrenko 2023). In these tasks, an Allegro Hand (16 DoF) is mounted the end of Kuka Arm (7 DoF). To further evaluate multi-arm dexterity, we also included the Two-Arms Reorientation task, where two Allegro-Kuka systems simultaneously manipulate single object in coordinated manner. For these hand-arm manipulation tasks, employed policy network with single-layer LSTM and set the horizon length to 16.
As non-dexterous manipulation tasks, we adopted two benchmarks: FrankaCubePusee and FrankaCube (8 DoF). For these tasks, we used an MLP. based policy network and set the horizon length to 8.
Gripper-based Manipulation Tasks:
Locomotion Tasks: We adopted two locomotion benchmarks flat ground: Humanoid (21 DoF) and Anymul (12 DoF). Although they involve high-dimensional control, their contact dynamics are relatively simpler compared to dexterous manipulation tasks, making them easier benchmarks in this context. For these tasks, we used a policy network with a single-layer LSTT and set the horizon length 16.
A.6.3 TRAINING HYPERPARAMETERS
For hand-only dexterous manipulation tasks, specifically Shad- owHand and AlnegroHand, we use an MLP-based Gaussian policy with an ELU activation applied after each layer. The discriminator for the adversarial reward is also implemented as MLP with ELU activations, consisting of four hidden layers with sizes [1024, 1024, 512, 512], and is trained using a fixed learning rate equal the initial value used for the policy. The hyperparameter settings for each task are summarized in T
Dexterous Manipulation Tasks:
For hand-arm tasks, specifically AllegroKuka Regrasping, Reorientation, Throw and Two-Arms Reorientation, we use a Gaussian policy that consists of an LSTM layer followed by an MLP with ELU activations applied after each layer. The discriminator for the adversarial reward is also implemented as an MLP with ELU activations, consisting of four hidden layers with sizes [1024, 1024, 512, 512], and trained using a fixed learning rate equal to the initial value used for the policy. The hyperparameter settings for each task are summarized in Table
Gripper-based Manipulation Tasks: For gripper-based manipulation tasks, specifically PrankaCubePushh and FrankaCubestack, we use Gaussian policy that consists of an LSTM layer followed by an MLP with ELU activations applied after each layer. The discriminator for the adver- sarial reward is also implemented as MLP with ELU activations, consisting of three hidden layers with sizes [256, 128, 64], and is trained using fixed learning rate equal to the initial value used for the policy. The hyperparameter settings for each task are summarized in Table
Locomotion Tasks: For locomotion tasks, specifically Humanoid and Anymal, we use MLP- based Gaussian policy with an ELU activation applied after each layer. The discriminator for the adversarial reward is also implemented as MLP with ELU activations. consisting of three hidden layers with sizes [768, 512, 256], and is trained using a fixed learning rate equal to the initial value used for the policy. The hyperparameter settings for each task summarized in Table
18

---

Published as a conference paper at ICLR 2026
Table 3: Training hy perpaaaraetee for Shadow Hand and Allegro Hand. The upper section lists hyperparameters shared by SAPG and CPO, while the lower section lists those specific to CPO.
Hyperparameter
Shadow Hand
Allegro Hand
Common Hyperparameters (SAPG and CPO)
Discount factor,
0.99
0.99
GAE smoothing factor,
0.95
0.95
MLP hidden layers
[512, 512, 256, 128]
[512, 256, 128]
Learning rate
5e-4
Se-4
KL threshold for LR update
0.016
0.016
Grad norm
1.0
1.0
Entropy coefficient
0.005
0
Clipping factor,
0.2
0.2
Mini-batch size
4 num num_env. envs
4 x num envs
Critic coefficient, X
4.0
4.0 8 0.0001
Horizon length
8
Bounds loss coefficient Mini epochs
0.0001 5
CPO-Specific Hyperparameters
9 in Eq. Forward KT. constraint temperature, A
0.001
0.0005
0.2
0.1
Reverse KI. constraint temperature, X 0
0
Adversarial reward scaling factor, Aadv
0.01
0.001
Table 4: Training hyperparameters for hand-arm dexterous manipulation tasks: AllegroKuka Regrasping, Reorientation, Throw and Two-Arms Reorientation. The upper section lists hyper- parameters shared by SAPG and CPO, while the lower section lists those specific to CPO.
Reorientation / Two-Arms Reorientation
Hyperparameter
Regrasping
Throw
Common Hyperparameters (SAPG and CPO)
Discount factor,
0.99
0.99
0.99
GAE smoothing factor, т
0.95
0.95
0.95
LSTM hidden units
768
768
768
MLP hidden layers
[768, 512, 256]
[768, 512, 256)
[768, 512, 256]
Learning rate
lc-4
lc-4
lc-4
KL threshold for LR update
0.016
0.016
0.016
Grad norm
1.0
1.0
1.0
Entropy coefficient
0
0.005
0
Clipping factor, e Mini-batch size
0.1
0.1
0.1
4 x num_envs
4 x num_envs
4 x num_envs
Critic coefficient, X'
4.0
4.0
4.0
Horizon length
16
16
16
LSTM Sequence length
16
16
16
Bounds loss coefficient Mini epochs
0.0001
0.0001 2
0.0001
2
CPO-Specific Hyperparameters
( in Eq. 11
0.001
0.001
0.001
Forward KL constraint temperature, A Reverse KL constraint temperature, X, Adversarial reward scaling factor, Andv
0.2
0.2
0.1
0
0
0
0
19

---

Published as a conference paper at ICLR 2026
Table 5: Training hyperparameters for gripper-based manipulation tasks: FrankaCube and TrankaCubeStack. The upper section lists hyperparameters shared by SAPG and CPO, while the lower section lists those specific CPO.
Hyperparameter
FrankaCubePush
FrankaCubeStack
Common Hyperparameters (SAPG and CPO)
Discount factor,
0.99
0.99
GAE smoothing factor,
0.95
0.95
LSTM hidden units
256
256
MLP hidden layers
[256, 128, 64]
[256, 128, 64]
Learning rate
5e-4
5e-4
KL threshold for LR update
0.008
0.008
Grad norm
1.0
1.0
Entropy coefficient
0.005
0.005
Clipping factor, c Mini-batch size Critic coefficient, X'
0.2
0.2
num_envs
x num envs
4.0
4.0
Horizon length LSTM Sequence length Bounds loss coefficient Mini epochs
16
16
16
16
0.0001
0.0001
8
8
CPO-Specific Hyperparameters
8 in Eq. Forward KL constraint temperature, Xy
0.0005
0.0005
0.1
0.1
Reverse KL constraint temperature, A,
0
0
Adversarial reward scaling factor, Andv
0
0
Table 6: Training hyperparameters for locomotion tasks: Humanoid and Anymal. The upper section lists hyperparameters shared by SAPG and CPO, while the lower section lists those specific to CPO.
Hyperparameter
Humanoid
Anymal
Common Hyperparameters (SAPG and CPO)
Discount factor,
0.99
0.99
GAE smoothing factor, т
0.95
0.95
MLP hidden layers
[768, 512, 256]
[768, 512, 256]
Learning rate
5e-4
3e-4
KI. threshold for LR update
0.008
0.008
Grad norm
1.0
1.0
Entropy coefficient
0,002
0.002
Clipping factor, e Mini-batch size
0.2
0.2
4 x envs
4 x num envs
Critic coefficient, X'
4.0
4.0
Horizon length
8
8
Bounds loss coefficient Mini epochs
0.0001 5
0.0001
CPO-Specific Hyperparameters
3 in Eq.
0.001
0.001
Forward KL constraint temperature, Ay
0.2
0.2
Reverse KL constraint temperature, X,-
0
0
Adversarial reward scaling factor, Xadv
0
20

---

Published as a conference paper at ICLR 2026
A.7 ENTROPY REGULARIZATION ABLATION ON SAPG
In this section, we investigate the relationship between follower-policy misalignment in SAPG and the entropy regularization term. To this end, conducted ablation study on the entropy coefficient across three tasks: Shadow Hand, AllegroKuka Regrasping, and AllegroKuka Reorientation. The learning curves are shown in Fig. and the inter-policy KL divergence during training visualized in Fig.
From Fig. consistent with observations in the SAPG paper (Singla et 2024), the effect of entropy regularization on training performance varies significantly across tasks. More importantly, Fig. 9
further shows that introducing entropy regularization SAPG consistently increases anter-policy KL
divergence and often leads to severe misalignment.
These results suggest that, while entropy regularization indeed promotes exploration and yields diverse data, it also induces policy misalignment that destabilizes the leader's learning, supporting our main claim. In contrast, CPO suppresses this disadvantage while still promoting exploration within a KL-bounded region, enabling stable leader updates and achieving superior performance.
Shadowiland
AliegroKuka Regrasping
AllegroKuka Reorientation
Egisada Rewards
Episode Successss
Episode successes
10202
2582
16
2300
075 1
Steps
155
+10
8.0
025
6:50
0.5
1.0
Steps
0.25
1,00 125 150 175 Steps
SAPG (0.005)
SAPG
Figure 8: Comparison of SAPG with and without entropy regularization. The values in parenthe- ses indicates the entropy coefficients. The effect of entropy regularization on learning performance varies across tasks.
21

---

Published as a conference paper at ICLR 2026
Shadow Hand
Allegro Kuka Regrasping
Allegro Kuka Reorientation
SAPG (A
SAPG (A 0.005)
SAPG (A
SAPG 0.005)
SAPG (A
O)
SAPG (A 0.005)
50 Elav Stegss
12
1
10G Eisv Steps
F
3
F
3
ESIV Steps
2
150
C 3
Figure 9: Comparison of the transition of KL divergence in SAPG with and without entropy regularization. Each heatmap shows the KL divergence between the leader and follower policies during training. Row i, column j indicates the forward KL from agent to agent j. The white circle marks the agent closest from each follower, excluding itself. It is demonstrated that entropy regularization causes follower's misalignment from the leader policy.
22


Reviews:
[{"review_id": "h6MnJZaQMB", "reviewer": "Reviewer_hCse", "summary": "This paper tackles the challenge of scaling reinforcement learning (RL) to large numbers of parallel environments, where a single policy’s limited exploration capacity can hinder performance. The authors focus on ensemble-based policy gradient methods that use multiple policies to promote exploration but note that excessive diversity can harm learning stability and efficiency. They present Coupled Policy Optimization (CPO), a method that constrains inter-policy divergence via KL regularization to balance exploration and coordination among ensemble members. Theoretical analysis establishes how controlled diversity improves learning efficiency, and empirical evaluations on dexterous manipulation tasks show that CPO outperforms prior methods such as SAPG, PBT, and PPO in both sample efficiency and final performance. Additional analysis demonstrates that follower policies self-organize around a leader, yielding structured and effective exploration. Overall, the work highlights the importance of regulated diversity for stable and efficient learning in ensemble policy gradient frameworks.", "questions": "* I would say that although DIAYN uses a discriminator, their work is primarily interested in gathering skills and maximizing entropy, would it be reasonable to introduce to the reader as (ICM)[2] which explicitly relies on prediction error? I don't consider this an issue, but there are other methods that leverage prediction error that might make a better fit. \n\n[2] Pathak, D., Agrawal, P., Efros, A.A., Darrell, T.: Curiosity-driven exploration by\nself-supervised prediction. In: International Conference on Machine Learning, pp.\n2778–2787 (2017). PMLR", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* It appears to be a novel and difficult task to introduce exploration incentives to the follower agents at risk of destabilizing the already off-policy training, however this paper appears to utilize the KL divergence and the discriminator in a way that promotes some exploration for the follower agents without training collapsing.\n * The paper is well written, and utilizes the background and method section well. The results are well thought out.\n* Although the main algorithmic contribution appears to be simple, the KL divergence, the addition is well justified and explored thoroughly in the paper.", "weaknesses": "* I would have liked to know why a discriminator was chosen specifically, in comparison to other exploration based algorithms, in particular there are other methods that do not require the additional external training or usage of a functional approximation [1]. \n\n* Going further on the second point, although the KL divergence effects were explained, section 5.2 appears to be the only discussion on the usage of an exploration algorithm, and it appears that there is little explanation on how the adversarial signal improves the performance of the algorithm. If this component of the paper was discussed or there was more time spent analyzing this more somewhere in the paper I would be more confident staying at an accept. \n\n[1] Susan Amin, Maziar Gomrokchi, Harsh Satĳa, Herke van Hoof, & Doina Precup. (2021). A Survey of Exploration Methods in Reinforcement Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D0kfa2NMEg", "reviewer": "Reviewer_FSNQ", "summary": "This paper addresses the problem of excessive policy diversity in ensemble policy gradient methods  by improving the policy optimization method for followers. Specifically, it introduces KL constraints during the follower updates to regulate the distance to the leader, and incorporates an adversarial reward to prevent policy overconcentration . The proposed method (CPO) achieves structured and efficient exploration. Experiments demonstrate that this method can effectively improve sample efficiency and final performance.", "questions": "1. I wonder whether \"misaligned significantly\" phenomenon consistently occur in all \"complex tasks\". Could you provide more analysis on what types or characteristics of environments are prone to causing this \"misaligned significantly\" state? Can this be primarily attributed to the environment's exploration complexity? Could I regard the KL constraint (CPO's core mechanism) essentially act as a mechanism to reduce exploration and increase exploitation;\n\n2. Are the marginal benefits of improving the IS and ESS metrics on the final performance limited? I observed that the ESS was optimized by over 40-fold (e.g., from 2.23% to 94.1% in ShadowHand ), yet the final sample efficiency (as shown in the reward curves ) only showed a limited improvement (approx. 2-2.5x).", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper's insight is presented very clearly, and the logic is rigorous; we can easily follow the author's train of thought and logic to understand the method. The paper's writing quality is high.\n\n2. The theoretical derivations are thorough, and the experimental validation is comprehensive. The visualization of KL divergence changes in Figure 4 is very valuable.\n\n3. The method achieves a significant breakthrough on a high-difficulty task (Two-Arms Reorientation).", "weaknesses": "1. The model's generalizability appears limited. It is only effective in specific environments, such as AllegroHand, where follower policies are prone to significant divergence (resulting in high variance). In contrast, on other tasks like Regrasping, the performance improvement is not as pronounced .\n\n2. The training cost is somewhat high. The paper's CPO method requires more backpropagation components (roughly 12 vs. 7 for SAPG) and more wall-clock training time per iteration (approximately 25% more) .\n\n3. Although the design of the Adversarial Reward is interesting—leveraging the idea from DIAYN to classify policies based on state-action pairs —the ablation study shows its impact on the results is not significant. The necessity of this module is therefore questionable .", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ljZvrJhOq4", "reviewer": "Reviewer_nLNK", "summary": "In this work, the authors theoretically analyze the impact of inter-policy diversity on learning efficiency in policy ensembles, and propose Coupled Policy Optimization (CPO), a method that regulates diversity via KL constraints between policies. The proposed CPO enables effective exploration and outperforms strong baselines (including SAPG, PBT, and PPO) across multiple dexterous manipulation tasks, showing advantages in both sample efficiency and final performance. Additionally, the experimental results are robust, and the paper is fluently written.", "questions": "1. Are there any specific guidelines or recommendations for the selection of the lambda hyperparameter?\n\n2. The paper adopts SAPG as its baseline method. Could the authors explain why SAPG outperforms Population-Based Training (at least in their experimental settings)?\n\n3. Why was KL divergence chosen as the policy distance metric? Would using reverse KL divergence alter the conclusions of this work?\n\n4. Why is the score of SAPG very low in Two-Arms Reorientation in Table 1, but the proposed method, which is based on SAPG, achieves the highest score?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-structured and clearly written, ensuring good readability.\n\n2. The authors rethink the existing SAPG method by providing theoretical analysis and key insights, which effectively motivate the design of the proposed CPO.\n\n3. The experimental results are robust and supported by sufficient evidence, and the overall writing flow is smooth.", "weaknesses": "1. More concrete examples could be added to illustrate and validate the key theoretical insights, which would strengthen the persuasiveness of the work.\n\n2. The design of the proposed CPO is relatively straightforward, as it only uses KL divergence to constrain the distance between follower and leader policies, lacking further optimization or innovative adjustments.\n\n- 2.1 The selection of the lambda hyperparameter in practice is somewhat heuristic, with no clear justification provided for its choice.\n\n- 2.2 KL divergence is a relatively trivial and commonly used distance metric. In the field of policy diversity research, numerous works have proposed alternative diversity metrics [1-3]. A more in-depth discussion of these existing metrics, along with the rationale for selecting KL divergence in this work, would improve the justification for the method’s design.\n\n[1] Effective diversity in population based reinforcement learning. \n\n[2] Policy space diversity for non-transitive games.\n\n[3] Quality-similar diversity via population based reinforcement learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LUjBqk0CFm", "reviewer": "Reviewer_ZcZ8", "summary": "This paper shows that in large-scale reinforcement learning, blindly pursuing policy diversity across multiple policies reduces sample efficiency and training stability. To address this, it proposes Coupled Policy Optimization (CPO): within a leader–follower framework it regulates diversity via a KL constraint and an adversarial intrinsic reward, and it uses importance sampling to efficiently absorb data from followers. Experiments show that CPO achieves higher sample efficiency and more stable, superior final performance than existing baselines on a range of dexterous manipulation tasks.", "questions": "See Weaknesses above.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposes CPO, which within a leader-follower framework regulates the exploration radius and coverage via a KL constraint to the leader policy coupled with an adversarial intrinsic reward, while retaining efficient use of off-policy data.\n2. This paper uses upper-bound analysis to connect the leader-follower KL with IS ratio deviation, ESS, and gradient bias, and, drawing on Pinsker-type inequalities and PPO clipping analysis, offers a quantitative justification for why distance control is needed.\n3. On dexterous manipulation tasks under ultra-large parallel settings, CPO shows higher sample efficiency and more stable, superior final performance, supported by ablations and visualizations.", "weaknesses": "1. CPO essentially adds a KL constraint to the leader and an adversarial intrinsic reward on top of a leader-follower framework. The novelty is incremental.\n2. The discriminator and KL regularization introduce additional compute overhead and hyperparameters, yet the paper lacks wall-clock comparisons under equal compute and equal interaction budgets, as well as a systematic hyperparameter sensitivity analysis.\n3. The experimental evidence focuses mainly on dexterous manipulation with a single platform/parallelism setting, and the breadth of tasks and difficulty axes is insufficient to support conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)