The Curse of Diversity in Ensemble-Based Exploration
Authors: Zhixuan Lin
Pierluca D'Oro
Evgenii Nikishin
Aaron Courville
Venue: ICLR 2024 poster (2024)
Keywords: deep reinforcement learning
ensemble-based exploration
off-policy learning
representation learning
auxiliary tasks
Primary area: reinforcement learning
TL;DR: We show that the off-policy learning challenge in ensemble-based exploration in deep RL can impair performance, provide extensive analysis, and show representation learning as a promising solution with a novel method CERL.

Abstract: We uncover a surprising phenomenon in deep reinforcement learning: training a diverse ensemble of data-sharing agents -- a well-established exploration strategy -- can significantly impair the performance of the individual ensemble members when compared to standard single-agent training. Through careful analysis, we attribute the degradation in performance to the low proportion of self-generated data in the shared training data for each ensemble member, as well as the inefficiency of the individual ensemble members to learn from such highly off-policy data. We thus name this phenomenon *the curse of diversity*. We find that several intuitive solutions -- such as a larger replay buffer or a smaller ensemble size -- either fail to consistently mitigate the performance loss or undermine the advantages of ensembling. Finally, we demonstrate the potential of representation learning to counteract the curse of diversity with a novel method named Cross-Ensemble Representation Learning (CERL) in both discrete and continuous control domains. Our work offers valuable insights into an unexpected pitfall in ensemble-based exploration and raises important caveats for future applications of similar approaches.

Published as a conference paper at ICLR 2024
THE CURSE OF DIVERSITY IN ENSEMBLE-BASED EXPLORATION
Zhixuan Lin; Pierluca D'Oro, Evgenii Nikishin & Aaron Courville Mila Quebec Al Institute, Université de Montréal
ABSTRACT
We uncover a surprising phenomenon in deep reinforcement learning: training a diverse ensemble of data-sharing agents well-established exploration strategy can significantly impair the performance of the individual ensemble members when compared to standard single-agent training. Through careful analysis, attribute the degradation in performance to the low proportion of self-generated data the shared training data for each ensemble member, as well as the inefficiency of the individual ensemble members to learn from such highly off-policy data. We thus name this phenomenon the curse of diversity. We find that several intuitive solutions - such as larger replay buffer or smaller ensemble size either fail to consistently mitigate the performance loss or undermine the advantages of ensem- bling. Finally, we demonstrate the potential of representation learning to counteract the curse of diversity with a novel method named Cross-Ensemin Representation Learning (CERL) in both discrete and continuous control domains. Our work offers valuable insights into an unexpected pitfall in ensemble-based exploration and raises important caveats for future applications of similar approaches.
INTRODUCTION
Ensemble-based exploration, i.e. training diverse ensemble of data-sharing agents, underlies many successful deep reinforcement learning (deep RL) methods (Osband et al. |l. 2016 2018 Liu et al. 2020: Schmitt al., 2020: Peng et al. 2020 Hong et al.] [020 Januszewski
al. (021).
The potential benefits of diverse ensemble are twofold. At training time, it enables concurrent exploration with multiple distinct policies without the need for additional samples. At test time, the learned policies can be aggregated into a robust ensemble policy, via aggregation methods such as majority voting (Osband et al. 2016) or averaging (Januszewski et al. 2021).
Despite the generally positive perception of ensemble-based exploration, we argue that this approach has potentially negative aspect that has been long overlooked. As shown in Figure [ for each member in data-sharing ensemble, only a small proportion of its training data comes from its own interaction with the environment. The majority its training data is generated by other members of the ensemble, whose policies might be distinct from its own policy. This type of off-policy learning has been shown to be highly challenging in previous work (Ostrovski et al. [202]). We thus hypothesize that similar learning difficulties can also occur in ensemble-based exploration.
We verify our hypothesis in the Arcade Learning Environment (Bellemare et al.) 2012) with the Bootstrapped DQN (Osband et al. (2016) algorithm and the Gym MuloCo benchmark (Towers et al., 2023) with an ensemble SAC (Haarnoja et al. 220 Sa) algorithm. We show that, in many environments, the individual members of a data-sharing ensemble significantly underperform their single-agent counterparts. Moreover, while aggregating the policies of all ensemble members via voting or averaging sometimes compensates for the degradation in individual members' performance, it is not always the case. These results suggest that ensemble-basee exploration has hidden negative effectt that might weaken or even completely eliminate advantages. We perform a series of experiments to confirm the connection between the observed performance degradation and the aff-policy learning challenge posed by a diverse ensemble. We thus name this phenomenon the curse of diversity.
"Correspondence 2x11n c.c.gmmaaaaa com

---

Published as a conference paper at ICLR 2024
Single-Agent Exploration
Ensemble Exploration
agent
agent
agent
generates
data
data
data
data
9gent
agent
agent
agent
learns from
learns from
Same amount of total interaction steps/date
Figure I: Comparison between standard single-agent exploration and ensemble- exploration. In single-agent training, one agent generates and learns from all the data. In ensemble-base exploration with N ensemble members, each agent generates I/N of the data but learns from all the data.
We show that several intuitive solutions such as a larger replay buffer or a smaller ensemble size either fail to consistently mitigate the performance loss undermine the advantages of ensembling. Inspired by previous work's 's finding that network representations play a crucial role in related settings OOsrrvski et al., [202]] Kumar al. (02T). investigate whether representation learning can mitigate the curse of diversity. Specifically, we propose a novel method named
Representation Learning (CERL) in which individual ensemble members learn each other's s value function as auxiliary task. Our results show that CERL mitigates the curse of diversity in both Atari and MuJoCo environments and outperforms the single-agent and ensemble-based baselines when combined with policy aggregation.
We summarize our contributions as follows:
1. We expose the curse of diversity phenomenon in ensemble- -ased exploration: individual members a data-sharing ensemble can vastly underperform their single-agent counterparts.
2. We pinpoint the off-policy learning challenges posed by a diverse ensemble the main cause of the curse of diversity and provide extensive analysis.
3. We show the potential of representation learning to mitigate the curse of diversity with novel method named continuous control domains.
Representation Learning (CERL) in both discrete and
PRELIMINARIES
outline some important specifications that we use throughout this work.
Einemble-basee exploration strategy We focus our discussion on simple ensemble-based
exploration strategy that underlies many previous works (Oshand al., 2016) 2018 Liu al. 22200) Schmitt et al. 2020 Peng al.) 2020: Hong al., 2020; Januszewski et al. 2021). depicted in
Figure
The defining characteristics of this strategy are as follows:
1. Temporally coherent exploration: At training time, within each episode, only the policy of one ensemble member is used for selecting the actions.
2. Relative independence between ensemble members: Each ensemble member has its own policy (may be implicit), value function, and target value function. Most importantly, the regression target Temporal Difference (TD) updates should be computed separately for each ensemble member with their own target network.
3. Off-policy RL algorithms with a shared replay buffer: Different ensemble members share their collected data via central replay buffer (Lin) [992). To allow data sharing, the underlying RL algorithm should be off-polics in nature.
Environments and algorithms We use 55 Atari games from the Arcade Learning Environment and 4 Gym MuJoCo tasks for our analysis. We train for 200M frames for Atari games and 1M steps for MuloCo tasks. For aggregate results over 55 Atari games, we use the interquartile mean (IQM) of numan-normaliin scores (HNS) recommended by Agarwal et al. (202)). In some results where
2

---

Published as a conference paper at ICLR 2024
HNS is less appropriate, exploit another score normalization scheme where Double DQN and random agent have a normalized score and respectively. This will be referred to as Double DQN normalized scores. More experimental details can be found in Appendix B
For our analysis, we use Bootstrapped DQN (Osband et al. 2016) with Double DQN updates (Hasselt et al. 2015) for Atari and ensemble version of the SAC (Haarnjja al. 201 8b) algorithm (referred to as Ensemble SAC in the following) for MuJoCo tasks. Ensemble SAC follows the same recipe as Bootstrapped DQN except with SAC the base algorithm. We provide pseudocode for these algorithms in Appendix A Correspondingly, for the single-agent baselines, we use Double DQN and SAC. For analysis purposes, for continuous control, we use a replay buffer of size 200k by default (as opposed to the usual 1M) since we find the curse of diversity is more evident with smaller replay buffers in these tasks. Also, to avoid the confounding factor of representation learning, we do not share the networks across the ensemble in Bootstrapped DQN by default. These factors will be analyzed in Section 3.3 Following Osband et al. (216)'s setup for Atari, we do not use data bootstrapping in our analysis. Throughout this work, we use L to denote the number of shared layers across the ensemble members and N to denote the ensemble size. Unless otherwise specified, we use L 0 and N 10. For each ensemble algorithm X where X can be either Bootstrapped DQN or Ensemble SAC we consider two different evaluation methods:
1. X (aggregated) or X (agg.): we aggregate the policies of all ensemble members during testing. For discrete action tasks, we majority voting (Osband et al. (016)) For continuous control, we average the actions of all policies as in Januszewski et al. (2021).
2. X (individual) or X (indiv.): for each evaluation episode, we randomly sample one ensemble member for acting. This interaction protocol exactly the same as the one used during training and aims measure the performance of the individual ensemble members.
We emphasize that these two methods only differ at test-time. We only train X once and then obtain the results of X (agg.) and X (indiv.) using the above interaction protocols during evaluation. Policy aggregation is never used during training. Also, the performance of X (agg.) is often what eventually care about. More implementation details can be found in Appendix C
THE CURSE OF DIVERSITY
The central finding of this paper as follows:
Individual ensemble members in data-sharing ensemble can suffer from severe performance degradation relative to standard single-agent training due to: (1) the low proportion of self- generated data in the shared training data for each ensemble member; and (2) the inefficiency of the individual ensemble members to learn from such highly off-policy data.
We name the performance degradation in individual ensemble members due challenging off-policy learning posed by diverse ensemble the curse of diversity. In the following, we demonstrate this phenomenon, verify its cause, and provide extensive analysis.
3.1 THE NEGATIVE EFFECT OF ENSEMBLE-BASEE EXPLORATION
We show the curse of diversity phenomenon with 55 Atari games and 4 MuloCo tasks in Figure 2 These results show a clear underperformance of the individual ensemble members (c.g.. Bootstrapped DQN (indiv.)) relative to their single agent counterparts (c.g.. Double DQN). Note that even though they are trained on different data distributions and thus are expected to behave differently, the agents in these two cases have access to the same amount of data and have the same network capacity. The underperformance happens the majority of Atari games and out of the 4 MuJoCo tasks, suggesting that this is a universal phenomenon. Surprisingly, simply aggregating the learned policies test-time provides a huge performance boost in many environments, and in many cases fully compensates for the performance loss in the individual policies (e.g.. Walker2d). This partially explains why previous works - which often only report the performance of the aggregated policies (Osband et al. 2016 Chiang al. 2020 Agarwal al. 2020 Meng et al. 2022) fail to notice this phenomenon.
3

---

Published as a conference paper at ICLR 2024
Human Hlmman-ormalinnn Normalized Score
Improverrec over Double DQN in HINS
Beotstraped DON (agg)
1.0
1QM
0.5
Bootstrapped DQN (idiv..)
10
0.0
0
100
200
Frames (x100)
Double DQN
Dane Nock
Bark Sdais
36 Son
(per
Lem
Coryy Pant
Junt
Bootstrapped DQN (indiv:)
Bootstrapped DQN (agg-)
Chayper
You
Noma
Hlumanoid
Walker2d-v4
HaliCheetahh.dd
Episodo Return
4000
5000
4000
10000
2000
2500
2000
5000
0
0.0
0.5
1,0
0.0
0.5
0.0
0.5
1.0
0.5
1.0
Steps
Steps
Stops
le6
Stops
le6
Ensemble SAC (indiv.)
Ensemble (agg)
Figure 2: (top-left) Comparison between Double DQN, Bootstrapped DQN (agg.), and Bootstrapped
DQN (indiv.) in 55 Atari games. Shaded areas show 95% bootstrapped CIs over 5 seeds. (top-right) Per-game performance improvement Bootstrapped DQN (indiv.) and Bootstrapped DQN (agg.) over Double DQN, measured as the difference in HNS. All methods use replay buffer size of 1M. (bottom) Comparison between SAC, Ensemble SAC (indiv.) and Ensemble SAC (agg.) in 4 MuJoCo
tasks with a replay buffer size of 200k. Shaded areas show 95% bootstrapped CIs over 30 seeds. All ensemble methods in this figure use N = 10 and I. = 0.
The significance of these results is threefold. First, it challenges some previous work's claims that the improved performance of approaches such as Bootstrapped DQN in some tasks is mainly due to better exploration. As shown in Figure in most games where Bootstrapped DQN (agg.) performs better than Double DQN, Bootstrapped DQN (indiv.) significantly underperforms Double DQN. This means most benefits of Bootstrapped DQN in these games come from majority voting, which is largely orthogonal exploration. However, this does not imply that better exploration and hence wider state-action space coverage brought by a diverse ensemble is not beneficial, as its effects might be overshadowed by the curse of diversity and thus not visible. Second, raises important caveats for future applications of ensemble-basee exploration, especially in certain scenarios such as hyperparameter sweep with ensembles (Schmitt et al., 2020 Liu et al., (2020) where we mainly care about individual agents' performance. Finally, presents algorithms that mitigate the curse of diversity while preserving the advantages of using an ensemble. To this end, we perform analysis to better understand the cause of the performance degradation.
opportunity for better ensemble
3.2 UNDERSTANDING ENSEMBLE PERFORMANCE DEGRADATION
We hypothesize that the observed performance degradation is due to (1) the low proportion of self- generated data in the shared training data for each ensemble member, and (2) the inefficiency of the individual ensemble members to learn from such highly off-policy data. Inspired by the tandem RL setup Ostrovski et al. (2021), we design a simple "p%-tandem" setup verify this hypothesis. Similar to the original tandem RL setup, we train a pair of active and passive agents sharing the replay buffer and training batches. For each training episode, with probability - p% we use the active agent for acting; otherwise, we use the passive agent for acting. In other words, the active agent generates 1 - p% of the data and the passive agent generates p% of the data. Note that this is different from the "p% eeff-genertted data" experiment in Ostrovski et al. (2021) separate buffers for the two agents. In contrast, in our p%otaandem setup, the two agents share the replay buffer and the training batches, and thus any performance gap between the active and passive
they two
4

---

Published as a conference paper at ICLR 2024
Human Noraalzzd Score (DMM
1.00
0.66. x1--
1.00
0.75
25%
23%
25%
2516
DDON BeatDON
Double
0.50
0.50
25%-tandem
Deable
0.25
0.25
beonerepped DC3 with 4 mambers
Activ Pasive (105))
0.00
50
100
150
200
0.00
0.25 0.50 0.75 1.00
Frames (x10*)
Active (90%) - Passive(10%)
Figure 3: (left) Different algorithms variants of the same ensemble algorithm. using N 4 as an example. Each block represented 25% of the generated data. Data blocks of the same colors generated by identical agents. (middle) Comparison between Double DQN, Bootstrapped DQN (indiv.) with N 10 and L = O, and the active and passive agents in the 10%-tandem setup. All methods use replay buffer of size 1M. Shaded areas show 95% bootstrapped Cls. Results are aggregated over 5 seeds and 55 games. (right) Correlation between (1) the performance gap between the active and passive agents and (2) the performance gap between Double DQN and Bootstrapped
DQN (indiv.) in different games. Each point corresponds to a game. We use Double DQN normalized scores instead of HNS since the scale of the latter can vary lot across games. Eight games where Double DQN's performance is close to random (HNS < 0.05), and one game whose data point lies on the negative half of the y-axis in the plot are omitted since they trivially satisfy our hypothesis.
agents can only be due to the difference in the proportions of the two agents' self-generated data and the inefficiency of the passive agent to learn from the shared data.
To understand why -otttndem may support our hypothesis, is useful to view Double DQN, p%- tandem with two Double DQN agents, and Bootstrapped DQN as variants of the same ensemble algorithm with N members, where each member generates 1/N of the data. Taking N = 4 and p% 25% as an example, we have the following exact equivalence (shown in Figure (left)):
ALGO 1 Double DQN 4 ensemble members, but all of them are identical;
ALGO 2 25%-tandem + 4 ensemble members, but the last 3 of them are identical;
ALGO 3 Bootstrapped DQN + ensemble members, and all of them are different.
Our core reasoning is as follows. If the first member AL.GO (i.e. the passive agent) suffers from severe inefficiency to learn from the shared data (i.e., it significantly underperforms the active agent), it should also suffer from the same learning inefficiency if we replace the other 3 identical members (i.e., the active agent) with 3 different members, which is just ALGO 3/Botstrappeed DQN. Further, the performance gap between the active and passive agents is comparable with or larger than the performance gap between Double DQN and Bootstrapped DQN (indiv.), then the observed learning inefficiency is sufficieni to cause the performance degradation we see in Bootstrapped DQN (indiv.).
Verifying the hypothesis In Figure 3 (middle) show the performance of Double DQN, Boot- strapped DQN (indiv.) with N = 100 and the active and the passive agents in the 10%-tandem setup. As expected, we see that the passive agent significantly underperforms the active agent even though they share the training batches, indicating the inefficiency of the passive agent to learn from the shared data. Also, the performance gap between the active and passive agents is comparable to the performance gap between Double DQN and Bootstrapped DQN (indiv.). A similar analysis for MuJoCo tasks is presented Appendix [.I and shows similar patterns. In Figure ( (right), we show a clear correlation between (1) the performance gap between the active and the passive agents and (2) the performance gap between Double DQN and Bootstrapped DQN (indiv.) in different games. These results offer strong evidence of connection between the off-policy learning challenges in ensemble- based exploration and the observed performance degradation.
Remark on data coverage We comment that another important aspect of having diverse ensemble is wider sate-actionn space coverage in the data. Even though the degree of ooff-policy-neee and the state-action space coverage are often correlated in practice, they are different: state-action space coverage is purely a property of the data distribution, while "off-policy-nee" involves both the data distribution and the policies. Our poo-tandmm experiment disentangles these two aspects by
5

---

Published as a conference paper at ICLR 2024
Epteoda Rotem
1300
Amidar
Keng Moster
Epicode Return
Humsmnod--e
6000
Ant-v4
20000
4000
10000
2500
Epionde Dotumn
Space havaddrs
Chopper Command
Eppieade Ratum Retemn
Walker22d.04
HalrChostab-v4
3000
4000
4000
5000
12000
2000
0.5
0.2
1.0
Replay Butfer Sizo (x10%)
Raplay Bufler Size (x10*)
Replay Butfer Size (x10")
Replay Butflr Size (x110")
Double DQN
Rouistrapped DQQ (indiv)
SAC
Ensemble SAC (indis.)
apped DQN (age.)
(Ag)
Figure 4: (left) The effects of replay buffer size in Atari games. Error bars show 95% bootstrapped Cls over 5 seeds. (right) The effects of replay buffer size in MuJoCo tasks. Error bars show 95% bootstrapped Cls over 30 seeds. We N 10 and I. = for Bootstrapped DQN.
Amidar
Kung
Space Chopper Command
Amidar
King Master
Space Invoders
Chapper Command
Nummalzed Soore
Nommalzad Sare
1.00
1.0
0.50
0.5
20
Linommi Size
20
Linsabble Size
5
20
linsemble Size
10
5
20
Unsemble Size
10
5
AShuned Layers
AStaned Layers
3
AShorod Layers
3
3 ASlaned Lagers
Vocies Enaogy
Voding Entrpy
1.0
20
5
20
10
3
20
10
Easembee Sizee
Enseuble Size
Easemble Stae
Ensemble Size
0Shared Layers
Layers
Layers
AShured Layers
Double DQN
Bnotetapped DQN (indi.)
Double DQN
Boctstapped DON (adiv.)
Beotstagped DQN (agg.)
Rootstrappe- (mpp)
Figure 5: (left) The effects of adjusting the ensemble size. We use L = for Bootstrapped DQN. (right) The effects of varying the number of shared layers. We use N 10 for Bootstrapped DQN. The top rows show Double DQN normalized scores. The bottom rows show the entropy of the normalized vote distributions. Error bars show 95% bootstrapped CIs over 5 seeds. All methods use a replay buffer size 1M.
having the active and passive agents trained on the same data (thus the same coverage) but experience different degrees of "off- seff-policy-nes".. Therefore, the active/passive gap implies that off-policy-ness. is detrimental, but it does not indicate whether wider state-action space coverage is beneficial or harmful. On the other hand, the fact that (1) Bootstrapped DQN (indiv.) likely suffers from greater "off-policy-ness than the passive agent due to the presence of more policies and that (2) Bootstrapped DQN (indiv.) outperforms the passive agent in Figure indicates that the wider state-action space coverage of the data generated by Bootstrapped DQN (indiv.) is likely highly beneficial. We leave rigorous analysis the effect of state-action space coverage for future work.
3.3 MITIGATING THE CURSE OF DIVERSITY: INITIAL ATTEMPTS
Having established the main cause of observed performance degradation, we examine whether several intuitive solutions can mitigate it in this section. More analysis can be found in Appendix [D.
Larger replay buffers A larger replay buffer provides better state-action coverage and may mitigate issues due to insufficient self-generated data such as erroneous value extrapolation (Fujimoto et al.. 2018 Sal Kumar et al. 2019 Ostrovski et al. 2021). In Figure 4 we probe the effects of replay buffer capacity in 4 Atari games and 4 MuJoCo tasks. For MuJoCo tasks, a larger replay buffer mitigates the curse of diversity (i.e., reduces the performance gap between SAC and Ensemble SAC (indiv.)) in Humanoid and Walker, though it still remains Ant. However, in Atari games, the performance gap between Double DON and Bootstrapped DON (indiv.) largely remains with larger replay buffers,

---

Published as a conference paper at ICLR 2024
91
92
9a
91
91
92
93
93
93
f
f2
f1
f2
fo
Figure 6: (left) The Q-value networks of a standard ensemble without CERL. (right) The Q-value networks of an ensemble with CERL. f and g represent neural networks. An arrow from X to Y, where X and Y are either g, or g¹, indicates X will be used as the target network of Y when performing TD updates. Auxiliary heads are in green. Dashed lines indicate the corresponding loss terms CERL auxiliary losses. Arrows with the same color originate from the same main head.
except for Amidar where the performance of Double DQN itself is severely impaired. The difference in results in the two domains may be due to the different numbers of samples required (IM versus 200M). It is possible that an even larger replay buffer may mitigate the curse of diversity in Atari, but this is extremely expensive and infeasible for us to test. Overall, these results suggest that though increasing the replay buffer capacity (within a reasonable memory budget) can mitigate the curse of diversity some environments, is not consistent remedy.
Reducing diversity The most intuitive way to mitigate the curse of diversity is, naturally, by reducing diversity. We test two techniques to do so: (1) reducing the ensemble size and (2) increasing the number of shared layers across the ensemble. In Figure [ we show the impact of these two techniques on performance and ensemble diversity in 4 Atari games. To quantify diversity, we measure the entropy of the distribution of votes among the ensemble members. As expected, both techniques reduce diversity and improve the performance of Bootstrapped DQN (indiv.). However, as the diversity of the ensemble decreases the advantages of policy aggregation also reduce, which can be seen from the tapering performance gap between Bootstrapped DQN (indiv.) and Bootstrapped DQN (agg.). These results show that even though reducing diversity can mitigate the curse of diversity, it also compromises the advantages of policy aggregation. Ideally, we would want a solution that alleviates the curse of diversity while preserving the advantages of using ensembles. In the next section, we show that representation learning offers promising solution.
4 MITIGATING THE CURSE OF DIVERSITY WITH REPRESENTATION LEARNING
Our method is motivated by simple hypothesis. We conjecture that the reason why sharing network layers mitigates the curse of diversity is twofold. First, mentioned above, sharing layers reduces ensemble diversity, thus making the generated data "less off-policy" to the individual members. Second, the shared network simultaneously learns the value functions of multiple ensemble members, leading to improved representations. This may lead to more efficient off off-policy learning by allowing the Q-value networks to better generalize to state-action pairs that have high probability under the current policy but are under-represented in the data. As we have shown, the diversity reduction aspect has the undesirable side effect of reducing the benefits of policy aggregation. We thus propose Cross-Ensemble Representation Learning (CERL), novel method that benefits from the similar representation learning effect of network sharing without actually needing to share the networks, thus preserving the diversity of the ensemble.
Method CERL, shown in Figure (right), is an auxiliary task that can be applied to most ensemble- based exploration methods that follow the recipe outline in Section 2 For ease of exposition, we use Ensemble SAC as an example. Extension to other methods is trivial. In CERL, for each Q-value network, we conceptually split it into an encoder f and head g. Our goal in CERL is to force the Q-value network encoder of each ensemble member to learn the value functions of all the ensemble members, similar to what happens when using explicit network sharing. To this end, each ensemble member has N Q-value heads (Q, a) = [g (f((s, a))] For ensemble member i, ( a) is the "main head" that defines member i's value function estimation. Each ensemble member still has only one policy n. (a)s), and the main head (s, a) the only head used update the policy. head Qf (s, a) of ensemble member where j * is used to learn the value function of ensemble member j as auxiliary task. These heads are referred to as "auxiliary heads" because their sole purpose is
7

---

Published as a conference paper at ICLR 2024
Aggregate Policies (agg.)
Individual Policies (indiv)
Diversity
Human-Nome Normalized Soore
Human Humun-Normalize Normalized Score
Voling Entropoy (Maan)
1.0
1.0
1.0
IQM
IQM
0.5
0.5
0.5
0.0
0.0
0.0
100
200
100
200
100
200
Frames (x100))
Frumes (x 109)
Promes (x10")
Double DQN
Bootstrapped DQN (L 33)
Boolstrapped DQN (L 0)
Bootstrapped DQN (L = CERL
wd
Ant
HalfCheea. vt
Epioods Ketun
6000
6000
Dpisode Retum
5000
13000
4000
4000
12000
4000
3000
11000
2000
0.5
1.0
0.2
0.5
1.0
0.5
1,0
0.2
1,0
Replay Butler Size (x119*)
Replay Butter Size (x19f)
Replay Butter Size (x:10")
Raplay Butter (x100)
SAC
Ensemble SAC (indiv)
Ensemblle SAC (agg.)
Ensemble SAC CERL (indiv.)
Dasemble SAC
CERL (agg.)
Figure 7: (top) Comparison between Double DQN, Bootstrapped DQN, and CERL in Atari. Results aggregated over 55 games and seeds. We show the performance of the agg- and indiv. versions of each ensemble algorithm in the top left and top middle plots respectively. Shaded areas show 95% bootstrapped CIs. All methods use a replay buffer of size 1M. (bottom) Comparison between SAC, Ensemble SAC, and CERL across different replay buffer sizes in MuJoCo tasks. Error bars show 95% bootstrapped CIs over 30 seeds. All ensemble methods in this figure uses N 10.
to provide better representations for the main heads. Specifically, given transition (s, a,,, 8'), we perform the following TD update for all N x N heads parallel as follows:
( a) + r + ( ddy, for i = 1,...,N, N, for j = I....,N
(1)
where Q3 the target network for Q3 and d' ~ (T((ss"). As usual, the update rule implemented as a one-step stochastic gradient descent.
Besides being conceptually simple, CERL is easy to implement. In our experiments, we find sufficient to duplicate the last linear layers of the networks as the auxiliary heads, which can be implemented by increasing the networks' final output dimensions. For the same reason, CERL is computationally efficient. For example, for the Nature DQN network (Mnih et al., 2015) used in this work, applying CERL to Bootstrapped DQN with N = 10 and L = O increases the number of parameters by no more than 5%, and the increase in wall clock time is barely noticeable. We provide pseudocode for CERL with Ensemble SAC and Bootstrapped DQN in Appendix A
Experiments We focus two questions: (1) Can CERL mitigate the curse of diversity, i.c. the
performance gap between individual ensemble members and their single-agent counterparts? (2) Do the improvements in individual ensemble members translate into better aggregate policy? To answer these questions, we test CERL on Bootstrapped DQN in 55 Atari games (Figure 7 (top)) and on Ensemble SAC in 4 MuJoCo tasks (Figure (bottom)). We compare with Bootstrapped DQN and Ensemble SAC without CERL as well as the single-agent Double DQN and SAC. To show the advantage of CERL over explicit network sharing, we also include Bootstrapped DQN with network sharing and report ensemble diversity as we did in Section 3.3 We use L to denote the number of shared layers across the ensemble. As the curse of diversity sensitive to replay buffer size in MuloCo tasks, we show results with different replay buffer sizes for these tasks. Additional results, including ensemble size ablations and an alternative design of CERL, can be found in Appendix [D.4
As shown in these results, CERL consistently mitigates the curse of diversity across the tested environments. For example, applying CERL to Ensemble SAC in Humanoid with a 0.2M-sized replay buffer reduces the performance gap between SAC and Ensemble SAC (indiv.) from roughly 3000 to around 500. More importantly, the improvements in individual policies do translate into improvements in aggregate policies, which enables CERL to achieve the best performance with policy aggregation in both domains. In contrast, even though Bootstrapped DQN (L = 3) also performs better than Bootstrapped DQN (I. = 0) when comparing individual policies, it does not provide any
8

---

Published as a conference paper at ICLR 2024
gain over Bootstrapped DQN (L= 0) when comparing aggregate policies, likely due to significantly lower ensemble diversity than Bootstrapped DQN (L = 0) as shown in Figure 7 (top right).
RELATED WORK
Ensemble-based exploration The idea of training an ensemble of data data-sharing agents that concur- rently explore has been employed in many deep RL algorithms (Osband et al. [2016] 2018 Liu et al. 2020; Schmitt et al. 2020 Peng et al. 2020, Hong et al., 2020 JanosJanueewwski et al. 2021 Most of these works focus on algorithmic design and the discussion of the potential negative effects of the increased offf-poiicy-ness compared to single-agent training has largely been missing. To the best of our knowledge, the only work that explicitly discusses the potential difficulties of learning from the off-policy data generated by other ensemble members is (Schmitt et al. (2020). However, since Schmitt et al. (2020) does not maintain explicit Q-values and relies on V-trace (Espeholt et al. 2018) for off off-policy correction, algorithmic changes that allow stable off-policy learning are a requirement in their work. In contrast, we show that even for Q-learning- based methods that do not require explicit off-policy correction, ensemble-based exploration can still lead to performance degradation. There also exist methods that use multiple ensemble members on per-step basis (Chen al. 2018 Lee
et al. [021]] Ishfag et al. [202]] Li et al., (203) as opposed to one ensemble member for each episode.
The discussion of these methods is more subtle and left for future work. [Sun et al. (2022) uses an
ensemble of Q-value networks to trade off exploration and exploitation. However, their method only uses one policy so our discussion does not apply to it.
Ensemble RL methods for other purpases Ensemble methods have also been employed in RL for purposes other than exploration, for example to produce robust value estimations (Anschel et al.
2016; Lan et al. 2020) Agarwal et al., 2020 Peer et al. [02T]] Chen et al. 2021: An et al. 202 I
Wu et al. [021] Liang et al. 2022) or model predictions (Chua et al. [2018] Kurutach et al. 2018). Despite the use of ensembles, most of these methods are still single-agent in nature (i.e., there is only
one policy interacting with the environment). Thus, our discussion does not apply to these methods.
Mutual distillation Mutual/collaboratiee learning in supervised learning (Zhang al. 2017 Anil et al., 2018)) Guo et al., 2020 Wu & Gong 2020> aims to train a cohort of networks and share knowledge between them via mutual distillation. Similar ideas have also been employed in RL (Czarnecki et al.) 2018 Xue et al., 2020; Zhao & Hospedales] 20200 Reid & Mukhopadhyay 2021). CERL is distinet from these works in that it does not try to distill different ensemble members predictions (i.c.. the value functions) into each other. Instead, CERL is only an auxiliary task, and different ensemble members in CERL only affect each other's representations via auxiliary losses.
Auxiliary tasks in RL Facilitating representation learning with auxiliary tasks has been shown to be effective in RL (Jaderberg et al. 2016)] Mirowski et al., 2016 Fedus et al., 2019 Kartal et al. 2019) Dabney et | 2020: Schwarzer et al. (020)). In the context of multi-agent RL, He & Boyd-Graber (2016). Hong et al. (2017); Hernandez- Leal et al. (201 9) and Hernandez et al. (2022) model the policies of external agents as an auxiliary task, and Barde et al. (019)) promotes coordination between several trainable agents by maximizing their mutual action predictability. Besides the clear differences in the problem domains and motivations, these multi-agen agent works only predict the actions of other agents, which typically contain less information than the Q-function used in CERL's auxiliary task.
DISCUSSION AND CONCLUSION
In line with recent efforts advance the understanding of deep RL through purposeful experi- ments (Ostrovski et al. 2021 Schaul et al. 2022, Nikishin et al. 2022 Sokar et al.] 2023), our work builds on extensive, carefully designed empirical analyses. offers valuable insights into a previously overlooked pitfall of the well-established approach of ensemble- exploration and presents opportunities for future work. As with most empirical works, an important avenue for future research lies in developing theoretical understanding of the phenomenon we reveal in this work.
A limitation of CERL is its reliance on separate networks for high ensemble diversity, which may become infeasible with very large networks. A simple improvement to CERL is thus to combine CERL with network sharing and encourage diversity with other mechanisms, such as randomized prior functions (Osband et al. 2018) and explicit diversity regularization (Peng et al. 2020).
9

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
Detailed pseudocode of the ensemble algorithms used in this work is provided in Appendix A Experimental and implementation details given in Appendix B and Appendix C respectively. The
source code is available at the following repositories:
Atari: http://ssssttttt.oo//////aa////a
MuJoCo: http:////g.ttuu...oo/////aaaaaaa
r1 cont inuous inuoouuu
ACKNOWLEDGMENTS
This research was enabled in part by support and compute resources provided by Mila (mi la guebec), Calcul Québec (www. ca culquebec. ca), and the Digital Research Alliance of Canada (a ialliinnecan... ca). We thank Sony for their financial support of Zhixuan Lin throughout this work.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. /CML, 2020.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Mare G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In NeurlPS, 2021.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- -basdd offline reinforcement learning with diversified q-ensemble. ArXiv, abs/2110.011488, 2021. URL https
7/api seapi.semanticccho.... wtt.riicccccc....ooooooouoooo 2:82599822
Rohan Anil, Gabriel Pereyra, Alexandre Passos, Róbert Ormándi, George E. Dahl, and Geof- frey E. Hinton. Large scale distributed neural network training through online distillation. ArXiv, abs/1804.03235, 2018.
Oron Anschel, Nir Baram, and Nahum Shimkin. Deep reinforcement learning with averaged target dqn. ArXiv, abs/161 1.01929, 2016.
Paul Barde, Julien Roy, Félix G. Harvey, Derek Nowrouzezahrai, and Christopher Joseph Pal. Promoting coordination through policy regularization in multi- multi-agent reinforcement learning. ArXiv, hass/990..02663 2019.
Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning envi- ronment: An evaluation platform for general agents (extended abstract). International Joint Conference Artificial Intelligence, 2012.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- and Qiao Zhang. JAX: composable transformations of Python+NumPP programs, 2018. URL
http:/ http:///gtthbt cwm/ wetgooglll/oo jax
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A research framework for deep reinforcement learning. ArXiv, fts//812.0611111 10, 2018.
Pablo Samuel Castro, Tyler Kastner, P. Panangaden, and Mark Rowland. Mico: Improved repre- sentations via sampling- based state similarity for markov decision processes. In Neural Informa- tion Processing Systems, 2021. URLhttts:://att http:/ppp..aaaneeemmmmeemmmm.m.e www.stemtttii. semanticscholarr wtt.hhol.l...o/oooooouu....
239050564
Richard Y. Chen, Szymon Sidor, P. Abbeel, and John Schulman. Ucb exploration via gennseblles. arXiv: Learning, 2018
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q-learning: Learning fast without model. ArXiv, abs/2101.0598 2021.
10

---

Published as a conference paper at ICLR 2024
Po-Han Chiang, Hsuan Yang, Zhang-Wel Hong, and Chun- Yi Lee. Mixture of step returns in bootstrapped dqn. ArXiv, his/20077.08229, 2020.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement
learning in a handful trials using probabilistic dynamics models. In NeurlPS, 2018.
Wojeiech M. Czarnecki, Siddhant M. Jayakumar, Max Jaderberg, Leonard Hasenclever, Yee Whye Teh, Nicolas Manfred Otto Heess, Simon Osindero, and Razvan Pascanu. Mix&match agent curricula for reinforcement learning. ArXiv, abs/1806.01 806.0178, 2018.
Will Dabney, Mark Rowland, Mare G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In AAAI Conference Artificial Intelligence, 2017. URL
https://aae //pp:./pp....m wts.hhottcsccchaaaam org/Corpus : 139930
Will Dabney, André Barreto. Mark Rowland, Robert Dadashi, John Quan, Mare G. Bellemare, and David Silver. The halue-improvement path: Towards better representations for reinforcement learning. ArXiv, abs/2006.02244 02243, 2020.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. Im- pala: Scalable distributed deep-rl with importance weighted actor-learner architectures. ArXiv, abs/1802.01561, 2018
William Fedus, Carles Gelada, Yoshua Bengio, Marc G. Bellemare, and H. Larochelle. Hyperbolic discounting and learning over multiple horizons. ArXiv, abs/1902.06865,, 2019.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, 2018a.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in
actor-critic methods. International Conference on Machine Learning, 2018b. https
//api. www.etttscchoooooo.oo.....
Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo.
Online knowledge distillation via collaborative learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1107-11026, 2020.
Tuomas Haarnoja, Aurick Zhou, P. Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 201 8a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, G. Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, P. Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. ArXiv, abs/18 12.05905, 2018b.
H. V. Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double g-learning. In AAAI Conference on Artificial Intelligence, 2015.
He He and Jordan L. Boyd-Graber. Opponent modeling in deep reinforcement learning. ArXiv, Mss/160090555, 2016
Daniel Hernandez, Hendrik Baier, and Michael Kaisers. Brexit: On opponent modelling in expert iteration. ArXiv, abs/2206.001 13, 2022.
Pablo Hermandez-Leal, Bilal Kartal, and Matthew E. Taylor. Agent modeling as auxiliary task for deep reinforcement learning. In Artificial Intelligence and Interactive Digital Entertainment Conference, 2019.
Zhang-War Wei Hong, Shih- Yang Su, Tzu- Yun Shann, Yi-Hsiang Chang, and Chun- Yi Lee. A deep policy inference nentwwork for multi-agent systems. ArXiv, tbs/1772.07793, 2017.
Zhang- Wei Hong, Prabhat Nagarajan, and Guilherme Maeda. Periodic intra-ensemble knowledge distillation for reinforcement learning. ArXiv, abs/2002.00149, 2020.
11

---

Published as a conference paper at ICLR 2024
Haque Ishfaq. Qiwen Cui, Viet Huy Nguyen, Alex Ayoub, Zhuoran Yang. Zhaoran Wang, Doina Pre- cup, and Lin Yang. Randomized exploration for reinforcement learning with general value func- tion approximation. ArXiv, abs/2106.07841, 2021. htths::s: semannisssaaiiccc....a.
org/CorpustD: 235435553
Max Jaderberg, Volodymyr Mnih, Wojciech M. Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. ArXiv, abs/1611.05397, 2016.
Piotr Januszewski, Mateusz Olko, Michal Królikowski, Jakub Bartlomiej Swiatkowski, Marcin Andrychowicz, Lukasz Kuci nski, Kuci'nski, and Piotr Milo's. Continuous control with ensemble deep deterministic policy gradients. ArXiv, abs/21 1.15382,, 2021.
Bilal Kartal, Pablo Hernandez-Leal, and Matthew E. Taylor. Terminal prediction as an auxiliary task for deep reinforcement learning, In Artificial Intelligence and Interactive Digital Entertainment Conference, 2019.
Ilya Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms in JAX, 10 2021.
htt.hhtps:///ee thub. www.ooossssrrrar....ooo jaxr
Aviral Kumar, Justin Fu, G. Tucker, and Sergey Levine. Stabilizing off- off-policy q-learning via bootstrapping error reduction. In NeurlPS, 2019.
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron C. Courville, Tucker, and Sergey Levine. Dr3: Value-based deep reinforcement learming requires explicit regularization. ArXiv, abs/2112.04716 2021. StRLhttps://aat semant wscccccchlccoocoococ wtt//oorr/seee.. ID: : 245005650
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and P. Abbeel. Model-ensemble trust- region policy optimization. ArXiv, abs/1882.10052 2018.
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learning: Controlling the estimation bias of q-learning. ArXiv, abs/2002. 2020.
Kimin Lee, Michael Laskin, A. Srinivas, and P. Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In ICML, 2021.
Qiyang Li, Aviral Kumar, Ilya Kostrikov, and Sergey Levine. Efficient deep reinforcement learning requires regulating overfitting. ArXiv, abs/2304. 10466, 2023.
Litian Liang, Yaosheng Xu, Stephen McAleer, Dailin Hu, Alexander T. Ihler, P. Abbeel, and Roy Fox. Reducing variance in temporal-different value estimation via ensemble of deep networks. ArXiv, abs/2209.07670, 2022.
Longxin Lin. Reinforcement learning for robots using neural networks. 1992.
Ge Liu, Rui Wu, Heng-Tze Cheng, Jing Wang, Jayden Ooi, Lihong Li, Ang Li, Wai Lok Sibon Li, Craig Boutilier, and Ed H. Chi. Data efficient training for reinforcement learning with adaptive behavior policy sharing. ArXiv, abs/2002.05229, 2020.
Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, and Michael H. Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. ArXiv, abs/ 709,060099 2017.
Li Meng, Morten Goodwin, Anis Yazidi, and Paal Einar Engelstad. Improving the diversity of bootstrapped dqn by replacing priors with noise. IEEE Transactions on Games, 2022.
Piotr Wojeiech Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andy Ballard, Andrea Banino, Misha Denil, Ross Goroshin, L. Sifre, Koray Kavukcuoglu, Dharshan Kumaran, and Raia Hadsell. Learning navigate in complex environments. ArXiv, abs/161 1t5//611..00 1.03673, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-533, 2015.
12

---

Published as a conference paper at ICLR 2024
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Puerre-Luc Bacon, and Aaron C. Courville. The primacy bias in deep reinforcement learning. In ICML, 2022.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In NIPS, 2016.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. ArXiv, abs/1806.03333 2018.
Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. NeurIPS, 2021.
Oren Peer, Chen Tessler, Nadav Merlis, and Ron Meir. Ensemble bootstrapping for q q-learning. In ICML, 2021.
Zhenghao Peng, Hao Sun, and Bolei Zhou. Non-local policy optimization via reveessty collaborative exploration. ArXiv, abs/2006.07781, 2020.
John Quan and Georg Ostrovski. DQN Zoo: Reference implementations of DQN-based agents, 2020.
URL httt::t/gtthhiii com http:///www....eeo//c/////e////a
Cameron Reid and Snehasis Mukhopadhyay. Mutual reinforcement learning with heterogenous
agents. 202/ IEEE International Conference Smart Computing (SMARTCOMP), pp. 395-397, 2021.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. CoRR, 000//5111055555 2015. /ttttpp://api..smmmttiiii htt.aa/mnicccccll.....oor/////oo 13022595
Tom Schaul, André Barreto, John Quan, and Georg Ostrovski. The phenomenon of policy churn.
ArXiv, abs/2206.007300 2022. URL https::/ semanticscholar, 249282416
Simon Schmitt, Matteo Hessel, and Karen Simonyan. Off-policy actor-critic with shared experience replay. ICML, 2020.
Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron C. Courville, and Philip Bach- man. Data-efficient reinforcement learning with self- self-predictive representations, In International Conference on Learning Representations, 2020.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phe- nomenon deep reinforcement learning. In International Conference Machine Learning. 2023.
semanticscholee ar org/ http//Crpuussi : 257219318 URL https: http:///aap..ssaaaaa...e..
Hao Sun, Lei Han, Rui Yang, Xiaoteng Ma, Jian Guo, and Bolei Zhou. Optimistic curiosity
exploration and conservative exploitation with linear reward shaping. 2022. URL https
//api. sccchooa wttp:oooaa.....ooooooouo....e :ttp::/CcorruII.:222277733333
Mark Towers, Jordan K. Terry, Ariel Kwiatkowski, John U. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G. Younis. Gymnasium, March 2023. URL htttss/ts::////t http:::::///eeooooom org wwtp://ddd..rrr/erooo//ooo. www.re/oorodooooo 02////2272255
Guile Wu and Shaogang Gong. Peer collaborative learning for online knowledge distillation. ArXiv, abs/2006. 0417,, 2020.
Yanqiu Wu, Xinyue Chen, Che Wang, Yiming Zhang, Zijian Zhou, and Keith W. Ross. Aggressive q learning with ensembles: Achieving both high sample efficiency and high asymptotic performance. ArXiv, 0hs/2111.0911 2021. URL. httpps:/:
semant semanticsche cschol orr.or///rppus0000
244270282
Zeyue Xue, Shuang Luo, Chao Wu, Pan Zhou, Kaigui Bian, and Wei Du. Transfer heterogencous knowledge among peer-to-peer teammates: A model distillation approach. ArXiv, abs/2002.02202, 2020.
13

---

Published as a conference paper at ICLR 2024
Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4320-4322 2017.
Chenyang Zhao and Timothy M. Hospedales. Robust domain randomised reinforcement learning through peer-to-peer distillation. ArXiv, abs/2012.04839, 2020.
14

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
Algorithms
16
B Experimental details
16
Atari
16
MuJoCa
16
p%o-tandem experiments
21
C
Implementation details
21
C.1
Implementation and hyperparameters
21
Gradient scaling
21
C.3
Computational costs
21
Additional results
21
D.1
10%-tandem experiments continuous control tasks
................................
21
D.2
Per-game results
................................
23
D.2.1
Per-game learning curves
23
D.2.2
Per- analysis of the benefits of majority voting
23
D.2.3
Per-gamo performance of CERL
23
D.3
Additional analysis of the curse of diversity
23
D.3.1
Over-sampling self- ceff-generated data the training batches
23
D.3.2
Episode termination condition
23
D.3.3
Distributional RL
28
D.3.4
Switching ensemble members per-step basis
29
D.3.5
Layer sharing experiment with larger replay butfer
30
D.3.6
Different levels of passivity for the p% tandem experiment
30
D.3.7
Data booistrapping
31
D.4
Additional analysis of CERL
31
D.4.I
Ensemble size ablation
31
D.4.2
An alternative design of CERL
31
D.4.3
Combining CERL with encoder sharing
32
D.4.4
Number of parameters]
33
D.4.5
Other representation learning methods: preliminary investigation
33
15

---

Published as a conference paper at ICLR 2024
A ALGORITHMS
In this section, we provide the pscudocode for the ensemble algorithms used in this work. For simplicity, we only describe the case of batch size / = 1. For B > 1 we simply average the loss across the batch. Note we use and & to denote the parameters of the entire ensemble. In other words, subset of 6. All ensemble algorithms do not use data bootstrapping, as Osband et al. (2016) finds it barely affects performance in complex domains. All ensemble gradient updates are performed in parallel in practice, though in the pseudocode they written for loops.
Qi(s, 8) only depends
The provided algorithms are:
Algorithm I and Algorithm 2 Bootstrapped DQN and Bootstrapped DQN CERL. The changes needed for CERL are highlighted in yellow.
Algorithm 3 and Algorithm Ensemble SAC and Ensemble SAC CERL. The changes needed for CERL are highlighted in yellow. For Ensemble SAC + CERL, we find it helpful use huber loss with threshold of 10 for the CERL auxiliary loss to prevent certain diverging ensemble members from affecting all other members. This is shown in the algorithm description. Also note that Clipped Double Q-learning (Fujimoto et al. 2018b) used in practice but omitted in the algorithm description.
Note the above only describes the behavior of these algorithms during training. When evaluating the individual ensemble members' performance, these algorithms behave exactly the same as training time except that there are no learning updates. When evaluating the aggregate policies, we aggregate the policies of different ensemble members. For Bootstrapped DQN (+CERL), the policies are the
greedy policies with respect to {Q:) (or the main heads {Q: for CERL) and are aggregated via majority voting; specifically, for each visited state during evaluation, for each action in the action space, we count the number of ensemble members that select that action (i.e., votes) and we select the action with the most votes. Ties are broken randomly. For SAC simply use the average of actions sampled from different policies. For example, if we visit state s during evaluation, we will sample a, (-(+ from each ensemble member, and then take the action N E a, in the environment.
B EXPERIMENTAL DETAILS
B. ATARI
For Atari, we use the same set of 55 games Agarwal et al. (202I). Following (Castro et al. 2018), the training process is divided into 200 iterations, each of which contains 1M frames. At the end of each iteration, the networks are frozen and evaluated with at least 500k frames. All Atari experiments use 5 seeds.
For each game, the human normalized scores (HNS) are computed follows
Score Score
Scorenormalized ScoreHuman ScoreRandom
where ScoreAgent is the raw score of the considered agent, and ScoreHuman and ScoreRandom are the raw scores of the human and the random agent respectively. The raw scores of an agent are calculated as the undiscounted evaluation returns averaged over the last 10 training iteration and 5 seeds. The scores of the human and random agents are taken from DQN Zoo (Quan & Ostrovski 2020). To obtain results in Double DQN normalized scores, simply replace ScoreHuman with ScorepdQN in the above equations. ScorepDQN is the raw score obtained by the Double DQN agent, averaged over the last 10 training iteration and 5 seeds.
When showing per-game improvements, simply compute the difference in human- mormalize scores. Per-game improvement bar plots (c.g., Figure (top-ightt) are shown in log-scale with a linear threshold at 0.1.
B.2 MuJoCo
Each agent is trained for total of 1M steps. Evaluation is performed every 20k steps with 30 evaluation episodes. When reporting the final performance in an environment, average the last
16

---

Published as a conference paper at ICLR 2024
Algorithm Bootsttrpped DQN (training)
Require: total interaction steps M, ensemble size N, gradient update period P, target update period
T, N value functions {Qi(s, a; )) N target value functions (8, ) replay buffer 'D True
Terminal state indicator/whether to start a new episode
for m < 1 to M do
t True then
b New episode
8m reset(env)
k ~ Uniform (f1...., NT)
Randomly sample a member for acting
else
8m <- 8
end if
am f- arg max, (s(sm, a; 0)
am e-greedy((mm m))
Epsilon greedy policy with decay schedule
sn T'm fm t step(env, 0m)
Add (sm, am. m', s'm fm) the shared replay buffer D if m mod P= then
b Gradient update
Sample transition (s,a, T, s1,) from
for j (1, (I,...,N) N} do
a' + arg max, 2 d); r + 12 a'; 0)
if False
Vj +
b Double DQN update
if l True
end for
L(0) Σ N huber loss(Q a; 0) yi) 50 + (o)
80 < scale (6)
Scale the gradients the encoder(s). See Section C.2
0 optimizer(O, 00)
end if
if mod T then
0
> Update target network
end if
end for
17

---

Published as a conference paper at ICLR 2024
Algorithm 2 Bootstrapped DQN CERL (training)
Require: total interaction steps M, ensemble size N, gradient update period P, target update period
T, N N value functions {Q( (s, 0)) N N target value functions {Q/ (s, a; 0))
replay buffer D True
Terminal state indicator/whether to start a new episode
for m + 1 to M do
if True then
> New episode
8m reset(env)
k ~ Uniform(f1, NF)
Randomly sample a member for acting
else
8m f- mm1
end if
am + arg max, Q (sm a;
<-greedy(Ams m)
Epsilon greedy policy with decay schedule
8m T'm lm 5- (ete,een, 0m)
Add (Sm, am: T'm s'm fm) to the shared replay buffer D m mod p = then
b Gradient update
Sample transition (s. s,a,r,s', stt from D for j E [1....,N[ N) do
a' + arg max.. ('(ss'
(s',
if ( False if l True
b Double DQN update
end for
L(0)- + Ei د loss a) yı)
80 < (oL
50 < scale (gadd 60)
> Scale the gradients the encoder(s). See Section C.2
0 optimizer(0,, 50)
end if
if mod T then
0 + 0
> Update target network
end if
end for
18

---

Published as a conference paper at ICLR 2024
Algorithm 3 Ensemble SAC (training)
Require: total interaction steps M, ensemble size N, gradient update period P. target update
temperature T, N value functions {Qi(s, a: 0) N target value functions {Qi az 0) N
policies (alss Φ) j= N entropy temperatures {a, J. { D Terminal state indicator/whether to start a new episode
replay buffer D
< True
for 1 to M do
if t = True then 8m f reset(env) k Uniform(f)) ..)
> New episode
Randomly sample a member for acting
else
f- 8'm-1
end if
TT(-118mm s'm T'm bm f- step(env, 0m)
Add (Sm, am. T'm S' fm) to the shared replay buffer D m mod p= then
D Gradient update
Sample transition ssa,r,s'' from D for (l,,..., N} do
a' d-10
Mj r 1
if False
r
if = True
(6) ~ (-(111;)))
b Note with the parametrization trick, depends on Ø
end for
Leritie (0) ( + Σn (Qi(s, 0) -) - 3²
and t optimizer (,, 50) Lator (0) + -
80 < V Lenue
b Update critic
- a, log n. (a:()sss: +))
de V Lo (0) and +- optimizer(6) 5Φ)
Update actor
Update the entropy temperatures (a;) for each member as standard SAC
end if
0 + (1 - T) 70
> Update target network
end for
19

---

Published as a conference paper at ICLR 2024
Algorithm 4 Ensemble SAC + CERL (training)
Require: interaction steps M, ensemble size N, gradient update period P. target update temperature
T, x N value functions {Q] a: -) N N target value functions (Q) (s, a: 0)
N entropy temperatures {as) replay buffer D True
Terminal state indicator/whether to start new episode
for m f 1 to M do
if True then
> New episode
8m + reset(env) k ~ Uniform(F)
NF)
Randomly sample a member for acting
else
8m f- 6'm-1 end if
am ~ TT( Tx(--smmi σ)
S T'ms tm step(env, 0m)
Add (8m, am, T'y Sm, (m) to the shared replay buffer D
if mod P= then
Gradient update
Sample transition (s,a, r, 8', from for j {1, [I....,N N) do
a' ~ x (-15/00)
+ 1+ d':0)
if False
--
if =
Truc
(Ø) (((()s
Note with the parametrization trick, aj depends on
end for
Lentic (8) +- ΣN Ey= huber (Qs
y,) Σi=11 (Qi(s, a: y:) P Update critic
80 t- VoLicritic (0) and + optimizer (0,
Lactor( f- Σi ( (0): a; log (a:
dΦ Vo ()) and ø + optimizer(f.66)
b Update actor
Update the entropy temperatures {a;} for each member as in standard SAC end if 0 +
T) -0
Update target network
end for
20

---

Published as a conference paper at ICLR 2024
5 evaluation results at the end of training. All MuloCo experiments 30 seeds except for the 10%-tandem experiments, where we use 10 seeds.
B.3 P76-TANDEM EXPERIMENTS
Taking Bootstrapped DQN as an example, the easiest way to understand the experimental setup by noticing that setting p 50% recovers the standard Bootstrapped DQN with ensemble members. The only differences of 2%%-tandem to Bootstrapped DQN with N = are (1) for each training 50% probability,
episode, instead of sampling each agent for acting with the active agent with probability - p% and the passive agent with probability p% and (2) we record the performances of both the active and passive agents separately during evaluation.
sample the
C IMPLEMENTATION DETAILS
C.I IMPLEMENTATION AND HYPERPARAMETERS
The SAC and ensemble SAC algorithms built JAXRL (Kostrikov 2021) and use the default hyperparameters. the actor implementation, we use tanh instead of hard clipping for the loq_sto parameter for better stability. Double DQN and Bootstrapped DQN are implemented using the Dopamine (Castro et al. 2018) framework in JAX (Bradbury et al., 2018). As our work heavily refers to Osband et al. (2016) and Ostrovski al. (2021). we mostly follow the hyperparameter setup in DQN Zoo (Quan & Ostrovski, (020)) because those are more similar to the ones used in Osband al. 2016) and Ostrovski et (202T). We do not directly build on DQN Zoo mainly because the implementation in Dopamine is more efficient. The hyperparameters for Double DQN are listed in Table !
Note one "agent steps" corresponds environment frames due the use of action repetitions/frame-siip
Bootstrapped DQN reuses all the hyperparameters Double DQN, with two additional hyperparam- eters: ensemble size N and the number of shared bottom layers T., We set N 10 and T. by default.
As mentioned in Appendix A for Ensemble SAC CERL, we find it helpful to use huber loss with threshold of 10 for the CERL auxiliary loss to prevent certain diverging ensemble members from affecting all other members. The main head still uses the regular MSE loss so the use of Huber loss only affects the auxiliary task.
C.2 GRADIENT SCALING
Following Osband et al. 2016), we scale the gradients of the encoder(s) to ensure that the magnitude of the gradients does not increase with the number of heads. For example, in CERL, the gradients of the encoders fa will be divided by N, which is the number of heads on top of this encoder.
C.3 COMPUTATIONAL COSTS
On a server with NVIDIA RTX8000 GPU and AMD EPYC 7502 CPU, each seed of our implementa- tion of Double DQN, Bootstrapped DQN (N 10, share 0 layers), and Bootstrapped DQN (N 10, share 3 layers) take roughly 2, 3.5, and 3 days respectively for 200M frames. Applying CERL does noticeably increase the wall clock time in our experiments. Running 10 seeds of SAC, Ensemble SAC, and Ensemble SAC + CERL with N = 10 on Humanoid-ve takes roughly 5, 12.5, 13 hours respectively.
D ADDITIONAL RESULTS
D.1 10%-TANDEM EXPERIMENTS FOR CONTINUOUS CONTROL TASKS
In Figure 8 we show the results of the 10%-tandem experiment in MuJoCo tasks, which shares the same patterns as the one in Atari. MuJoCo tasks, the performance gap between the active and the passive agent is larger than that between Ensemble SAC (indiv.) and SAC. This might be because
21

---

Published as a conference paper at ICLR 2024
Table Hyperparameters for Double DQN
Parameter
Value
Gray-scaling
True
Observation down-sampling
84 x 84
Frames stacked
Action repetitions
4
Sticky actions
False
Reward clipping
[-1, 1, 1]
Terminal on loss of life
False
Max frames per episode
108k
Q update rule Discount factor
Double DQN
0.99
Minibatch size
32
Replay buffer size
10
Optimizer
RMSProp
Optimizer: learning rate
0.00025
Optimizer: R.SPPrpp decay
0.95
Optimizer: RMSProp centered
True
Optimizer:
1 1/322
Huber loss
True
Evaluation frames
500k
Evaluation period in frames
1M
Min replay size for sampling
50000
Gradient update period in agent steps
4
Target network update period in agent steps
30000
Exploration: during training
1.0
0.01
Exploration: decay period in agent steps
16M
Exploration: during evaluation
0.01
Q network: channels
32, 64, 64
Q network: filter size
8, x 4,
Q network: stride
4, 2,
Q network: hidden units
512
Q network: padding type
valid convolution
Q network: share bias across action heads
True
Q network: initialization
See Quan & Ostrovski 2020
22

---

Published as a conference paper at ICLR 2024
Humanoid-v4
Ant-y4
Walker2d-v4
HallCheetah-v4
Episodo Return
4000
10000
4000
4000
2000
2000
2000
5000
0 0.0
0
0.5 Steps
1.0
0.0
0.5
1.0
0.0
0.5 Steps
1.0
0.0
0.5 Steps
1,0
le6
Steps
le6
1e6
le6
SAC
Ensemble SAC (indiv.)
Active (90%)
Passive (10%)
Figure 8: 10%-tandem experiments in MuloCo tasks with replay buffer size of 200k. Results are aggregated over 10 seeds for the active and passive agents and seeds for others. Shaded areas show 95% Cls.
Ensemble SAC (indiv.) provides better exploration and partially compensates for the performance loss due to challenging off-policy learning.
D.2 PER-GAME RESULTS
D.2.1 PER-GAME LEARNING CURVES
shows per-game comparisons between Double DQN, vanilla Bootstrapped DQN, and CERL with policy aggregation. Figure T0 shows similar comparisons without policy aggregation. Figure shows the per-game results for the 10%-tandem experiment.
Figure 9
D.2.2 PER-GAME ANALYSIS OF THE BENEFITS OF MAJORITY VOTING
In Figure 12 we show the performance gap between Bootstrapped DQN (agg.) and Bootstrapped DQN (indiv.) in each game. As shown the results, majority voting provides significant performance gains in almost all games.
D.2.3 PER-GAME PERFORMANCE OF CERL
Per-game performance of CERL is shown in Figure and Figure improvements of CERL over Bootstrapped DQN in Figure 13
We also show per-game
D.3
ADDITIONAL ANALYSIS OF THE CURSE DIVERSITY
D.3.1 OVER-SAMPLING SELF-GENERATED DATA IN THE TRAINING BATCHES
Even though we cannot control the proportion of self- self-generated data in the replay buffer for each ensemble member, it is possible to control this proportion in the training batches. In this experiment, for each ensemble member, with 50% probability we only sample the training batches from self- generated transitions; otherwise, we sample uniformly from all the data as usual. This ensures the proportion of self-generated data in the training batches for each ensemble member is at least 50%.
Figure 14 shows the effects of over-sampling self-generated data the training batches for each ensemble member. As can be seen, this technique does not mitigate the performance loss relative to single-agent Double DQN.
D.3.2 EPISODE TERMINATION CONDITION
In Atari, we have the option terminate episode when life is lost, or only when the game over. The former option may help the agent quickly learn the significance of death. In the context of ensemble-based exploration, shorter episodes mean it is less likely that single ensemble will dominate the environment interaction for long stretch of time, during which other ensemble members perform interaction at all.
23

---

Published as a conference paper at ICLR 2024
Alien
Amidar
Assault
Asterix
Asteroids
1ppods Rmnre
4200
20000
2000
1003
1.0
1c6
Atlantis
BankHeist
BartleZZone
BeamRider
000
Berzerk
0.5
5000
400
200
Bowling
Boxing
Breakout
Centipode
ChopperCommand
Epaale Ketan
4000
200
2000
Crazy Climber
DermonAttack
DoubleDunk
Enduro
FishingDerby
Rours
100200
1000
50
-2
Freeway
Frosthite
Gopher
Gravitar
Hero
Epecck Natum
200
2003
10003
20000
LocHockey
Jamesbond
Kangaroo
Krull
Kunastaaatter
Ippost kaun
10000
7500
30000
-10
5000
10000
MontezumaRevemge
MsPacman
Name ThisGiam
Phoenix
Pitfall
Epied Koars
L0003
5900
2002
5003
Pong
Private Eye
Qbett
Riverrid
ReadRanner
20
15900
1003
L0000
10000
20000
-20
Robotank
Seaquest
Skiing
Solaris
SpacoEnvaders
Spiied Roau Roam
20003
10
-20003
2000
10000
1000
-30003
StarGunner
Tennis
TimePilot
Tutunkham
UpNDown
10003
50000
20000
25000
5003
Venture
VidkoPinbail
WizardOWWor
YarsRewange
Zaxxon
Eglodd Betua
200
10003
10000
5000
20000
5000
10000
100
100
(410f))
(x:11f)
Framcs (x:10)
Double DQN
Bestrrppped DQN (2:
Dooesrrpped DGN (A.:
CERL
Deststappol DUNNC
Figure 9: Per-game comparisons between Double, vanilla Bootstrapped DQN, and CERL with voting. All results are aggregated over 5 seeds. Shaded areas show 95% bootstrapped Cls. The learning curves smoothed with sliding window of 5 iterations.
24

---

Published as a conference paper at ICLR 2024
Alien
A Amidar
Assauit
Asterix
Asteroids
2003
Eppodd Rarre
4000
10020
500
Atlantis
BankHeist
BattleZZone
BeamRider
Berzerk
30003
600
20003
400
10003
200
Bowling
Boxing
Breakout
Centipode
ChopperCommand
Eplage Ketan
200
2000
Crazy Climber
DemcnnAtiack
DoubleDunk
Enduro
FishingDerby
50000
Freeway
Frosthite
Gopher
Gravitar
Hero
Epecck Natum
10003
200
2003
5003
LooHockey
Jamesbond
Kangaroo
Krull
KuMastat
Ippoot kaun
7500
30000
5000
5000
10000
MontezumaRevemee
MsPacman
Name ThisGiame
Phoenix
Pitfall
3002
10003
4000
2000
1003
Pong
Private Eye
Qbett
Riverrii
RoadRanner
20
15000
250
10000
20000
5000
-20
-259
Robotank
Seaquest
Skiing
So Saris
SpacoEnvaders
Spiied Roau Roam
2000
10003
25003
1000
2000
-30003
StarGunner
Tennis
TimePilot
Tutunkham
UpNDown
10000
50000
5000
100
10020
25000
Venture
VidkoPinball
WizardOWWor
YargRewage
Zaxxon
Egtodd Betua
30000
200000
20000
10000
100
202
100
200
(4100))
(x:11f)
France (x110)
Double DQN
Bestrtapped DQN (Z:
Doousrapped DGN (
CE2L
Bosttrappal DAN (3)
Figure 10: Per-game comparisons between Double, vanilla Bootstrapped DQN, and CERL without voting. All results are aggregated over 5 seeds. Shaded areas show 95% bootstrapped Cls. The learning curves are smoothed with sliding window of 5 iterations.
25

---

Published as a conference paper at ICLR 2024
Alien
A midar
Assauit
Asterix
Asteroids
2003
Eppodd Rarre
4000
10000
1003
2000
500
Atlantis
BankHeist
BattleZZone
Bice
Berzerk
Epicods Raturs
30003
600
20003
5000
400
10003
200
Bowling
Boxing
Breakout
Centipode
ChopperCommand
Epiald Ketin
201
CazyClimber Climber
DemonnAttcck
DoubleDunk
Enduro
FishingDerby
10003
5003
500
Freeway
Frosthite
Gopher
Gravitar
Hero
Epecck Natum
10000
2003
200
5000
1000
0
LocHockey
Jamesbond
Kangaroo
Krull
K KuMastrr
Iphoot kaun
7500
20000
5000
5000
10000
MontezumaRevegge
MsPacman
Name ThisGiame
Phoenix
Pitfall
1000
10003
4000
85
2000
0.0
2003
Pong
Private Eye
Qbett
Riverrid
RoadRanner
20
15000
250
10000
20000
soco
-20
-250
Robotank
Seaquest
Skiing
Salaris
SpacoInvaders
Spiied Roau Roam
20000
2000
10003
-25003
1000
2000
-30003
StarGunneer
Tennis
TimePilot
Tutunkham
UpNDown
10000
50000
5000
25000
50
Venture
VidcoPinball
WizardOWWor
YareRewange venge
Zaxxon
10000
SgloodRReun Setun
30000
10000
200000
5003
20000
10000
100
203
130
100
200
(410f))
(x:1ff)
France (x110)
DQN
Boottrapped (iddvssaa)
(5072)
Pessive
Figure 11: Per-game comparisons of the 10%-tandem experiment. All results aggregated over 5 seeds. Shaded areas show 95% bootstrapped Cls. The learning curves are smoothed with a sliding window of 5 iterations.
26

---

Published as a conference paper at ICLR 2024
Performance gap between Bootstrapped DQN (agg.) and Bootstrapped DQN (indiv.)
10'
Differences in HNS
10a
10
Skinn
Solums
Gravitar
Venture
Revengs
lyye
Asteroids
Pitall
Pong
Freewny
Centiprue
Phoenix
Bergelk
Bowling
Yars Revenge
Zorxon
Jame dhend
Beam Rider
Chopper Commund
Ice Hockey
Hero
Alien
Frostbit
Amader
Kung Fly Master
Seaquest
Bank Hoist
River Rivermid
Name This Gamoo
Robolank
A Asautt
Tuaankhan
Fishing Derhy
Invaders
Up N. Down
Enduro
Asterix
Wizard Of Wor
Krall
Kanparoo
Cra.y Clanber
Boxing
Gurmer
Road Rumner
Cuglter
Double Dunk
Demon Artack
Atlambs
Breakout
Video Pinbell
Montezuna
Figure 12: Per-game performance gap between Bootstrapped DQN (agg.) and Bootstrapped DQN (in-
div.) in HNS.
Bootstrapped DQN CERL (apg.)
10'
Improvement over Bootstrappen DQN
10
Human- Normalliee Score
-10
10"
Bootstrappod DQN CERL (indiv.)
10
10
-10
-10"
Allanis
Double Drink
Crazyy Thim Clibber
Time Pilot
Enduro
Sapgaroo
Zibpson
Bank feis
R0 warrald
Selgness
Baitle Zone
Vemaons
Centiponn
Pong
Bow ling
Pri vate Eye
Montezuma Reg evengee
Phrall
Solatt
Asticrodd
Beam Rider
Jamoshon band
Obeert
Ice phoenix
Prosibitg
Berzerk
ovenge
Amiiar
Scilng
Alien
Boxing
Tymenk
Chaggerr Connina and
Wizard Wor
Down
Pasman
Road goott
Kung Fu Master
Fistung Derby
Name This Game
A ssaul
Grenner
Breakkol Kous
Tennis
Astori
Clogher
Demon Aftack
Video ingaddes
Yors
Figure 13: Per-game improvements of Bootstrapped DQN + CERL over Bootstrapped DQN HNS.
with and without policy aggregation.
27

---

Published as a conference paper at ICLR 2024
Amidar
ChopperConmand
KungPu Maattr
Spacelnvaders
Episode Return
4000
20000
1000
2000
2000
10000
0
100
200
0
100
200
100
200
100 Frumes (x10*)
200
Frumes (x10*)
Frumes (x 10*)
Frumes (x100)
Bootstrapped DQN (aga.)
Boalstrupped DQN (agg.) oversampling
Double DQN
Amidar
ChopperCommand
KungFuMaste
Spacelnvaders
Episode Return
4000
20000
1000
2000
2000
10000
0
0
100
200
0
100
200
100
200
100
200
Frames (x10*)
Frames (x 10
Frames (x10*)
Framos (x100)
Bootstrapped DQN (indiv.)
Bootstrapped DQN (indiv.) + oversampling
Double DQN
Figure 14: Effects of oversampling self- self-generated data in the training batches. (top) Performance with voting, (bottom) Performance without voting, All results are aggregated over seeds. Shaded areas show 95% bootstrapped Cls. The learning curves are smoothed with sliding window of iterations.
Amnidar
ChopperCommaand
KungFuMaste
SpaceInvaders
Episode Return
1000
4000
20000
2000
2000
1000
100
200
100
200
100 Fmmes (x10*)
200
200
Frames (x10*)
Frames 10*)
Frames (x105)
Bootstrapped DQN (agg)
Double DQN
Amidaar
ChopperCommnand
KungFuMastr
Spacelnvaders addes
Episode Retura
4000
www
2000
1000
20000
2000
1000
100
200
100
200
100
200
100
200
Frames (x10*)
Framos (x10*)
Framos (x10')
Framos (x10*)
Bootstrapedd DQN (indiv.)
Double DQN
Figure 15: Results with the episode termination condition set to life loss. (top) Performance with voting. (bottom) Performance without voting. All results are aggregated over seeds. Shaded areas show 95% bootstrapped Cls. The learning curves are smoothed with a sliding window of 5 iterations.
Our work follows the recommendation of Machado et al. (2017) and only terminates an episode when the game is over. In Figure we show the performance of Bootstrapped DQN and Double DQN when we set the termination condition to life loss. As can be seen, the curse of diversity still remains.
D.3.3 DISTRIBUTIONAL RL
Recent work (Agarwal et al. 2020) has shown that the QR QRDDQN algorithm (Dabney et al. 2017) can be more effective than DQN offline RL. In Figure 16 we test variant of Bootstrapped DQN where we train an ensemble of DQN). We use the implementation and default hyperparameters of QR-DQN in Dopamine (Castro)
agents instead of Double DQN agents (named Bootstrapped QR-
28

---

Published as a conference paper at ICLR 2024
Amidar
Chopper Command
KungPuMasser
Spacelnvaders
Episode Return
2000
10000
30000
2000
1000
5000
20000
1000
10000
0
0
100
0
100
100 Frames (x10*)
100
Frames (x10*)
Frames (x10*)
Frames (x10")
QR-DQN
Bootstrapped QR DQQ (agg.)
Bootstrapped QR-DQV (indiv.)
Performance of Bootstrapped DQN when using QR-DQN as the base algorithm. All results are aggregated over 5 seeds. Shaded areas show 95% bootstrapped CIs. The learning curves are smoothed with a sliding window of 5 iterations.
Figure 16:
Amidar
ChopperCommand
KunseuMaster
SpaceInvaders
Episode Return
4000
20000
1000
2000
2000
10000
0
100
200
0
100
200
100
200
1000
200
Frames (x109)
Frames (x10*)
Frames (x11F)
Frames (xX10%
Bootstrapped DQN (agg.)
Bootstrapped DQN (agg.) (per stop)
Double DQN
Amidar
ChopperCommand
KungFuMaster
Spacelnvaders
Episode Roturn
4000
20000
1000
2000
10000
0
100
200
0
100
200
100
200
100
200
Frames (x109)
Frumes (x 10
Frimes (x110F)
Frimes (x100)
Bootstrapped DQN (indiv.)
Bootstrapped DQN (indiv.) (per step)
Double DQN
Figure 17: Effects of switching ensemble members a per-step basis instead of per-episode. (top) Performance with voting. (bottom) Performance without voting. All results are aggregated over 5 seeds. Shaded areas show 95% bootstrapped Cls. The learning curves smoothed with sliding window of 5 iterations.
et al. 2018) except that we do not use prioritized experience replay (Schaul et al. 2015) as it is unclear whether is appropriate to prioritize transitions based on the TD error of the entire ensemble.
As shown in Figure 16 there exists significant performance gap between QR-DQN and Bootstrapped QR-DQN. This suggests that the curse of diversity is also present in distributional RL.
D.3.4 SWITCHING ENSEMBLE MEMBERS ON PER-STEP BASIS
The original Bootstrapped DQN switches the ensemble member used for acting at the start of each episode. In Figure 17, we test a variant of Bootstrapped DQN where we switch the ensemble member for acting exploration" " (Osband et al.) (0166) which is the original motivation of Bootstrapped DQN, but might mitigate the off-policy learning issue it allows all ensemble members (as opposed just one of them) to generate data within the period of an episode.
per-step basis. Note that this longer allows "temporally extended
As shown in Figure 17 switching ensemble members more frequently does not mitigate the curse of diversity. This is not surprising as it does not change the proportion of self- seff-generated data for each ensemble member in the replay buffer.
29

---

Published as a conference paper at ICLR 2024
Amidar
Kung Fu Master
Space Invaders
Chopper Command
Normalized Score
1.2
1.0
1.2
1.0
1.0
0.5
0.8
0.8
3 #Shared Layers
#Shared Layers
#Shared Layers
#Shured Layers
Voting Entropy
1.0
1.5
1.0
1.0
0.5
0.5
0.5
0.0
0.0
3
3
Shared Layers
#Shared Layers
#Shared Layens
#Shared Layers
Double DQN
Bootstrapped DQN (indiv.)
Bootstrapped DQN (agg-)
Figure 18: The effects of varying the number of shared layers when using replay buffer of 4M transitions. The top row shows Double DQN normalized scores. The bottom row shows the entropy of the normalized vote distributions. Error bars show 95% bootstrapped CIs over 5 seeds.
Amidar
Kung Fu Master
Spuce Invaders
Chopper Command
Episode Return
20000
3000
4000
1000
2000
10000
1000
2000
20% Passive Data Ratio (p)
0% 5%10%
20% Passive Data Ratio (p)
0% 5%110% 10%
20% Passive Data Ratio (p)
0% 5%%00%
20% Pussive Data Ratio (p)
0% 5%10%%
Active
Passive
Figure 19:
The effects of varying p the p%-tandem experiment. Error bars show 95% bootstrapped
Cls over 5 seeds.
D.3.5 LAYER SHARING EXPERIMENT WITH A LARGER REPLAY BUFFER
repeat the layer-sharing experiment in Section 3.3 but with a replay buffer of 4M transitions. This allows Bootstrapped DQN (agg.) to outperform Double DQN three games. However, the trade-off between the advantages and disadvantages of diversity remains: increase the number of shared layers and hence reduce diversity,
In Figure 18
The curse of diversity, i.e., the gap between Bootstrapped DQN (indiv.) and Double DQN reduces;
The performance gain get from majority voting, i.e. the gap between Bootstrapped DQN (agg.) and Bootstrapped DQN (indiv.), also reduces. An exception is Space Invaders, where the gap seems slightly increase when L increases from 0 to which might be related to certain properties of this game.
D.3.6 DIFFERENT LEVELS PASSIVITY THE P%-TANDEM EXPERIMENT
In Figure 19 we show the effect of varying in the p%-tandem experiment. As expected, increasing p (i.c., reducing the degree of passivity) reduces the performance gap between the active and passive agents.
30

---

Published as a conference paper at ICLR 2024
Amidar
ChopperConmaad
KungPu Master
SpaceInvaders
Episode Return
4000
20000
1000
2000
2000
10000
0
100
200
0
100
200
100
200
100 Frumes (x10*)
200
Frumes (x10%)
Frumes (x 10%)
Frumes (x10"*)
Boolstrapped DQN (aga.) (p 0.75)
Bootstrapped DQN (agg) (p 1.0)
Double DQN
Arnidar
ChopperCormmann
KungFuuMi FuMsttr
Spucelnvaders
Episode Retum
20000
1000
2000
2000
10000
0
0
100
200
0
100
200
100
200
100
200
Frames (x10*)
Framos (x 10")
Frames (x10")
Framos (x10%)
Bootstrapped DQN (indiv.) (p 0.755
Bootstrapped DQN (indiv.) 1p=110)
Dauble DQN
Figure 20: Effects of data bootstrapping, p refers to the probability of masking out a certain sample for each ensemble member. Performance with voting. (bottom) Performance without voting, All results are aggregated over seeds. Shaded areas show 95% bootstrapped CIs. The learning curves are smoothed with sliding window of 5 iterations.
D.3.7 DATA BOOTSTRAPPING
As mentioned in Section and Appendix A we do not use data bootstrapping Bootstrapped DQN (i.e., set the masking probability p 1.0), as Osband al. (2016) does not find it to be useful in Atari games. In Figure 20 we probe the effect of data bootstrapping on performance, with bootstrap masking probability p = 0.75 SreeOd al. (2016) for how p is used to approximate bootstrapping). As shown in the results, data bootstrapping damages performance on the environments we tested. This is not surprising because masking out samples essentially reduces the number of transitions each member has access to. It also effectively reduces the batch size. Besides, the fact that bootstrapping promotes diversity can exacerbate the curse of diversity. However, is difficult to attribute the precise cause of the damaged performance.
D.4 ADDITIONAL ANALYSIS OF CERL
D.4.1 ENSEMBLE SIZE ABLATION
In Figure 21 we test different ensemble sizes for CERL. With CERL, we see clear trend that increasing the ensemble size gives better performance, though it saturates at N 10 for two environments.
D.4.2 AN ALTERNATIVE DESIGN OF CERL
We consider the following alternative update rule for CERL:
(f - +
for = 1,..., N, forj=1,...
(2)
where Q is the target network for Qf and al ~ (-1s") The only difference between the original CERL is that the original CERL uses Q) on the right-hand side while here use Qf the right-hand side. Since the next action still comes from kj. this update rule is still trying to make the j-th head of ensemble i learn the value functions of ensemble member but in different way. Specifically, each auxiliary head uses itself to compute the TD target instead other ensemble members' main heads. We refer to this variant as CERL (self-target),
In Figure 22 we show the performance of this variant on 4 games. As can be seen, performs very well. However, in preliminary experiments in MuJoCo tasks with Ensemble SAC, this variant
31

---

Published as a conference paper at ICLR 2024
DDOM- Dnoo-Nomelizell Spone Soone
Kung
Space Chopper
1.0
1.00
0.8
1.0
0.6
0.50
0.5
5 Ensemble Size (2)
Euseebblc Size (N)
Ensemble Size (2)
Busembli Size (N)
Voting Entropy
1.5
1.0
1.0
0.5
0.5
10
20
10
20
10
$
20
10
5
Ensemble Size (N)
Ensemble Size (M)
Ensemble Size (N)
Ensemble Size (iz)(
Deuble DQN
Boulstrapped DQN (indiv.)
Bootstrapped DQN (agg.)
CERL (indis:)
CERL (ogg-)
Figure 21: Impact of ensemble size on Bootstrapped DQN and CERL. with or without majority voting. Shaded areas show 95% bootstrapped CIs over 5 seeds.
Amidar
ChopperCommand
KungFuMaser
Spacelnvaders
Episode Rotum
4000
20000
1000
5000
2000
0
100
200
0
100
200
100 Frumes (x10*)
200
100
200
Frumes (x10*)
Frumes (x 10*)
Frumes (x10*)
Bouistrapped DQN (agg))
Boutstrapped DQN (agg-)
(targe)))
Bootstrapped DQN (ugg) CERL
Double DQN
Amidar
ChopperCommand
KungFu Masttrr
SpaceInvaders
Episode Rotum
20000
1000
2000
2000
10000
0
0
100
200
0
100
200
100
200
100
200
Frumes (x10*)
Frames (x 10*)
Frumes (x10*)
Frumes (x10")
Boutstrapped DQN (indiv:)
Boutstrapped DQN (indiv:) CERL (self-turget)
Bootstrapped DQN (indiv.)
CERL
Double DQN
Figure 22: Performance of CERL (self-target) in 4 Atari games. (top) Performance with voting. (bottom) Performance without voting. Shaded areas show 95% bootstrapped Cls over 5 seeds. The learning curves are smoothed with a sliding window of 5 iterations.
leads instability in learning and we do not have good explanation at this moment. Also, this variant requires additional forward passes for SAC (note we need to evaluate ( for each and j).
D.4.3 COMBINING CERL WITH ENCODER SHARING
Even though our original motivation obtain the representation learning effect of encoder sharing without actually sharing the encoders, we comment that it is possible to use CERL with encoder sharing. This results hierarchical architecture where the network "branches" twice near the output. In Figure 2 apply CERL to Bootstrapped DQN (L = 3). Unfortunately, this provides almost no improvement over Bootstrapped DQN (L = 3). We conjecture that there are two reasons for this result. First, both methods shape the representations via jointly learning multiple value functions and their effects will likely overlap. Second, sharing layers reduces the diversity, and thus the learning signals from CERL less informative.
32

---

Published as a conference paper at ICLR 2024
Aggregated Policies (agg.)
Individual Policies (indiv.)
Human-Norne Hlmunn-Nrmalizze Normalized Score
1.0
1.0
1QM
0.0
0.0
0
100
200
0
100
200
Frames (x 10
Frames (x 10%)
Diversity
Voting Entropoy (Mean)
1.0
Double DQN
Bootstrapped DQN (l. =
Bootstrapped DQN (L
0.5
Bootstrapped DQN (L CERL
Bootstrupped DQN (L = 0) CERL
0.0
100
200
Frames (x10%)
Figure 23: Effect of combining CERL and encoder sharing. Shaded areas show 95% bootstrapped
Cls over 5 seeds.
D.4.4 NUMBER OF PARAMETERS
As mentioned in the main text, applying CERL Bootstrapped DQN (N 10) without network sharing increases the number of parameters by more than 5%. Further, these parameters are used for auxiliary tasks and thus not affect the capacity of the part of the network that actually predicts the value functions. Though it is extremely unlikely that CERL's improvements come from increased parameters, for completeness we also test Bootstrapped DQN without CERL but with more parameters. We do so by increasing the output size of the penultimate layer such that the number of increased parameters is roughly the same as that introduced by CERL. As shown in Figure 24 this has almost impact on performance.
D.4.5 OTHER REPRESENTATION LEARNING METHODS: A PRELIMINARY INVESTIGATION
The success of CERL suggests that representation learning in general may be promising for mitigating the curse of diversity. We perform preliminary investigation with MICo (Castro al., 2021) and multi-horizon auxiliary task (MH) (Fedus et al.) 2019)) in 55 Atari games. MICo is metric-based method that explicitly shapes the representations, while MH does so implicitly by learning value functions of different horizons
auxiliary task.
For the multi-horizoo auxiliary task implementation, we jointly learn K = 10 value functions with { where 7/4
2.(Hmax/KK) where Hmax 100 leading to ук 0.99. The architecture change involved is the same that for CERL. Only the heads that correspond to the longest horizon 7k are used for acting. For MICo, we follow the author implementation with 3 = 0.1. We search the MICo weight coefficient a in {0.01, 0.1, 0.5} on the four Atari games we used in the main text based on the performance of Bootstrapped DQN + MICo. This results in the final selection of a = 0.01.
We show the aggregate performance Bootstrapped DQN MH and Bootstrapped DQN + MICo in Figure 25 and Figure (eeespectiiel.. Per- Per-game results are summarized in Figure 27 and Figure 28 respectively. As these methods are also applicable to single-agent methods, we also show the performance of Double DQN MH and Double DQN MICo for completeness. As shown in
https://github..... http:///w...aaaaoooooooooooaaaae
33

---

Published as a conference paper at ICLR 2024
Amidar
ChopperConmaad
KungPu Maater
Spdcelnvaders
Episode Return
4000
20000
1000
2000
2000
10000
0
100
200
0
100
200
100
200
100 Frumes (x100)
200
Frumes (x10*)
Frumes (x 10*)
Frumes (x10")
Bootstrapped DQN (agg.)
Baotstrapped DQN (agg.) (more params)
Douhle DQN
Amidar
ChopperCommand
KungFuMaan FuMaster
Spacelnvaders
Episode Return
4000
20000
1000
2000
2000
10000
0
100
200
0
100
200
100
200
100
200
Frames (x10*)
Frames (x 10*)
Frames (x10*)
Framos (x10")
Bootstrapped DQN (indiv.)
Bootstrapped DQN (indiv:) (more params)
Double DQN
Figure 24: Bootstrapped DQN with more parameters. (top) Performance with voting. (bottom)
Performance without voting. Shaded areas show 95% bootstrapped CIs over 5 seeds. The learning curves are smoothed with a sliding window of 5 iterations.
Aggregate Policies (agg.)
Individual Policies (indiv.)
Diversity
Human-Normin Normalized Score
Human- Nermalizdd Score
Voring Entropoy (Mea))
1.0
1.0
IQM
IQM
0.5
0.5
0.5
0.0
0.0
0.0
100
200
100
200
0
100
200
Frames (x10*)
Frames (x10*)
Frames (x10*)
Double DQN
Double DQN MH
Bootstrapped DQN
Bootstrapped DQN MH
Figure 25: Comparison between Double DQN, Double DQN + MH, Bootstrapped DQN, and Bootstrapped DQN + MH in Atari. Results are aggregated over 55 games and 5 seeds. We show the performance of the agg- and indiy. versions of each ensemble algorithm in the top left and top middle plots respectively. Shaded areas show 95% bootstrapped Cls.
these results, these methods do not provide a clear improvement to Bootstrapped DQN (indiv.) and Bootstrapped DQN (agg.) as CERL does. Note that in the result we find that MICo damages the performance of Double DQN. This is likely because our hyperparameter setup (largely based on DQN Zoo (Quan & Ostrovski 2020). mentioned in Appendix C is very from those used in the original MICo paper, which Is based on the setup used Dopamine (Castro et al. 2018).
We emphasize that these preliminary investigations, and the results may vary lot based on the base algorithm, hyperparameters, and environments. A thorough analysis of what types of representations most suited for addressing the curse of diversity is left for future work.
34

---

Published as a conference paper at ICLR 2024
Aggreguie Policies (agg.)
Individual Policies (indiv.)
Diversity
Temam-Normaiiee Score
Cammm-Nornalizee Score
Voting Eatropoy (Mean)
1.0
1.0
1.0
IQM
IQM
0.5
0.5
0.5
0.0
0.0
0.0
100
Frames (x10")
Frames (x10')
(x10*)
Double DQN
Double
MICo
Bootstrapped DQN
Bootstrapped DQN MICo
Figure 26: Comparison between Double DQN, Double DQN MICo, Bootstrapped DQN, and Bootstrapped DQN + MICo in Atari. Results are aggregated over 55 games and 5 seeds. We show the performance of the agg. and indiv. versions of each ensemble algorithm in the top left and top middle plots respectively. Shaded areas show 95% bootstrapped Cls.
Bootstrapedd DQN MH (agg.)
10
Improvement over Bootstrapped DQN
10
0
-10
in Normalized Score
-10
-10
Bootstrapped DQN MH (indiv.)
00
1011
10
10"
-10'
Double Dunk
Video Pinbail
Atlants
Breskous
Up N Down Zaxxon
lime Plis
Jame mamessond shond
cam Kider
Bank Heiss
Ratte leature
Gravita vivitar
Porg
Firtal
Amitar
Astert ds
Ihyate Eve
Montezumna Freeway
Qhert
Assglui
Allen
Yors R evenge
Salars
Phogrnt
Bowling
Pasmann
Teanis
Froshiite
Berzerk
Scaques
Kangaroo
Road Runner
Kung Fu Master
Boxiing
Hero
Riverrad
Enduro
Skung
Name Thes Cenipedé
of War
Chapper Command
Star Gunner
Krall
Astorxx
V
Fishingoww
Turak
los
Fishing
Demon
Ms
Space
Crazy
Figure 27: Per-game improvements of Bootstrapped DQN + MH over Bootstrapped DQN in HNS.
with and without policy aggregation.
35

---

Published as a conference paper at ICLR 2024
Bootstrapped DQN MICo (agg.)
10²
Improvement over Bootstrapped DQN
10:
in Human- Human-Norman Normalized Score
-10
Rootstrapped DON MICo (indiv.)
10
10
10
-10
Double Dunk
Up K. Down Goplerr
Time Pilot
Star KLangaroo
lutankbam
Amuda
Ice Hockey
Enituro
Seaquest
sumesbond shond
Road Runner
Bank Helet
Pulall
Rowling
Yars Revengg
Name This Game
Private Eyye
Montezuma Revenggo
Freeway
inynders
Obert
Agerooids
Grew slar
Ms Pacman
Solari
diveraid
Bean Rider
Bezzerk
Crazy Clinn beber
Fishing Demby
Westure
CCentinesee
Skinng
Chopper Commani
Wizerd Of Wor
Robolant hoolank
Kmill
Boxing
steri
Phoonix
Breakout
Demon Attack
Video Pinball
Buitle
Spece
Kung
Figure 28: Per-game improvements of Bootstrapped DQN + MICo over Bootstrapped DQN in HNS,
with and without policy aggregation.
36


Reviews:
[{"review_id": "k6sAfHsOM0", "reviewer": "Reviewer_5Qz1", "summary": "This paper is a primarily empirical paper investigating what the authors term \"the curse of diversity\". This phenomenon is where\nindividual members of a data-sharing ensemble underperform relative to their single agent counterparts due the high proportion of off-policy data relative to each individual ensemble member and the agents' inability to learn effectively from such off-policy data.\n\nThe authors first demonstrate in the bootstrapped DQN setting (a common ensemble Q-learning method) that individual ensemble members perform poorly. Then, the authors run experiments to show the underperformance of an individual learner when there is a low self-generated to \"other-generated\" ratio in the replay buffer. This experiment, along with some reasoning, leads to the conclusion that the curse of diversity indeed exists.\n\nThey then investigate whether the curse of diversity can be ameliorated through larger replay buffers. The results are mixed, suggesting larger replay buffers may sometimes mitigate the curse, though largely do not. They then investigate ways to reduce the curse of diversity by directly reducing diversity in the ensemble, but find that it can adversely impact the underlying ensemble method.\n\nThey then introduce Cross-Ensemble Representation Learning (CERL), as a mechanism to mitigate the curse while preserving the benefits of the underlying ensemble method. CERL works by adding auxiliary tasks to ensemble members, having them learn value functions of other ensemble members and find that CERL mitigates the curse of diversity without adversely impacting performance.", "questions": "Question: Did you try different levels of passivity? Other than 10%?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Presentation - The paper is extremely clear and excellently written. The problem, motivation, and experiments are articulated very clearly. I like that they look introspectively at their own experiments and reason clearly about what can be inferred/concluded from their experiments without making unreasonable intellectual leaps.\n\nContributions:\n1. They show that perhaps the aggregation/majority voting aspect of ensembling methods may contribute to improved performance more than previously attributed. Previously, the intuition/hypothesis was more along the lines of ensembling methods inducing diversity in the data, which leads to higher-quality value estimates.\n2. They demonstrate that the curse of diversity exists, at least for ensemble Double DQN methods.\n3. They look at some natural ways to mitigate the curse of diversity and analyze their effectiveness.\n3. They start the steps towards mitigating the curse of diversity\n\nExperimentation: The experiments are reasonably thought out, and cover a wide range of environments (Atari/Mujoco).", "weaknesses": "I do not have any many major qualms with the paper, but I'll list a few thoughts.\n\nPerhaps this is out of scope of the paper, but I do feel it's difficult to draw conclusions about deep RL more broadly without investigating distributional RL. For example, this paper (https://openreview.net/pdf?id=ryeUg0VFwr) shows that distributional RL will likely do better with this off-policy data. It would be interesting to investigate the extent of this phenomena in the distributional setting. \n\nCERL does seem to scale poorly with the ensemble size, since if the ensemble size is N, there are N x N Q-functions. In practice, this does not seem like a major problem.\n\nFor the bar plots in Figure 2, it might be helpful to plot the top plot minus the bottom plot to highlight the point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TPcH3Eyss4", "reviewer": "Reviewer_wgEM", "summary": "The paper introduces a novel observation for ensemble-based exploration: each ensemble member has significantly worse performance than single-agent baselines, even when the aggregate policies outperform them. The authors demonstrate this in experiments with Double DQN and SAC on Atari and Mujoco environments. They show that increasing the replay buffer and sharing layers reduce the effect, but do not lead to better performance. The paper proposes a novel representation learning method (CERL), in which ensemble members predict each other's targets with additional network heads. Results indicate that CERL improves the performance individual ensemble members without loosing too much diversity of their predictions.", "questions": "- If one accepts the idea that ensembles are primarily for exploration, then the diversity of responses is actually a feature, not a bug. Given that few wrong decisions per episode can significantly reduce the performance of an individual ensemble member, why do the authors actually consider the poor individual performance an issue, in particular when the aggregate performance is better than the baseline (e.g. on Mujoco)?\n- How did the baseline (Double DQN and vanilla SAC) explore? If the baseline worked so well, doesn't this imply exploration is not very important in the tested environments? How would the analysis look like if it would concentrate on hard-exploration environments?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The reviewer liked the paper a lot. The main hypothesis makes sense and is substantiated in multiple experiments that show the effect nicely. The paper is well written and the figures are clearly readable. More detailed figures for individual environments are provided in the appendix, which is welcome to get an idea how trustworthy the aggregate performance measures are. The proposed method is not terribly innovative, but to the best knowledge of this reviewer novel. The discussion on other representation learning methods is nice, too.", "weaknesses": "While the main paper is very well written and the experiments appear quite thorough, the reviewer took issue with the way that some conclusions were presented. In particular the connection to exploration (which is in the title) ignores some major alternative explanations of the results. While the reviewer recommends to accept the paper, some phrases *need* to be changed, and some discussion needs to be added, to prevent the casual reader from misinterpreting the text and results. These are:\n\n1. \"First, it challenges some previous work’s claims that the improved performance of approaches such as Bootstrapped DQN in some tasks comes from improved exploration\" (p.3): as mentioned earlier in the paper, ensemble methods are widely used to drive exploration into unseen regions of the state-action space. However, the authors claim that \"Surprisingly, simply aggregating the learned policies at test-time provides a huge performance boost in many environments\" (p.3), and proceed to downplay the effect of exploration in favor of an explanation based on \"aggregation\". This is a problem in statements like \"the two cases[, DQN(indiv.) and Double DQN,] have access to the same amount of data and have the same network capacity\" (p.3), as the change in exploration means that the two algorithms are based on different data distribution and will therefore behave differently, irrespective of the on- and off-policy-ness of the data. This potential interaction between exploration (which part of the state-space is covered) and on-policy-ness (which member got to sample it) cannot be distinguished with the presented experiments, but the text reads as if the latter is the only obvious conclusion (see below).\n\n2. \"These results confirm our hypothesis regarding the cause of the observed performance degradation\" (p.5). This is dangerous abductive reasoning: experimental evidence that is consistent with your hypothesis does not make your hypothesis true (one can only falsify a hypothesis). This may sound like a nitpick, but the paper does not discuss the exploration effect of ensembles enough. For example, in Figure 3 (middle), the BootDQN (indiv.) curve is significantly above the Passive (10%) curve. If indeed Algo 2 and Algo 3 are exactly equivalent except for how often they sample the other (non-passive) ensemble members, and *only* this affects the performance due to off-policy sampling, then both curves (which have seen the same fraction of off-policy samples) should perform identical. The fact that they do not shows that there must be another effect at work here. This could be the exploration, but the experiments do not allow to distinguish these potential effects. To be fair, the reviewer also does not know how to distinguish exploration and on-policy-ness with the given setup, but another setup might be able to (e.g. with intrinsic-reward methods where the exploration can be separated from who acts). The presented analysis is strong enough for publication (a new setup is not needed), but these ambiguities must be discussed!\n\n3. While the reviewer liked most experiments (e.g. Figure 3 and 4 are great!), Figure 5 seems to test DQN with a very small replay buffer, where even the aggregated ensemble does not perform as well as Double DQN. This makes some conclusions suspect. For example, does the aggregated performance improves when more layers are shared? Or does the ensemble become indistinguishable from the baseline? If the authors would have picked a larger replay buffer (e.g. 4M), the performance would improve in the first and worsen in the second case. This would have an impact, as it shows that sharing layers in the ensemble reduces the positive effect it can have (e.g. by exploration).\n\n4. The main hypothesis of the paper is that \"the curse of diversity\" hurts individual member performance due the large number of off-policy samples. This makes sense, as an individual member can make 90% of all decisions correctly and still have poor performance, whereas a majority vote of the ensemble will agree with high certainty on the correct action. Ultimately this is the reason why BootDQN explores well. However, CERL seems to have almost as much diversity as the L=0 ensemble. CERL is a representation learning algorithm, which seems to be at odds with the main hypothesis. Contrast this with BootDQN which switches ensemble members mid-episode. If this algorithm would improve individual performance, the reviewer would accept it as a fix of the identified problem. With CERL presented as the \"solution\", the reviewer wonders: is the underlying problem is indeed \"diversity\"? Please discuss why representation learning fixes the off-policy-ness of the data (there are some attempts, but it is still vague). \n\nIn summary: there is a lot to love about the paper, but the authors need to reformulate some critical points (or convince the reviewer that those are not critical). Only under the condition that this happens, this reviewer recommends acceptance.  \n\n**Minor comments**\n\n- The figure captions should contain more information about the shown algorithms (ensemble size, replay buffer size, L, ...) to make them more self-contained for cross-reading.  \n- Ensembles are primarily used because they are good out-of-distribution detectors. An experiment measuring this would have been very welcome.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k38nr4jDmX", "reviewer": "Reviewer_H7rr", "summary": "This work studies the problem of off-policy-ness in ensemble learning. Motivated by the recent discovery in offline RL (i.e., the tandem paper), and empirical observations in ensemble learning methods of BootstrapDQN and naive ensemble SAC, the authors introduce a representation learning approach that considers the off-policy data in auxiliary tasks.\nExperiments on continuous control and discrete control demonstrate the effectiveness of the proposed method.\n\n\n----\n\nAfter rebuttal, I increased my overall rating from 5 to 6, soundness score from 2 to 3, and contribution score from 2 to 3.", "questions": "Please see the weakness section", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I like the way this paper is presented. It is clearly motivated by empirical discoveries, together with reasonings, and followed by solutions to the identified problems.\n\nThe authors made great efforts in conducting and presenting experiments. Results are reported in a statistically-identifiable way. I really appreciate it.", "weaknesses": "### On high-level Motivation:\n\nI’m lost in the motivation of using ensemble in **policy learning**. As has been demonstrated in [EDAC] and [REDQ], I acknowledge that using ensemble learning for the **value function** could lead to improved performance, as the value can be more accurate, with uncertainty. But what is the motivation for having **multiple policies** for ensemble (because they are sample generators, rather than learners). Should not those samplers aim at more efficiently decreasing the uncertainty in the value function?\n\nThis is a concern with the continuous setting. BootstrapDQN is purely a **value-based** method, and in discrete tasks, the epsilon-greedy exploration can be regarded as sampling from an off-policy uniform policy, this should not be a problem.\n\n### On BootstrapDQN:\n\nIn BootstrapDQN, different samples are assigned different weights to analog bootstrap sampling. It would not be surprising to see individual BootstrapDQN perform worse than DDQN. To verify the claim that *BootstrapDQN’s performance improvement is purely based on voting* made by the authors, a comparison between Bootstrap Sampling/ normal sampling; and a comparison between majority voting and \n\n\nFigure 2: The reported results of SAC can not match the open-source implementations. This might be due to the usage of a different size of buffer (200k). Could the authors please explain more about this?\n\n\n### On Performance:\n\nThe performance of CERL seems marginal in Figure 7. Also in Figure 7, BootstrapDQN seems not helpful at all.\n\n\n### On related work:\n\n\nThis work [https://arxiv.org/pdf/2209.07288.pdf] discussed the ensemble learning that trades off between exploration and exploitation using different reward values may also be worth a discussion.\n\nThis work [https://openreview.net/forum?id=NOApNZTiTNU] and [REDQ] discussed the usage of the ensemble in Q-learning. REDQ has been considered to be state-of-the-art since 2021.\n\n\nReferences:\n\n[EDAC] An, Gaon, et al. \"Uncertainty-based offline reinforcement learning with diversified q-ensemble.\" Advances in neural information processing systems 34 (2021): 7436-7447.\n\n[REDQ] Chen, Xinyue, et al. \"Randomized ensembled double q-learning: Learning fast without a model.\" arXiv preprint arXiv:2101.05982 (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xuwLNxN4gS", "reviewer": "Reviewer_ZXyY", "summary": "The paper proposes a new way of ensembling RL agents. N neural network value functions are trained, consisting of a convolutional encoder and a value head. Each encoder has a main value head, so we have N main value heads. For each encoder, N value heads are trained with temporal difference, each using as the target one of the main value heads. That gives us N^2 value heads in total. This improves representation learning and data utilization. There are extensive experiment showing that training ensembles of RL agents with other methods are hard, while the proposed method works well. There are experiments on atari with DQN and mujoco with SAC.", "questions": "\\-", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The observation that training ensembles of RL agents is hard is interesting and is analyzed well\n- The proposed method is novel and sound\n- The experiments are comprehensive and demonstrate the benefits of the method", "weaknesses": "No major weaknesses", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)