Maximizing Ensemble Diversity in Deep Reinforcement Learning
Authors: Hassam Sheikh
Mariano Phielipp
Ladislau Boloni
Venue: ICLR 2022 Poster (2022)
Keywords: Ensemble Based Reinforcement Learning
Ensemble Diversity

Abstract: Modern deep reinforcement learning (DRL) has been successful in solving a range of challenging sequential decision-making problems. Most of these algorithms use an ensemble of neural networks as their backbone structure and benefit from the diversity among the neural networks to achieve optimal results. Unfortunately, the members of the ensemble can converge to the same point either the parametric space or representation space during the training phase, therefore, losing all the leverage of an ensemble. In this paper, we describe Maximize Ensemble Diversity in Reinforcement Learning (MED-RL), a set of regularization methods inspired from the economics and consensus optimization to improve diversity in the ensemble-based deep reinforcement learning methods by encouraging inequality between the networks during training. We integrated MED-RL in five of the most common ensemble-based deep RL algorithms for both continuous and discrete control tasks and evaluated on six Mujoco environments and six Atari games. Our results show that MED-RL augmented algorithms outperform their un-regularized counterparts significantly and in some cases achieved more than 300$\%$ in performance gains.

Published as a conference paper at ICLR 2022
MAXIMIZING ENSEMBLE DIVERSITY IN DEEP REINFORCEMENT LEARNING
Hassam Ullah Sheikh
Mariano Phiclipp Intel Labs
Intel Labs
hassam. shei kh www.heeetonncoommmmmmm
mariano. i wph.elippeintee........ com
Ladislau Bölöni
Department of Computer Science University of Central Florida 1boloni (cs. w.f. edu
ABSTRACT
Modern deep reinforcement learning (DRL) has been successful in solving range of challenging sequential decision-making problems. Most of these algorithms use an ensemble of neural networks as their backbone structure and benefit from the diversity among the neural networks to achieve optimal results. Unfortunately, the members of the ensemble can converge to the same point either the parametric space or representation space during the training phase, therefore, losing all the leverage of an ensemble. In this paper, we describe Maximize Ensemble Diversity Reinforcement Learning (MED-RL), set of regularization methods inspired from the economics and consensus optimization to improve diversity in the ensemble- based deep reinforcement learning methods by encouraging inequality between the networks during training. We integrated MED-RL in five of the most common ensemble-based deep RI. algorithms for both continuous and discrete control tasks and evaluated on six Mujoco environments and six Atari games. Our results show that MED-RL augmented algorithms outperform their un-regularized counterparts significantly and in some cases achieved more than 300% in performance gains.
INTRODUCTION
Reinforcement learning (RL) agents trained with high capacity function approximators such deep neural networks have shown to solve complex sequential makiing problems, including the board games of Chess, GO and Shogi (Silver al. 2016 2017 2018), achieving super-human performance in video games (Miih et al. 2015 et al. 2019) and solving robotic manipulation tasks (Liu et al. 2021). Despite achieving these tremendous goals, modern deep reinforcement learning (DRL) algorithms have plethora of limitations. For example, it is well-known that DRL algorithms are sample-inefficee
and require stupendous amount of environment interactions to learn an optimal policy (Lukasz Kaiser et al. 2020). Additional problems encountered and exacerbates during training a DRL agent includes the overestimation bias that occurs while estimating the target values for Q-lcarning (Fujimoto et al.) 2018 Lan et al. 2020: Hado van Hasselt et al.) (0166), error propagation during Bellman backup Kummar et al.) 2019) and trade-off between exploration and
exploitation (Chen et al.) (017))
Recently, the use of ensemble has been a popular choice to address the above mentioned issues. These methods combine multiple neural networks model the value functions (and) the policy (Osband et al. 2016) Chen et al., 2017 et al. 2020 Lee et al.) 2020). For example, TD3 (Fujimoto et al. 2018) used two critics address the overestimation bias problem in continuous control problems while MaxminDQN (Lan et al.) 2020) provided a mechanism to use the cardinality of the ensemble to use as knob tune between over and under estimation in deep Q-learning. Similarly, Bootstrapped
DQN (Osband et al. 2016 Chen et al. 2017) used ensemble for effective exploration.
Partial work done while being Ph.D student at University of Central Florida

---

Published as a conference paper at ICLR 2022
The primary insight of this paper is that the performance of ensemble based methods is contingent on maintaining sufficient diversity between the neural networks of the ensemble. If the neural networks in the ensembles converge to common representation (we will show that this is the case in many scenarios), the performance of these approaches significantly degrades. We note that even with different representations, the Q-values will still converge towards a shared optimum, but they statistically less likely to follow the same learning trajectory elsewhere.
In this paper, we propose Maximize Ensemble Diversity in Reinforcement Learning (MED-RL), set of regularization methods inspired from the economics and consensus optimization to improve diversity and to prevent the collapse of the representations in the ensemble-based deep reinforcement learning methods by encouraging inequality between the networks during training. The objective of the regularizers is solely to keep the representations different, while still allowing the models to converge to the optimal Q-value. The motivation for the regularizers came from topic of income distribution in economic theory that provides rich source of mathematical formulations that measure inequality. While in economics, high inequality is seen as negative, in our case we used the inequality metrics to encourage diversity between the neural networks.
To summarize, our contributions are following:
1. We empirically show that high representation similarity between neural network based Q-functions leads to degradation in performance in ensemble based Q-learning methods.
2. To mitigate this, we propose five regularizers based inequality measures from economics theory and consensus optimization that maximize diversity between the neural networks in ensemble based reinforcement learning methods.
3. We integrated MED-RE RL in TD3 (Fujimoto et al. 2018) SAC (Haarnoja al. 2018 and REDQ (Chen et al. 202T) for continuous control tasks and in MaxminDQN Lan ct al. 2020) and EnsembleIDON (Anschel et al., 2017) for discrete control tasks and evaluated on six Mujoco environments and six Atari games. Our results show that MED-RL augmented algorithms outperform their un-regularized counterparts significantly and in some cases achieved more than 300% in performance gains and are up to 75% more sample-efficient
4. We also show that MED-RL augmented SAC is more sample-efficient than REDQ, an ensemble based method specifically designed for sample-efficiency, and can achieve similar performance to REDQ up to 50 times faster on wall-clock time.
2 RELATED WORK
Ensembles in Deep RL: Use of an ensemble of neural networks in Deep RL has been studied in
several recent studies for different purposes. In (Fujimoto et al. 2018 Anschel et al. 2017 [aan et al.) 2020) have used an ensemble to address the overestimation bias in deep Q-learning based methods For both continuous and discrete control tasks. Similarly, Bootstrapped DQN and extensions (Osband) et al. 2016 Chen et al. 2017) have leveraged ensemble of neural networks for efficient exploration. The problem of error propagation Bellman backup was addressed in (Kumar et al. 2019) using an ensemble of neural networks. Sample efficiency, notorious problem in RL has taken advantage from an ensemble (Chen et al., 2021). Recently, SUNRISE (Lee et al. 2020) proposed a unified framework for basedbleebaedd deep reinforcement learning.
Diversity in Ensembles: Diversity in neural network ensembles has been studied years before the resurgence of deep learning (Brown 2004). Even though diversity is an important topic in neural networks, most of the studies in this topic revolve around addressing problems in supervised learning settings. More recently there has been number of studies that have diversity in ensembles to measure and improve model uncertainty. Jain et al. (2020) have proposed a diversity regularizer to improve the uncertainty estimates in out-of-data distribution. Lee al. (2015) have used Multiple choice Learning to learn diverse Convolutional Neural Networks for image recognition.
Regularization in Reinforcement Learning: Regularization in reinforcement learning has been used to perform effective exploration and learning generalized policies. For instance. (Grau-Moyal regularization to optimize a prior action distribution for better
ct al. 2019) uses performance and exploration, (Cheng et al. 2019) regularizes the policy T(als) using control
2

---

Published as a conference paper at ICLR 2022
prior, (Galashov et al. 2019) uses temporal difference enror regularization to reduce variance in Generalized Advantage Estimation (Schulman et al., 2016). Generalization reinforcement learning refers the performance of the policy different enviromment compared to the training environment. For example, (Farebrother et al. 2018) studied the effect of T.² norm on DQN on generalization, (Tobin a (2077) studied generalization between simulations vs. the real world, (Pattanaik et al.) 2018) studied parameter variations and (Zhang et al. 2018) studied the effect of different random seeds in environment generation.
Diversity in Reinforcement Learning: Diversity in reinforcement learning is active area of re- search. (Pacchiano et al. 2020) uses Determinantal Point Processes to promote behavioral diver- sity, Luppu al. (2021) have used policy diversity to improve zero-shot coordination in multi-agent setting. (Tang et al. 2021) uses reward randomization for discovering diverse strategic policies in complex multi-agent games. In (Li al. (02T) proposed CDS that uses informationntheorette objective to maximize the mutual information between agents' identities and trajectories and en- courage diversity. More recently (An al. 202I) have used diversified Q-ensembles address overestimation in offline reinforcement learning.
Representation Similarity: Measuring similarity between the representations learned by different neural networks is active of research. For instance, (Raghu et al.) 2017) used Canonical Correlation Analysis (CCA) to measure the representation similarity. CCA find two basís matrices such that when original matrices are projected on these bases, the correlation is maximized. (Raghu iet al. 2017] [Mrouh et al. 2015) used truncated singular value decomposition on the activations to make it robust for perturbations. Other work such as (Li al. 20155) and (Wang al. 2018) studied the correlation between the neurons in the neural networks.
3 BACKGROUND
Reinforcement learning: We consider an agent as a Markov Decision Process (MDP) defined as a five element tuple P,,, y). where S is the state space, A is the action space, P S x A x S - [0, 1] are the state-action transition probabilities, r S x A x S + R is the reward mapping and y y 00, is the discount factor. At each time step l. the agent observes the state of the environment
8, E S and selects an action E A. The effect of the action triggers a transition to a new state the transition probabilities P. while the agent receives scalar reward R3 = r (st, a, $t+1). The goal of the agent is to learn policy π that maximizes the expectation of the discounted sum of future rewards.
8141 E S according
Representation Similarity Measure: Let X R"xpi denote matrix of activations of p1 neurons for n examples and Y E "p=xp= denote a matrix of activations of pa neurons for the same n examples. Furthermore, we consider Kij (r) and Laj (yt, yj) where and I two kernels.
Centered Kernel Alignment (CKA) (Krrbllth et al. 2019 Cortes et al. 2012 Cristianini et al. 2002) is method for comparing representations of neural networks, and identifying correspondences between layers, not only in the same network but also on different neural network architectures. CKA is normalized form of Schbert-Schmidt Independence Criterion (HSIC) (Gretton et al., 2005). Formally, CKA is defined as:
HSIC (K, L)
CKA (K, T))
V /HSIC K) HSIC (L, L)
HSIC a test statistic for determining whether two of variables are independent. The empirical estimator of HSIC is defined as:
HSIC (K,L)= (n - 1)22 tr (KHLH)
where is the centering matrix H,, = In 11T
3

---

Published as a conference paper at ICLR 2022
MaxminDoN
Episode 500
1.0
50
40
0.8
0.68
0.79
0.67
0.75
0.40
Layer
Average
0.4
10
~
0.68
0.79
0.50
0.76
0.17
0.2
0.0
-10
500
1000
1500
2000
2500
3000
Training Episodes
Layer
Hcatmap at point A
Episode 1000
Episode 2000
Episode 3000
1.0
1.0
1.0
0.00
0.19
0.05
0.00
0.8
0.8
0.8
0.55
0.91
0.92
0.96
0.00
0.79
0.85
0.87
0.38
0.26
0.57
0.97
0.94
0.93
0.00
Layer
Layer
0.6
0.45
0.68
0.96
0.96
0.96
0.17
2
0.80
0.90
0.89
0.49
0.20
0.70
0.97
0.94
0.93
0.09
-.22
0.2
0.2
0.55
0.39
0.31
0.00
0.0
0.0
Layer
Layer
Layer
Heatmap at point B
Heatmap at point C
Heatmap at point D
The training graph and CKA similarity heatmaps of MaxminDQN agent with neural networks. The letters on the plot show the time when CKA similarities were calculated. Heatmaps at and C' have relatively low CKA similarity and have relatively higher average return as compared to heatmaps at point B and D that have extremely high similarity across all the layers. See diagonal values from bottom left to top right.
Figure 1:
MAXIMIZE ENSEMBLE DIVERSITY IN REINFORCEMENT LEARNING
In this section, we propose MED-RL: Maximize Ensemble Diversity in Reinforcement Learning, set of regularizers inspired from the Economics and consensus optimization improve diversity and to prevent the collapse the representations the ensemble-based deep reinforcement learning methods by encouraging inequality between the networks during training. This section is organized as follows:
1. We empirically show that high representation similarity between neural network based Q-functions leads to degradation in performance in ensemble based Q-learning methods.
2. we present the Economics theory and consensus optimization inspired regularizers with their mathematical formulation.
4.1 EMPIRICAL EVIDENCE TO CORRELATE PERFORMANCE AND REPRESENTATION SIMILARITY
The work in this paper starts from the conjecture that high representation similarity between neural networks in an ensemble-bases Q-learning technique correlates to poor performance. empirically verify our hypothesis, we trained MaxminDON (Lan et al. 2020) agent with two neural networks on the Catcher environment (Qingfeng 2019) for about 3000 episodes (5 x 10 training steps) and calculated the CKA similarity with linear kernel after every 500 episodes. The training graph along with the CKA similarity heatmaps are shown in Figure T Notably at episode 500 (heatmap A) and episode 2000 (heatmap C), the representation similarity between neural networks is low but the average return is relatively high. In contrast, at episode 1000 (heatmap B) and episode 3000 (heatmap D) the representation similarity is highest but the average return lowest.
4

---

Published as a conference paper at ICLR 2022
4.2 REGULARIZATION FOR MAXIMIZING ENSEMBLE DIVERSITY
In order to maximize the ensemble diversity, we propose to regularize the training algorithm with an additional criteria that favors diversity between the ensembles. In the following, .N the number of neural networks in the ensemble, L is the L norm of the i-th neural network s parameters, é is the mean all the 2 norms and l is the list of all the T,² norms.
The first four metrics consider based inequality measures from economic theory. While in economics, inequality is usually considered something to be avoided, in our case we aim to increase inequality (and thus, ensemble diversity).
The Atkinson Index (Atkinson et al. 1970) measures income inequality and useful in identifying the end of the distribution that contributes the most towards the observed inequality. Formally, is defined as
for 0 S Eat +
A.
i=
(I)
for Eat
where fat is the inequality aversion parameter used tune the sensitivity of the measured change. When fat 0, the index is more sensitive to the changes at the upper end of the distribution, while it becomes sensitive towards the change at the lower end of the distribution when e,t approaches 1.
The Gini coeflicient (Allison) 1978) is statistical measure of the wealth distribution or income inequality among population and defined the half of the relative mean absolute difference:
E,l
G 1= ==1
2N
The Gini coefficient is more sensitive to deviation around the middle of the distribution than at the upper or lower part of the distribution.
The Theil index (Johnston 1969) measures redundancy, lack of diversity, isolation, segregation and income inequality among a population. Using the Theil index identical to measuring the redundancy in information theory, defined the maximum possible entropy of the data minus the observed entropy:
Tr
(3)
The variance of logarithms (Ok Foster 1997) is widely used measure of dispersion with natural links to wage distribution models. Formally, it is defined as:
1 N N
Vz(()
In & In)(
(4)
where g(f) is the geometric mean defined as (II 4)1/N.
The final regularization method we use is inspired from consensus optimization. In consensus method (Boyd et al.) (201T) T), a number of models are independently optimized with their own task- specific parameters, and the tasks communicate via penalty that encourages all the individual solutions to converge around a common value. Formally, it is defined as
M
0.1
(5)
Where is the mean of the parameters of all the neural networks and 6, represents the parameters of Vector throughout this paper. For
the i th neural network. We will refer this regularizer completeness, the algorithm shown Algorithm Notice that the regularization term appears with negative sign, the regularizers essentially inequality metrics that we want to maximize.
5

---

Published as a conference paper at ICLR 2022
4.3 TRAINING ALGORITHM
Using the regularization functions defined above, we can develop diverssty variants of the the ensemble based algorithms. The training technique is identical to the algorithms described (Lan) al. 2020: Anschel et al. 2017] Fujimoto al. 2018 Haarnoja et al. 2018: Chen et al. (02T), with a regularization term added to the loss of the Q-functions. The loss term for i-th Q-function with parameters v/ is: C (v)) = Ba,a.,e' [(0] (s,a) ()) XI (4,
where Y is the target value depending the algorithm, I is the regularizer of choice from the list above and À is the regularization weight. Notice that the regularization term appears with a negative sign, the regularizers essentially inequality metries that want to maximize. As reference the modified algorithm for MaxminDQN is shown in Algorithm
EXPERIMENTS
5.1 ISN'T RESAMPLING AND DIFFERENT INITIALIZATION OF WEIGHTS ENOUGH?
The most common question that comes to mind to address the diversity issue is why not initialize the neural networks with different weights and train each network with a different sample from the buffer? This approach has been thoroughly discussed in (Brown 2004) and have been shown to be ineffective. To re-iterate the findings in (Brown 2004), we performed a regression experiment in which we learnt sine function using two different three layered fully connected neural networks with 64 and 32 neurons in each hidden layer with ReLU. The neural networks were initialized using different weights and were trained using different batch sizes (512, 128) and learning rates (1e-1, le 1e-3). The Figure 2a shows the learnt functions while Figure representt their CKA similarity heatmap before and after training. The odd numbered layers represent pre-ReLU activations while the even numbered layers represent post-ReLU activations. It can be seen that before training, the CKA similarity between the two neural networks from layer and onward is relatively low and the output being 0% similar while after training, the trained networks have learnt highly similar representation while their output being 98% similar.
Before Training
After Training
0
0:14
0.50
0.21
0.27
0.00
0:30
0:30
0.28
0.15
0.88
0.8
CKA Simlarity
Layer
0.94
0.99
0.55
0.64
0.58
0.50
0.58
0.6
0.96
0.59
0.59
0.76
0.75
0.64
0.55
0:85
0.04
0.53
0.96
0.89
0.86
0.29
0..4
0.58
1.00
0.99
6.84
0.72
0.60
0.46
0.97
1.00
1.00
0:64
0.69
0.64
0.29
0.99
0.72
0.60
0.52
0.60
0:60
0.2
0.0
1
Layer
Layer
(a) Regression using two different (b) CKA similarity heatmap between different layers of the two neural neural networks
networks used for the regression experiment.
Figure 2: Left: Fitting sine function using two different neural network architectures. The upper function was approximated using 64 neurons in each hidden layer while the lower function used 32 neurons in each hidden layer. Right: Represents the CKA similarity heatmap between different layers of both neural networks before and after training. The right diagonal (bottom left to top right) measures representation similarity of the corresponding layers of both neural networks. The trained networks have learnt similar representations while their output was 98% similar. See diagonal values from bottom left to top right.
This example shows that neural networks can learn similar representation while trained on differ- batches. This observation is important because in MaxminDON and EnsembleDQN training,

---

Published as a conference paper at ICLR 2022
each neural network is trained separate batch from the replay buffer but still learns similar representations.
6030
12000
5000
Average Return
Average Return
5000
4030
3000
3000
SAC
6000
SAC
SAC MeanVectar
Mearieectar
4000
Meariiecter
Gini
1000
2000
1000
VOL
VOL
Environment Interactions (x10%)
Environment Interactions (x10f)
Environment interactions (x10F)
(a) Ant
(b) HalfCheetah
(c) Walker
50000
200000
Average Return
Average Return
.000
175000
150000
3000
122000
Heankacter
2000
Hearieector
1000
Ginl
Cinl
Gini
Thui
Ackinson
1000
Actincos
Thai
75000
Theel
Alkimaonn
VOL
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
Environment Interactions (x109)
Environment Interactions (x109)
Environment Interactions (x10%)
(d) Hopper
(e) Humanoid
Humanoid-Standup
Figure 3: Training curves and 95% confidence interval (shaded area) for the MED-RL augmented variants for SAC
12000
5000
Average Return
Average Return.
10000
4000
4000
3002
MearVaecer
TD3
6000
TD3
Reanteectrr
Average
3000
2000
MoanVetter
TD3
Ginl
Ginl
1000
2000
1.000
VOL
Environment Interactions (x10%)
Environment Interactions (x10%)
Environment Interactions (x10*)
(a) Ant
(b) HalfCheetah
(c) Walker
Average Return
3000
Average Return
1.00000
4000
2000
3000
Average
120000
2000
1000000
1.000
Ginl
Gint
Arkinows
1000
Arkinoon
Aldimson
Thai
Thei
Theil
VOL
60000
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
Environment Interactions (x10*)
Environment Interactions (x10*)
Environment Interactions (x10%)
(d) Hopper
(e) Humanoid
Humanoid-Standug
Figure 4: Training curves and 95% confidence interval (shaded area) for the MED-RL. augmented variants for TD3
7

---

Published as a conference paper at ICLR 2022
5.2 EXPERIMENTAL SETUP
Continuous control tasks: We evaluated MED-RL augmented continuous control algorithms such as TD3, SAC and REDQ on the Mujoco continuous control benchmark environments. We compared the results of MED-RL with un-regularized counterparts. We report the mean and standard deviation across five runs after IM timesteps on six complex environments: Cheetah, Walker, Hopper, Ant, Humanoid and Humanoid- Standup.. For REDQ, we evaluated on Cheetah, Walker, Hopper and Ant for 300K timesteps only.
5.3 COMPARATIVE EVALUATION
Continuous control tasks: Tables to show the average returns of evaluation roll-outs for all the continuous control methods. RL consistently improves the performance of SAC, TD3 and REDQ in all the experiments. Even though the focus of this work is maximize the average return, we find that MED-RL augmented algorithms are more sample-efficien than their un-regularized counterparts (see Figures and p. For example, it can be seen Figure 5bjthat baseline SAC reaches the average return of 10K in about 1M environment interactions while all MED-RL variants reach the same average return in approximately 250K environment interaction, therefore, being approximately 75% more sample efficient than the baseline SAC. This improvement in sample-efficiency can be noted in all nearly all the experiments except Aum-nnidd and TD3- 333-Wa experiments. The training plots for REDQ are shown in Appendix.
Table 1: Max Average Return for MED-RL SAC over trials of million time steps. Maximum value for each task is bolded. + corresponds to single standard deviation over trials
Environment
Baseline
MeanVector
Gini
Atkinson
Theil
VOL
HalfCheetah
10380.3 +: 681
12278.1 + 160.3
11691.2 + 715.6
12117.2 +: 304.7
12212.7 4: 2167
12339.5 +: 284.9
Ant
4802.4 = 605.1
6298.7 + 101.8
6047.8 167.4
6163.2 207.6
6091.5 222.3
5965.1 196.4
Пlopper
2882.5 1. 738.3
3604.3
3552.9 ::
3560.4 1:
1:
1:
Walker2d
3954,9 = 356.7
4525.7 = 340.9
4523.0 440.1
4659.8 253.0
4753.8 394.7
4653.4 391
Humanoid
4582.2 + 592.4
5389.1 42.0
$224.6 + 105.1
5275.1 + 40.4
5355.2 137.3
5311.7 + 49.1
Humanoid-
153633.2 2
177666.5
170592.6
164967.6
180268.1
179645.1
Standup
+ 8256.6
+ 30044.1
+ 29346.3
+ 19464.6
+ 33080.4
+ 29980.4
Table 2: Max Average Return for MED-RL TD3 over trials of 1 million time steps. Maximum value for each task is bolded. + corresponds to single standard deviation over trials
Enviromment
Baseline
MeanVecton
Gini
Atkinson
Theil
VOL
HallCheetah
9583.1 = 682.6
11539.4 +: 278.1
11477.9 405.2
11442.5 187.8
11232.7 323.6
11393,66 532.7
Ant
3829.1 = 675.7
4829.6 = 1036.9
4611.5 781.9
4565.7 = 908.4
4810.7 + 347.9
4881.1 = 831.6
Hoopper
2965.3 1: 423.5
3629.5
3582.3 = 153.9
62.7
Walker2d
4140.6 + 334.2
4396.3 837.5
4666.3 319.7
4652.7 310.0
4528.5 + 507.1
4630.0 +:2 406.
Humanoid
4347.4 = 456.2
5060.5 + 127.4
5048.8 + 199.3
5116.3 = 278.5
5096.1 98.1
5040.0 + 112.7
Uumamoid-
135176.6
160293.8
151123.1
154652.2
160481.5
146970.3
Standup
+ 7991.2
+ 19657.2
+ 12712.8
+ $607.7
+ 15229.6
13199.6
Table 3: Max Average Return for MED-RL REDQ over 5 trials of 300K time steps. Maximum value for each task is bolded. + corresponds to single standard deviation over trials
Environment
Baseline
Atkinson
VOL
HallCheetah
8368.3 1: 56.3
10067.9 1: 360.7
10234.4 1: 74.4
9926.9 1: 319.0
10161.6 1. 461.7
9664.8 1. 1975.2
Ant
3001.3 +: 2083.5
5446.8 + 1867
5801.7 + 42.3
5616.2 + 86.3
$885.6 6 + 181.0
5897.4 + 16.7
Hopper
2876.9 1: 584.7
3477.3 1: 43.6
3565.9 1: 40.9
3524.8 1: 2.8
3596.6 1: 72.1
3550.8 4 50.1
Walker2d
3722.3 52.6
4282.7 414.5
4217.1 + 150.6
4133.9 + 145.9
5028.4 205.6
4249.2 201.3
8

---

and Published as a conference paper at ICLR 2022
5.4 SAMPLE EFFICIENCY AND COMPUTE TIME
Tables] to show that MED-RL augmented continuous control algorithms outperform the baseline versions significantly and visual inspection of Figures Show that MED-RL augmented algorithms are more sample-efficient as well. But are they more sample-efficient than algorithms that are specifically designed for sample-efficiency such as REDQ? To answer this question, we took the bolded results from Table [ referred as MED-RL in this section, and evaluated the number of environment interactions and wall-clock time took for MED-RL to reach similar performance as that of baseline REDQ. As shown in Table MED-RL achieves similar performance to REDQ in 50% and 20% few environment interactions on Ant and HalfCheetah environment respectively and have significantly surpassed REDQ on 300K environment interactions. MED-RL does not only improve cumple-effcienge but significantly improves compute time. As shown in Table 4 MED-RL achieves similar performance to REDQ up 50 times faster on wall-clock time. Note it can be argued that REDQ can be parallelized to achieve faster wall-clock time but here we are only comparing standard sequential implementations but that will not address the sample-efficinny issue.
Table 4: Comparison of MED-RL augmented SAC with baseline REDQ on wall-clock time.
HalfCheetah
Hopper
Walker2d
REDQ average reward
8368.6 56.3
3001.3 2083.5
2876.9 584.7
3722.3 $2.6
Environment interactions taken by MED-RL to reach REDQ performance
232K + 36.5K
254K + 24K
152K + 40.9K
283.15K + 77.8K
Wall clock time REDQ (in mins)
1670.54 + 188.66
1853.17 + 66.26
1647.97 + 202.57
1690.27 + 245.1
Wall clack time taken by MED-RL reach REDQ performance (in mins)
49.61 + 9:25
43.81 + 9.42
32.23 10.3
65.2 16.2
MED-RL average reward after 300K environment interactions
9369.23 + 509.68
4095.38 + 433.24
3482.74 + 95.30
3487.751 874.14
Wall clack taken by
MED-RL. for 300K environment interactions
64.17 + 9.71
51.29 + 6.75
62,94 + 6.71
69.44 + 1.86
Discrete control tasks: We evaluated MED-RL augmented discrete control algorithms such as MaxminDQN and EnsembleDQQ on the PyGames (Qingfeng 2019) and MinAtar (Young & Tian 2019). We chose these environments to have fair comparison since used the source code provided by MaxminDQN authors. We reused all the hyperparameter settings from (Lan et al. 2020) except the number of neural networks, which we limited to four and trained each solution for five fixed seeds. The results on the discrete control tasks are shown in the Appendix.
6 CONCLUSION
In this paper, we proposed Maximize Ensemble Diversity in Reinforcement Learning (MED-RL), a set of regularization methods inspired from the economies and consensus optimization improve diversity in the ensemble-based deep reinforcement learning methods by encouraging inequality between the networks during training. We also empirically showed that high representation similarity between the networks of the ensemble could cause degradation in the performance. Our experiments have shown that MED-RL not only improves the average return of ensemble based reinforcement learning algorithms but can increase their sample-efficiency by approximately 75% when compared to their un-regularized counterparts. Additionally we have shown the SAC when augmented with MED-RL can outperform REDQ, algorithm specifically designed for sample-efficiency, in both sample-efficiency and compute time.
Acknowledgement: This work had been supported in part by the National Science Foundation under grant number CNS-1932300

---

Published as a conference paper at ICLR 2022
REFERENCES
Paul D. Allison. Measures of incquality, American Sociological Review, 43(6):865-880, 1978. ISSN 1224 URL httt::///www jstor. Stttor..rg//ttaae///0966..
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. arXiv preprint arXiv:2110.01548, 2021.
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-DQN: Variance reduction and stabilization for deep reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML-2017), pp. 176-185, 2017.
Anthony B Atkinson et al. On the measurement of inequality. Journal of economic theory, 2(3): 244-263, 1970.
Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 311):1-122, 2011.
Gavin Brown. Diversity Neural Network Ensembles. PhD thesis, University of Birmingham, United Kingdom, 2004. Winner, British Computer Society Distinguished Dissertation Award.
Richard Y Chen, Szymon Sidor, Pieter Abbecl, and John Schulman. Ucb exploration via g-ensembles. arXiv preprint / 770.00150, 2017.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q-learning: Learning fast without model. In International Conference on Learning Representations, 2021.
URL htt.ttttss//peeeeeeeee net.:iiwwwwieeeeeeeeeee..eeeeeee http:///e..pppaaeee..eeeeeeeee/r
ZmCt.dd
Richard Cheng, Abhinav Verma, Gabor Orosz, Swarat Chaudhuri, Yisong Yue, and Joel Burdick. Con- trol regularization for reduced variance reinforcement learning. arXiv preprint arXiv:195..0530
2019.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered aligmment. Journal of Machine Learning Research, 13(Mar):795-88) 2012.
Nello Cristianini, John Shawe-Tayie Taylor, Andre Elisseeff, and Jaz S Kandola. On kernel-target alignment. In Proceedings of the Advances in Neural Information Processing Systems (NIPS-2002), pp. 367- 373, 2002.
Jesse Farebrother, Marlos C. Machado, and Michael Bowling. Generalization and regularization in DQN, 2018.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 1802.09977 2018.
Alexandre Galashov, Siddhant Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojeiech Czamecki,, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry in KL-regularized RL. In Proceedings of the International Conference on Learning Representation (ICLR-2019), 2019.
Jordi Grau-Moya, Felix Leibfried, and Peter Vranex. Soft Q-learning with mutual-information regularization. In Proceedings of the International Conference Learning Representations (ICLR-2019), 2019. URL/https:
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. Algorithmic Learning Theory (ALT), pp. 63-77, 2005.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critie: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv: 1801.02800 2018.
Hado Hado van Hasselt, Arthur Guez, and David Silver. Deep Reinforcement Learning with Double Q-learning. In Proceeding of Conference on Artificial Intelligence (AAAI-2016), 2016.
10

---

Published as a conference paper at ICLR 2022
Siddhartha Jain, Ge Liu, Jonas Mueller, and David Kenneth Gifford. Maximizing overall diversity for improved uncertainty estimates in deep ensembles. In AAAI, 2020.
J. Johnston. H. Theil. Economics and Information Theory. The Economic Journal, 79(315):601-60.. 09 1969. ISSN 0013-0133. doi: 10.2307/2230366. URL httph:tt////oooo http://.0....//0......3.... 2230396
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Proceedings of the International Conference Machine Learning (ICML-2019), pp. 3519-3529, 09-15 Jun 2019.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- off-policy Q-learning via bootstrapping emror reduction. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS-2019), pp. 11761-11771, 2019.
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin Q-learning Controlling the estimation bias of Q-learning In Proceeding of the International Conference on Learning Represen- wwt//////ouaee dokg
tations (ICLR-2020), 2020. URL htt.h:tts////i http::::///peeeevvveeiee
Kimin Lee, Laskin Michael, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. arXiv preprint X7Xiv:2007.04938 2020.
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David J. Crandall, and Dhruv Batra. Why heads are better than one: Training a diverse ensemble of deep networks. ArXiv, 2015.
14,
Chenghao Li, Chengjie Wu, Tonghan Wang, Jun Yang, Qianchuan Zhao, and Chongjie Zhang. Cele- brating diversity in shared multi-agent reinforcement learning. arXiv preprint arXiv:2106.02199. 2021.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? In Proceedings of the International Workshop on Feature Extraction: Modern Questions and Challenges NIPS 2015, volume 44, pp. 196-212, 2015.
Rongrong Liu, Florent Nageotte, Philippe Zanne, Michel de Mathelin, and Birgitta Dresp-Langley. Deep reinforcement learning for the control of robotic manipulation: A focussed mini-review. Robotics, 10(1):22, Jan 2021. ISSN 2218-6581, doi: 10.3390/rbootes0000000000 URL http 7/dx doi. org/10 10. 0t10//11.3330/0000 3390/robo1cc11011000
Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. PMLR, 18-24 Jul 2021. URL
Iupu2la.ht html
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-5533 February 2015. ISSN 00280836.
Youssef Mroueh, Etienne Marcheret, and Vaibhava Goel. Multimodal retrieval with asymmetrically weighted truncated-svd canonical correlation analysis. CoRR, tts//5111.0666, 2015.
Efe A. Ok and James Foster. Lorenz Dominance and the Variance of Logarithms. Working Papers 97-22, C.V. Starr Center for Applied Economics, New York University, 1997. URL https
//ideas.. repec. otteee..rg/////ea/eee http:///wwvv//ttarre//-----2 html
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pp. 4026 2016.

---

Published as a conference paper at ICLR 2022
Aldo Pacchiano, Jack Parker-Holder Krzysztof Marcin Choromanski, and Stephen Roberts. Effective diversity in population based reinforcement learning. NeurIPS 2020 (spotlight), 2020. URL
attts:://ariiv org/ abs 2002.002... 00632
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Ro- bust deep reinforcement learning with adversarial attacks. In Proceedings of the International Conference on Autonomous Agents and MultiAgent Systems (AAMAS-2018) pp. 2040- 2042, 2018.
Lan Qingfeng. Gym compatible games for reinforcenment learning. https: http:://gtthb.ccom com/ glan3/ 3atp:///mmmmmmmsss/ games, 2019.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Proceedings of the Advances in Neural Information Processing Systems (NIPS-2014 pp. 6076-60866 2017.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR-2016), 2016.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lueas Baker, Matthe Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676)354-335 359, 2017.
David Silver, Thomas Hubert, Julian Schritwieeser, Ioannis Antonoglou. Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1140-1144, 2018.
Zhenggang Tang, Chao Yu, Boyuan Chen, Huazhe Xu, Xiaolong Wang. Fei Fang. Simon Du, Yu Wang, and Yi Wu. Discovering diverse multi-agent strategic behavior via reward randomization. arXiv preprint 10Xxx:2103.0456/: 2021.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation the real world. In Proceed- ings of the International Conference on Intelligent Robots and Systems (IROS-2017), pp. 23-30, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Ma- chine Learning Research, vandermaaten htm
2008. htRt:/tpp://ww jml r org/ www.oogggpppeeee httpps/pppsss///// v
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung. David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillierap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in StarCraft II using multi-agen reinforcement learning. Nature, 575(7782): 350-354, October 2019.
Liwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, and John Hopcroft. Towards understanding learning representations: To what extent do different neural networks learn the same representation. In Proceedings of the Advances in Neural Information Processing Systems (NIPS-2018). pp. 9584-9593, 2018
Kenny Young and Tian Tian. Minatar: An atari- insiiid testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv: 19000003177 2019.
12

---

Published as a conference paper at ICLR 2022
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study overfitting in deep reinforcement learning. arXiv preprint arXiv: 06893, 2018.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blažej Osiński, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski. Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model based reinforcement learning for atari. In International Conference on Learning Representations, 2020.
13

---

Published as a conference paper at ICLR 2022
Algorithm 1: MED-RL: MaxminDQN version
The differences between the baseline MaxminDON and MEDRL-MaxminDON are highlighted Initialize N Q-functions "Q''.....0 Q^) parameterized by {al1..... JNn}
Initialize empty replay buffer
Observe initial state
while Agent is interacting with the Environment do
Qmin (s, a) mi...( (*, ()) Va A
Choose action by 6-greedy based on Qmin
Take action a, observe r, s'
Store transition (s, a, r, s') in D
Select subset S from (1....,N) N} (e.g., randomly select one to update) for do
Sample random mini-batch of transitions (sD, aD, TD, s/p) from D
Get update target: yM <-
max.ea Qmin (8 a')
Generate list of L norms L = (e..l?.....,I....
Update Q' by minimizing Esp Exp.aD.D. (oi. (sD, - yum² -XI (x
end
s
end
14

---

Published as a conference paper at ICLR 2022
A TRAINING PLOTS OF REDQ
12000
6000
Average Return
Average Return
10000
8000
4000
6000
3000
REDQ MeanVector
REDQ
4000
MeanVector
2000
Gini
Gini
Atkinson
2000
Atkinson
1000
Thail
Thail
1
2
Environment Interactions (x10²)
Environment Interactions (x105)
(a) Ant
(b) HalfCheetah
5000
Average Return
Average Return
3000
4000
3000
2000
2000
MeanVector Gini
MeanVector
Gini
1000
Atkinson
Atkinson
VOL
VOL
1 Environment Interactions (x105)
0
Environment Interactions (x105)
(c) Walker
(d) Hopper
Figure 5: Training curves and 95% confidence interval (shaded area) for the augmented variants for REDQ together with baseline REDQ.
15

---

Published as a conference paper at ICLR 2022
B RESULTS ON DISCRETE CONTROL TASKS USING MAXMINDON
60
Average Retur
Averrage geg Return
60
50
Maxela
Atlinsen M-2
Maxmin N-3
Addinse M-3
40
Maxesin Hodd
Alkinsen Wod
Maxmin N+2
Abkinsoon No
Average
30
Maxmin M..
Maxmin Itee
Alkinson
Atkinean
10
-10
Environment interactions (x106)
Environment Interactions (x109)
Environment Interactions (x109)
Average Return
Average Return
60
Maxmin No
50
Macmin
Gini N
Gisi M
Gind Nt Ni.d
Girl N-
Maxmin Nez
Average
30
Masmin N-3
Maxmin Mes
20
10
-10
Environment Interactions (x10%)
Environment Interactions (x109)
Environment Interactions (x10f)
Hu-
Average Return
50
Average Return
50
Maxmin Ho4
Maxmin
MeanVector Ni.2
MeanVector Ni-3
Meanthector Rtd
40
Meaniiector Rod
Average
MeanVector
20
Masmin N+-3
Maxmin No2
30
Masmin
20
MeenVectar
-10
Environment Interactions (x10%)
Environment Interactions (x10%)
Environment Interactions (x10*)
80
70
Average Return
50
Average Return
60
Naxmin
60
Moxmin Hed
Maxmin
Thail Ni12
Maxesn Hos Macetn Hndd
Thai Red
20
Marmin Mes
Moumin N
Thail
Thail Ma-
Average
Maunein
20
Theil
10
Theil
-10
Environment Interactions (x10%)
Environment Interactions (x10%)
Environment Interactions (x10*)
80
Average Return
50
Average Return
Macmin
Maxmin Hod
VOL Row2
Maxmin Mas
Maumin N
46
20
Maxmin Nog Maceiin Hild
VOL Row2
Average
40
20
20
Rowd
10
-10
Environment interactions (x109)
Environment Interactions (x10%)
Environment Interactions (x10*)
Catcher
PixelCopter
Asterix
16

---

Published as a conference paper at ICLR 2022
20
Mazmin Re2
Nacein N-2
Mazamnn Ne2
Average Return
Maxein Mi-3
Average Return
Naxmin
Return
Atkinnon N-3
AtSinson
Attinesnn N-.3
Atkinsen Med
Atkinsen Mod
Atkinson fimd
10
40
20
Environment Interactions (x10%)
Environment Interactions (x10%)
Environment Interactions (x109)
Maxmin mee
80
Masmin Ne2
Maxmin Nea
Average Return
Average Return
2.5
600
Gind N-3
Giel N--
N--3
Giei Modd
Gied Mod
Giini Nodk
20
Environment Interactions (x10%)
Enviranment Interactions (x10%)
Environment Interactions (x109)
Naxmin Re2
80
Naxmin He2
250
Maxoin
Average Retumn
Average Return
Heaniwettor
100
Hean/eector
Mene/Vecter
Meaniructor No3
MeariVector N-3
MaunVacter
10
MeaniVector Hi.d
Neaniiector Hi.d
Average
MennVecttr Mi.d
150
20
1.00
Environment Interactions (x10%)
Environment Interactions (x10%)
Environment Interactions (x10%)
80
Maxmin
300
Maxoin
Average Return
Average Return
15
Thei No3
150
Thai Med
Thail
250
Theil fowd
Theil
Thail
Thail Midd
Theil Rind
1.0
30
150
20
50
Environment Interactions (x10*)
Environment Interactions (x10*)
Environment Interactions (x10*)
Average Return
15
Average Return
80
Maxmii Max
Masomin M
300
VOL
H
10
VOL Nod1
VOL Noot
Hid
200
VOL No-d
VOL Hod
40
150
20
50
Environment Interactions (x10*)
Environment Interactions (x10*)
Environment interactions (x10*)
Breakout
ScaQuest
SpaceInvader
Figure 6: All MaxminDON Results. Top Bottom: Logarithms
Atkinson, Gini, Mean Vector, Theil, Variance of
17

---

Published as a conference paper at ICLR 2022
B.1 RESULTS TABULAR FORM
Table 5: Max Average Return for MED-RL MaxminDQN with two neural networks PyGames and MinAtar environments. Maximum value for each task bolded. + corresponds to single standard deviation over trials.
Environment
Asterix
35.51 +: 8.81
59.20 +: 5.60
54.37 1: 8.32
52.61 :: 11.86
58.42 1: 13.00
56.82 d: 18.54
Catcher
54.72 + 4.73
51.52 + 2.31
57.88 + 1.80
58.00 + 1.00
58.58 + 0.34
56.16 + 2.96
Copter
37.83 +: 3.30
67.36 8.23
68.08 +: 4.28
65.20 1: 5.86
71.80 +: 5.34
71.85 8.64
12.75 d:
::
d:
3.75
Seaquest
14.60
28.75
29.03
SpaceInvader
135.63 + 14,10
289.64
66.49
350.00
108.27
252.02 + 43.85
261.86 +: 29.00
251 64 80.36
Table Max Average Return for MED-RL MaxminDQN with three neural networks PyGames and MinAtar environments. Maximum value for each task is bolded. + corresponds to a single standard deviation over trials.
Environment
Baseline
MeanVector
Gini
Atkinson
Theil
VOL
Asterix
28.34 +
54.17 8.68
45.85
47.54
51.71 9.59
$4.50 11.81
Breakout
8.10 1.49
8.78 + 1.38
11.44 1.34
10.78 2.36
12.54 + 3.46
12,66 +: 2.49
Catcher
44.07 1: 6.10
58.79 1: 0.17
58.81 0.20
$8.68 1: 0.17
58.55 1: 0.61
$7.97 d: 1.54
Copter
36.80 + 5,75
68.81 + 4.78
73.20 = 3.63
67.55 + 3.67
69.25 + 4.00
65.83 1:3::: 5,16
Seaquest
2.79
3:48
64.28
13.38
21.33
18.43
38.02 + 28.75
46.76 25.84
$9.59 37.29
Spacelavader
126.98 + 18.59
276.24 66.06
140.18
1.52
180.30 20.80
140.32 + 40.36
213.85 + 59.75
Table 7: Max Average Return for MED-RL MaxminDQN with four neural networks on PyGames and MinAtar environments. Maximum value for each task is bolded. + corresponds to single standard deviation over trials.
Environment
Baseline
Mean Vector
Gini
Atkinson
Theil
VOL.
1:
0.71
7.51 0.81
Catcher
53.17 :: 3.24
58.71 0.14
58.64 64 0.06
58.83 0.17
58.80 =:: 0.06
58.87 0.08
Copter
43.20 3.06
68.62 3.45
66.40 = 6.89
67.18 6.51
74.83 + 4.71
63.33 4.79
Seaquest
0.00 1: 0.00
10.26 1: 19.72
1.16 1: 0.75
15.67 1: 30.06
1.03 0.65
0.87 1: 0.45
Spacelwvader
74.69 # 10.36
91.09 1: 36.08
87.63 : 18.68
120.25
43.34
96.54 16.24
93.89 26.17
Figure 6 shows the training curves for the all the six environments and Tables 5 7 represent the results in the tabular, similar to our observations for the continuous control experiments, MED-RL augmented MaxminDON outperformed un-regularized MaxminDON significantly on average return and sample-efficiency metrics. Notably, in Seaquest and Spacelnvader environments where MED-RL MaxminDQN with two neural networks achieved 400% and 300% increase performance. Note that the baselines shown in Figure 6 are strong baselines. For example, in the MaxminDON paper, best performance baseline MaxminDQN on Asterix and SpaceInvader environments around 20 and 50 respectively while we have achieved an average reward of 35 and 135 respectively.
18

---

Published as a conference paper at ICLR 2022
C RESULTS ON DISCRETE CONTROL TASKS USING ENSEMBLEDQN
Drsemble Entemblle Alkinsen Atsinson No3 Atkinsen
Average Retur
Average Return
Average Return
50
Entamble N
Atlinsen M=2
Atdimsee W-3
30
Alkinsen Mad
Ensemble N=2
Eneamble
Ensembble
26
20
Atsenson
-10
Environment Interactions (x10f)
Environment Interactions (x10%)
Environment Interactions (x109)
Average Return
Average Return
50
Giod M--
Giei Mo
Gini Ne
Giel N-
20
Ensemble Row
Ennamble n3
Average
30
Rnsembbe
20
20
10
-10
Environment Interactions (x10%)
Environment Interactions (x109)
Environment Interactions (x10f)
80
60
Average Return
30
Average Return
100
Ensemble H-
Enseebble
MeanVector Ni.2
40
Average
40
Mean/Vector
Meanihector 2tw3
20
Ensembble N-2
Ensemble In33
Sncembbte
Ensemble Rud
Ensemble N-3
Ensembll Hud
20
30
MeunVecttr
Mesnteccte
-10
Environment Interactions (x10%)
Environment Interactions (x10%)
Environment Interactions (x10*)
80
Average Return
50
Average Return
100
60
Ensamble Hird
Thei Mez
Thail Ni-
Thail
20
Ensembin 2222
Ensemble tt
Encombble N
Ensembll tu3
Average
Ennamble
Eneamble
-10
Environment Interactions (x10*)
Environment Interactions (x10%)
Environment Interactions (x10%)
Average Return
km
Average Return
Ensemn bis Rork
Ensemblle H-
VOL Ninz
Roud
Entambie 2
40
Cneamble N
Average
Rowd
Ensembie Mas
Enzemble
Ensemble
Ensemblee tus
20
VOL Rond
VOL Rood
-10
Environment interactions (x10%)
Environment Interactions (x10%)
Environment Interactions (x10*)
Catcher
PixeICopter
Asterix
19

---

Published as a conference paper at ICLR 2022
Ensemble R=
Ensemble N
Average Return
25
ewerege Return
Ensemblle
Average Return
Ensemble
AtSinson
Attinesnn Ni.3
a0
Atkinsen Med
Atkinson fimd
1.
Encemble
Ervemble Ne4
20
artineon Ni-2
Alkiroon Ren
Atklneon
Environment Interactions (x10%)
Environment Interactions (x20%)
Environment Interactions (x109)
50
Ensembble Ned
80
Ersembble We
Ensemble N
Average Return
Ensemble
Average Return
Ensemble
Ensemble
200
Ginl Ni-3
Giel N--
Giel Rood
40
Gied Mod
Average
150
Giini Nodd
20
200
20
Environment Interactions (x109)
Enviranment Interactions (x10%)
Environment Interactions (x109)
Insemble Riz
Ensamble
Average Return
Average Return
40
20
MenaVeecter
15
Meanivector Hi-d
MeanVVecto N-3
200
MaunVacter
MennViactor N
Average
Ensemble R--3
Insemble Riaa
20
MeanVector Maz
MeanVactar
MeenVecter
Environment Interactions (x10%)
Environment Interactions (x10%)
Environment Interactions (x10%)
Ensemble
Ensemblle
300
Ensemble
Average Return
30
Average Return
M.
Thail
Theil
Thail Ne3
Thai Med
Thail fedd
Thail Ni-d
Thail Midd
Theil Nind
Environment Interactions (x10*)
Environment Interactions (x10*)
Environment Interactions (x10*)
Average Retum
40
Ensembble Nez
Enewmble
Average Return
Brsembbl
300
Ensemble
6D
VOL Noo
2
VOL food
200
VOL Hod
VOL Rod
VOL Rood
VOL Hedd
Average
220
100
20
Environment Interactions (x10*)
Environment Interactions (x10*)
Environment Interactions (x10*)
Breakout
ScaQuest
SpaceInvader
Figure 7: All EnsembleDQN Results. Top to Bottom: of Logarithms
Atkinson, Gini, Mean Vector, Theil, Variance
20

---

Published as a conference paper at ICLR 2022
C.I RESULTS TABULAR FORM
Table 8: Max Average Return for MED-RL EnsembleDQN with two neural networks PyGames and MinAtar environments. Maximum value for each task bolded. corresponds to single standard deviation over trials.
Environment
Baseline
Atkinson
Asterix
31.06 4.19
43.07 1: 4,96
44.49 1: 2.61
47,52 1: 11.76
46.05 1: 5.37
48.21 12.68
Catcher
44.06 4.33
58.29 + 0.70
57.29 2.00
58.59 + 0.25
55.87 # 5.94
56.03 + 3.64
Copter
31.03 8.83
64.50 +: 4,93
69.28 6.54
67.45 7.81
64.66 12.40
64.03 4: 2.37
20.91
10.35
d:
Seaquest
44.17
SpaceInvader
81.97 + 15.62
196.34 =: 51.71
213.32
38.77
182.58
31.34
190.08 + 27.03
217.76 + 23.80
Table 9: Max Average Return for MED-RL EnsembleDQN with three neural networks on PyGames and MinAtar environments. Maximum value for each task is bolded. + corresponds to a single standard deviation over trials.
Environment
Baseline
VeanVector
Gini
Atkinson
Theil
VOL
Asterix
35.32 + 10.66
51.32 13.39
47.56 12:45
49.91 = 8.05
57.92 : 8.50
49.37 9.78
Breakout
19.51 2.87
18.74 + 3.13
25.52 + 2.81
26.30 = 5.63
31.15 8.19
24.14 + 6.66
Catcher
37.07 5.24
58.80 1: 0.17
57.95 1: 1.16
58.65 1: 0.33
58.30 1: 0.80
58.83 1. 0.06
Copter
30.01 4.92
73.60 + 4.38
67.22 : 8.10
66.44 = 0.84
68.12 = 5.93
69.81 + 7.48
Seaquest
19.99 3.60
44.55 25.23
48.13
25.68
38.31 + 6.98
38.89 + 14.30
37.84 + 3:41
Spacolavader
86.33 + 12.05
187.88 + 42.91
168.60
3.60
221.38 8.48
158.65 10.45
170.43 + 37.19
Table 10: Max Average Return MED-RL EnsembleDQN with four neural networks on PyGames and MinAtar environments. Maximum value for each task is bolded. + corresponds to single standard deviation over trials.
Environment
Baseline
MeanVector
Gini
Atkinson
Theil
VOL
+:
17.85 1.82
2.87
22.71 1:
Catcher
47.77 + 6.62
58.74 0.22
58.33 081
$2.44 5.33
58.24
1.11
58.23 0.81
Copter
39.29 + 9.75
75.37 + 2.35
67.63
6.01
67.73
5.21
75.57 = 5.79
70.16 3,58
Seaquest
15.46 1. 8.11
37.00 1: 7.16
48.18
33.34
58.95
12.31
23.59
1: 12.35
37.60 6.73
Spucelavader
106.76 4: 32.14
269.96 + 71.46
227.60
20.75
218.65 :: 54.32
239.84 : 54.58
227.46 = 63.46
Figure shows the training curves for the all the six environments and Tables to represent the results in the tabular, similar to our observations for the continuous control experiments. MED-RL augmented EnsembleDON outperformed un-regularized EnsembleDQN significantly on average return and sample-efficieng metrics. Notably, in Copter, SeaQuest and SpaceInvader environments where MED-RL EnsembleDQN with two neural networks achieved 200%, 350% and 250% increase in performance.
21

---

Published as a conference paper at ICLR 2022
D T-SNE VISUALIZATIONS
To visualize the impact of the regularization, Figures[andd and t-SNE (van der Maaten & Hinton 2008) visualization of the activations of the last layer the trained networks. Figure show the network trained for the Catcher environment, while Figures (8b and 9 the network trained for the PixelCopter environment. The upper row of the figure shows the original, unregularized models, while the lower row a regularized version. For all combinations, we find that the activations from the original MaxminDQN and EnsembleDQN versions do not show any obvious pattern, while the regularized ones show distinct clusters. An additional benefit of t-SNE visualizations over CKA similarity heatmaps that the CKA similarity heatmaps are useful to show representation similarity between two neural networks, but they become counter intuitive as the number of neural networks increases.
Baseline MaxminDON
Baseline EnsembieDQN
Daseline MaxminDON
Baseline EnsembleD@N
McanVector MaxminDQN
EnsembleDO@N
Alkinson MaxminDDN
VOL EnsembleDON
Catcher
(b) PixelCopter
Figure 8: Clustering last layer activations from Catcher and PixelCopter after processing them with t-SNE to map them in 2D. The regularized variants have visible clusters while the baseline Ma and EnsembleDQN activations are mixed together with no visible pattern.
Baseline MaxminDQN
Baseline EnsembleDQN
Theil-MaxminDQN
Theil-EnsemlleDON
Figure 9: Clustering last layer activations from PixelCopter after processing them witht-SNE to map
them in 2D
22

---

Published as a conference paper at ICLR 2022
E PLOTTING THE GINI INEQUALITY
We measured the 2 norm inequality of the baseline MaxminDON and EnsembleDQN along with their regularized versions. We trained baseline MaxminDQN and EnsembleDQN with two neural networks along with their Gini index versions with regularization weight of 10 on the PixelCopter environment on a fixed seed. Figure 10rree100ppp represenns the L norm inequality of the experiments along their average return during training. Notably, despite each neural network being trained on a different batch, the 1,2 norm of the baseline MaxminDQN and EnsembleDQN are quite similar while the 1,? norm of the regularized MaxminDQN and EnsembleDQN have high inequality.
80
Maxmin
Ensemble
0.25
Average Return
Gini-Maxmin
Gini Index
0.20
60
Gini-Ensemble
Maxmin
Ensemble
40
Gini Mani-Maxiin
Ginl-En EeilEEnsembee
0.10
20
0.05
0.00
1.5 Training Episodes (x106)
0.5
2
Training Episodes (x106)
Figure 10: Left: Plot representing the L norm incquality between the two neural networks using Gini index trained on PixelCopter environment. Right: Plot representing the average return during training.
23

---

Published as a conference paper at ICLR 2022
F IMPLEMENTATION DETAILS AND HYPERPARAMETERS
For our implementation of MaxminDQN and EnsembleDQN, we used the code provided by the MaxminDQN authors that has implementations of different DQN based methods For the baseline experiments, we used most of the hyperparameter settings provided in the configuration files by the authors except the number of ensembles which limited to four.
Table 11: Hyperparameters for discrete control tasks
Hyperparameter
Value le [le 3 le
Target Weight
Actor Learning Rate
4
Regularization Weight
le
1e le-
1e
Replay Buffer
l
Batch Size
32
Exploration Steps
5000
Optimizer
Adam
Table 12: Hyperparameters for continuous control tasks
Hyperparameter
Value
Target Weight 7
1
Actor Learning Rate
1e 4 3e 5]
Critic Learning Rate
[le 4 3e $]
Replay Buffer
l
Batch Size
[256]
Exploration Steps
25000
Optimizer
Adam
Hidden Layer Size
256
Number of crities (REDQ)
10
Regularization Weight
1e-6 6
24

---

Published as a conference paper at ICLR 2022
F.1 COMPUTING INFRASTRUCTURE
All the experiments were performed on Kubernetes managed cluster with Nvidia V100 GPUs and Intel Skylake CPUs. Each experiment was run an individual Kubernetes job with 11 CPUs, 16GB of RAM and GPU (if needed). This configuration allowed us to run experiments without any interference from other applications which was important to accurately measure the wall-clock time.
Even though the focus of this paper was empirical testing, there are several different research questions that needs to addressed, for example how we select the best regularizer for particular environment ensemble. Another interesting line of research is to study how different inequality distributions effect the diversity of the ensembles.
Future Work:
algorithm with some N numbers of networks in the
25


Reviews:
[{"review_id": "f7tGEuQyHYq", "reviewer": "Reviewer_6miY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strong Points of the Paper\n\n- S1. It is nice that one can gain a considerable sample efficiency and performance improvement by such a simple technique.\n- S2. It is intriguing that the performance of ensemble-based deep RL methods highly (negatively) correlates with the representation similarly among ensembles of neural networks.\n- S3. I highly appreciate that the paper provides many experimental results with different regularizers, baseline algorithms, and the number of ensembles.\n\n# Weak Points of the Paper\n\nI give detailed comments about these points in the next section.\n\n- W1. From Figure 1, I am not so convinced that the performance of ensemble-based deep RL methods highly (negatively) correlates with the representation similarity.\n- W2. It is unclear how hyper-parameters and seeds are chosen.\n- W3. Some comparison seems to be unfair.\n\n# Detailed Comments\n\nAs W1 above, what is meant by \"high\" and \"low\" CKA similarity? In heatmaps at point A and C, (near-)diagonal elements have low values, but other elements typically have higher values. In contrast, heatmaps at point B and D show the opposite tendency, and it is not so convincing to me that there is a negative correlation between performance and layer x-y similarity. Furthermore, only showing results at 4 points is not convincing to me. I trust the authors, but it is known that there is an implicit bias that causes cherry-picking without any intention. Can you plot curves of layer x-y similarity along with the performance?\n\nAs for W2 above, how did you choose hyperparameters and seeds? There are some places where the authors say \"seeds are fixed\". However, does this mean that you fixed random seeds, ran experiments with different hyperparameters, and pick up the best results? If you did this, there may be a maximization bias. In addition, running experiments with only 5 seeds is probably insufficient (Henderson et al. 2018). Please run experiments with more random seeds.\n\nAs for W3, it seems to me that REDQ's performance in this paper is significantly lower than that of original paper. Since the authors of REDQ's paper opensourced their code [here](https://github.com/watchernyu/REDQ), I don't think reporting re-implemeted REDQ's result with a lower performance is fair. (Or you use it, and I just missed that?)\n\n# Questions\n\n- Q1. As for proposed indices, it seems to me that you can easily maximize some (maybe all?) of them, especially when ReLU is used. For example, in case of Gini coefficient, you can multiply weights of the second to the last layer by a huge constant, and then, multiply the weights of the last layer by a small constant. By choosing the constants carefully, you can get as high $L^2$ norm as you like without affecting the output and causing a high Gini index. Why doesn't this occur? Maybe I am missing something?\n- Q2. Why don't you simply minimize the sum of CKA? It is possible to backpropagate through it, right? Figure 1 indicates that low CKA implies high performance, so it is a very natural idea to minimize the sum of CKA.\n\n# Minor Comment\n\nWould you replace Figure 2(a) with a lighter one? It takes a long time to load the figure on my PC, and I can't print out the PDF probably because of it. I guess it is because the figure is a vector format, and there are too many points in the figure. Maybe replacing it with JPG or PNG would resolve this issue.\n\n# References\n\nDeep Reinforcement Learning that Matters; Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger. AAAI 2018\n", "summary_of_the_paper": "The paper considers a problem of ensemble-based deep RL methods that ensembles of critic networks converge to the same point in the representation space. To address it, the paper proposes a regularization technique that forces representations of a critic network to be dissimilar from those of other critic networks in the ensemble. It is empirically shown that this regularization technique improves the sample efficiency and asymptotic performance of baseline algorithms.", "summary_of_the_review": "Figure 1, which is intended to demonstrates the negative correlation between representation similarity and performance, is not convincing to me. Furthermore, the experiment conditions seem to be insufficient or not explained well.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "T_6FwbduQQ2", "reviewer": "Reviewer_TfGq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths: \n+ A simple but general regularization method to improve the performance of Q-function ensemble-based RL methods is proposed. \nAs I mentioned in the summary of the paper, a method to regularize the similarity of the parameters of the Q-functions is proposed in the paper. \nThe similarity of parameters between Q functions is evaluated based on five measures (e.g. Gini coefficients), and then added to the loss for the Q functions. \nSince we only need to add a similarity score to the loss, the proposed method can be easily introduced into the existing ensemble-based RL methods. \n+ The proposed regularization method is shown to be effective in various RL methods and environments.  \nThe proposed regularization method is shown to be effective in five different RL methods (SAC, TD3, REDQ, Maxmin-DQN, and Ensemble-DQN). \nIn addition, the regularization method is evaluated in continuous control (6 Mujoco envs.) and discrete control (6 Atari envs.).  \n\n\n\nweaknesses: \n\n- There is no theoretical justification for introducing the regularization method based on the similarity measures.  \n  - For example, it is not clear if Q-functions converge to the optimal one (under reasonable assumptions) when they are trained with the regularized method.  \n  - Also, there is no theoretical discussion of how the use of similarity measures affect overall performance improvement.  \n  - Nevertheless, I regard the study presented in the paper as an empirical one, and thus do not think that the lack of theoretical justification is a severe weakness of the paper. \n\n- The quality of the presentation is not sufficiently good: \n  - Connections among the sections in the paper are somewhat unclear.  \n    - For example, in Sections 3 and 4.1, the similarity of the Q functions (neural nets) is measured by the similarity of their activation units' outputs. However, in Section 4.2 and later, the similarity of Q functions is measured by the similarity of their parameter values. In sum, the criterion used for evaluating similarity differs among the sections.  \n    - In addition, Algorithm 1 in Section 4 describes the Maxmin-DQN variant that uses the proposed regularization method. However, this algorithm (and Maxmin-DQN) do not appear in the later sections (except Appendix). \n\n  - Experiment setups are not clearly explained. In particular, the setup in Section 5.4 is very unclear, e.g.,:  \n    - Steps taken by MED-RL to reach REDQ performance:  How this number of steps is counted? How MED-RL's performance is evaluated? (is it MED-RL's return in a single test episode? or its average return over multiple test episodes?)  \n    - Wall clock time REDQ (in mins): Is this the time for REDQ to complete 300k interactions with an environment? If so, what if REDQ's performance converges before 300k interactions? \n    - REDQ baseline: Is the REDQ's learning curve the one shown in Figure 5? If so, why is it much worse than the one shown in the original REDQ paper?  \n\nMinor comment:  \nThe figures in the pdf of the paper (e.g. Figures 3 and 4) take a very long time to be rendered.\nThis is probably because all the data points in the graphs are output in vector format.\nIf you want to put graphs with many data points on the same page, it is better to rasterize the graph. \n\n", "summary_of_the_paper": "In this paper, it is proposed to regularize the parameters of each Q function in ensemble so that they are not similar to each other. Through various experiments, it is shown that RL methods that use ensembles of Q-functions (e.g., REDQ, Minmax-DQN, etc) can improve its sample efficiency by using the regularization. ", "summary_of_the_review": "I'm leaning to recommend reject. \nI acknowledge the significance of the finding that it is useful to keep the divergence among the Q-function. \nI also acknowledge the authors' effort to demonstrate the usefulness of the regularization method in various RL methods and environments. \nHowever, the clarity of the current version of the paper does not meet the threshold for publication, and a non-trivial revision is needed. \n\n----- Update after author's revision 20211125 ------------------------------  \nThe authors have improved the explanation of experimental setups, but the connections between the sections still have not been sufficiently improved. \nIn particular, some of the other reviewers are concerned about the mismatch between the similarity criterion (CKA) used in Section 4.1 and the similarity criterion used in later sections. \nI have seen their discussions with the authors, and think that their concern about the mismatch is not sufficiently resolved. \nI think that replacing the analysis based on CKA with an analysis using criteria based on parameter values (e.g., equation (4)) would make the discussion in the paper more consistent.  \n\nOverall, I acknowledge that the paper was improved to some extent in the revision, and thus slightly improve the score (WR->WA). \nHowever, as mentioned above, I have still concerns about the clarity of the paper and cannot strongly champion the paper to accept. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NCXxbJZXX_P", "reviewer": "Reviewer_i4M1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe claim in abstract that “members of the ensemble can converge to the same point either the parametric space or representation space during the training phase “. This is not true for DRL. Neural networks converge to different solutions given the initialization is different and multiple local minima. \n\nQuestions:\n1. Why these environments were chosen? \n\n2. I get your Sec. 4.1. However, this is far from compelling. The claim of your conjecture is pretty big and here you have a small experiment about only one algorithm at only two time points. \n\n3. In Alg 1, what is I()? Is it a general function that you have a few alternatives in Sec 4?\n\n4. Sec 5.1: how do you initialize the networks in reaching the conclusion that “each neural network is trained on a separate batch from the replay buffer but still learns similar representations. “. Did you try another initialization method? How come the cross-diagonal of the after-training not exactly being ones?  Is the CKA non-symmetric? Why do you say the similarity is high? 0.15 between layer 6 and 7 is pretty low. \n\nEnsembles in Deep RL: Most the cited papers used ensembles for the critic (Q function). There is also ACE algorithm that uses ensembles from actors. Building an ensemble from actors can help agents explore at the option level. \n\nACE: An Actor Ensemble Algorithm for Continuous Control with Tree Search\nhttps://arxiv.org/abs/1811.02696\n\nThe results are not clearly separatable. Many lines are hard to see their statistical importance especially only 5 runs were performed. \n\nMinor:\nLan Qingfeng. Gym compatible games for reinforcenment learning. \nThe author name for this paper reversed first name and last name. \n", "summary_of_the_paper": "MED-RL\nThis paper studies foster diversity in ensemble of DRL networks by regularization methods. The paper is an empirical one and compared five ensemble methods with and without their diversity algorithm in six Mujoco and six Atari games, and showed some results. \nThe algorithm proposed is a modification of MaxMinDQN by Lan et.al. with a regularization. ", "summary_of_the_review": "The paper had a claim that initialization of different networks is not effective in proving diversity. This is not well supported. It can give a wrong message to the literature with the small and insufficient studies. this is my main concern.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KlPqJ6iEkTk", "reviewer": "Reviewer_a9sA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is clearly written and easy to follow. I'm always happy to see simple effective techniques that can substantially improve the performances of existing methods in challenging testbeds, which looks promising to me. \n\n**Question**\nMy biggest question is that all the criteria are evaluated w.r.t. the l2 norm of the whole network parameters, which is very counter-intuitive to me. Note that the paper simply uses the CKA metric to evaluate layer similarities for the purpose of visualization and interpretation. Wouldn't it be a more natural choice to use some metrics that at least involve CKA or layer-wise information? For example, can we simply optimize CKA directly in the pseudo-code instead of using $\\mathcal{I}(l_i,l)$ as the auxiliary objective? Could the authors provide some in-depth analysis and justifications on why simply the l2 norm is preferred? Or, is it possible to provide some more experiments that use CKA directly?\n\n**Possible Improvement**\nFrom the additional diversity training objective highlighted in the pseudo-code, it is conceptually related to the diversity learning literature in population-based-training, where a population (corresponding to ensemble to some extent) of policies are trained and each policy not only optimizes its reward objective but also optimizes a diversity metrics for the purpose of effectively covering diverse strategic modes. So it would be appreciated if the authors can provide some discussions in the related work section. Some of the references are listed here for the purpose of helping the authors to survey the literature more easily: https://arxiv.org/abs/2002.00632, http://proceedings.mlr.press/v139/lupu21a.html, https://arxiv.org/abs/2103.04564, https://arxiv.org/abs/2106.02195 ", "summary_of_the_paper": "This paper proposes a collection of diversity metrics to improve ensemble diversity, which substantially improves learning efficiency for a variety of RL methods.", "summary_of_the_review": "In general, although the paper can be still improved, the general results and suggested techniques are promising.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper concerns ensemble methods in deep reinforcement learning, examining several such methods, and proposes to address an important issue wherein ensemble members converge on a representation of approximately the same function, either by their parameters converging to an identical point or equivalent points that give rise to the same function. The authors propose a set of regularization methods aimed at improving diversity, and benchmark these augmentations on five ensemble methods and a dozen environments.

3 of 4 reviewers generally praised the method's simplicity and generality, and found the experiments convincing. Reviewer a9sA describes it as "clearly written and easy to follow", although others found clarity lacking in parts. There was agreement among these 3 reviewers that this was an interesting problem to tackle. Reviewer TfGq notes that this method lacks theoretical justification or guarantees, but that as a largely empirical paper this is perhaps of secondary importance. Reviewers 6miY and a9sA had questions about the precise choice of metrics, hyperparameters and seeds; the resulting discussion cleared up many of these concerns.

The most critical reviewer, i4M1, disputes the existence of the phenomenon at all, saying that "Neural networks converge to different solutions given the initialization is different and multiple local minima." The remainder of i4M1's criticisms seem centered on the choice of environments and the number of seeds (also raised by other reviewers). The issue of seeds has been addressed partially and the authors have committed to strengthening their results in this regard.

Reviewer i4M1's statement on the convergence of neural networks to different minima matches a bit of dated folk wisdom about neural networks, but the AC disputes this. The authors have cited a study from before the DL era properly began that identifies this issue and Section 5 addresses these criticisms directly. In practice, modern neural networks, especially with non-saturating activations, tend to be surprisingly consistent across random seeds when trained against the same data stream, and more recent work posits that the loss landscape is less riddled with local minima than with saddle points (see e.g. Dauphin et al, 2014). _Equivalent_ minima are of course common due to scaling and permutation symmetries but SGD has a well documented preference for low norm solutions in the former case, and the authors' have chosen methods that would at least conceivably overcome these issues, by focusing on summary statistics of the representations rather than their precise values (and indeed, CKA is designed with these concerns in mind).

Despite i4M1's incredulity I am inclined to agree with the majority of reviewers and view the paper as a worthwhile contribution to the body of knowledge (purely empirical though it may be) on both NN ensemble methods and DRL ensembles in particular. The introduction of measures from economics is clever and original, and the results are promising. A more exhaustive study on the entire Atari57 benchmark but can appreciate the resource problem this poses, and find that the suite of considered environments, combined with the augmentation of 5 different DRL ensemble methods, strikes a good balance. I concur on the issue of seeds and would encourage authors to include as many as possible for the camera ready, but on balance would recommend acceptance.
Decision: Accept (Poster)