DEEP REINFORCEMENT LEARNING
Year: 2022

Abstract:
Modern deep reinforcement learning (DRL) has been successful in solving a range of challenging sequential decision-making problems. Most of these algorithms use an ensemble of neural networks as their backbone structure and benefit from the diversity among the neural networks to achieve optimal results. Unfortunately, the members of the ensemble can converge to the same point either the parametric space or representation space during the training phase, therefore, losing all the leverage of an ensemble. In this paper, we describe Maximize Ensemble Diversity in Reinforcement Learning (MED-RL), a set of regularization methods inspired from the economics and consensus optimization to improve diversity in the ensemblebased deep reinforcement learning methods by encouraging inequality between the networks during training. We integrated MED-RL in five of the most common ensemble-based deep RL algorithms for both continuous and discrete control tasks and evaluated on six Mujoco environments and six Atari games. Our results show that MED-RL augmented algorithms outperform their un-regularized counterparts significantly and in some cases achieved more than 300% in performance gains.

heading: 1 INTRODUCTION
text: Reinforcement learning (RL) agents trained with high capacity function approximators such a deep neural networks have shown to solve complex sequential decision-making problems, including the board games of Chess, GO and Shogi (Silver et al., 2016; 2017; 2018), achieving super-human performance in video games (Mnih et al., 2015; Vinyals et al., 2019) and solving robotic manipulation tasks (Liu et al., 2021). Despite achieving these tremendous goals, modern deep reinforcement learning (DRL) algorithms have plethora of limitations. For example, it is well-known that DRL algorithms are sample-inefficient and require stupendous amount of environment interactions to learn an optimal policy (Łukasz Kaiser et al., 2020). Additional problems encountered and exacerbates during training a DRL agent includes the overestimation bias that occurs while estimating the target values for Q-learning (Fujimoto et al., 2018; Lan et al., 2020; Hado van Hasselt et al., 2016), error propagation during Bellman backup (Kumar et al., 2019) and trade-off between exploration and exploitation (Chen et al., 2017).
Recently, the use of ensemble has been a popular choice to address the above mentioned issues. These methods combine multiple neural networks to model the value functions or (and) the policy (Osband et al., 2016; Chen et al., 2017; Lan et al., 2020; Lee et al., 2020). For example, TD3 (Fujimoto et al., 2018) used two critics to address the overestimation bias problem in continuous control problems while MaxminDQN (Lan et al., 2020) provided a mechanism to use the cardinality of the ensemble to use as a knob to tune between over and under estimation in deep Q-learning. Similarly, Bootstrapped DQN (Osband et al., 2016; Chen et al., 2017) used ensemble for effective exploration.
∗Partial work done while being a Ph.D student at University of Central Florida
The primary insight of this paper is that the performance of ensemble based methods is contingent on maintaining sufficient diversity between the neural networks of the ensemble. If the neural networks in the ensembles converge to a common representation (we will show that this is the case in many scenarios), the performance of these approaches significantly degrades. We note that even with different representations, the Q-values will still converge towards a shared optimum, but they are statistically less likely to follow the same learning trajectory elsewhere.
In this paper, we propose Maximize Ensemble Diversity in Reinforcement Learning (MED-RL), a set of regularization methods inspired from the economics and consensus optimization to improve diversity and to prevent the collapse of the representations in the ensemble-based deep reinforcement learning methods by encouraging inequality between the networks during training. The objective of the regularizers is solely to keep the representations different, while still allowing the models to converge to the optimal Q-value. The motivation for the regularizers came from topic of income distribution in economic theory that provides a rich source of mathematical formulations that measure inequality. While in economics, high inequality is seen as a negative, in our case we used the inequality metrics to encourage diversity between the neural networks.
To summarize, our contributions are following:
1. We empirically show that high representation similarity between neural network based Q-functions leads to degradation in performance in ensemble based Q-learning methods.
2. To mitigate this, we propose five regularizers based on inequality measures from economics theory and consensus optimization that maximize diversity between the neural networks in ensemble based reinforcement learning methods.
3. We integrated MED-RL in TD3 (Fujimoto et al., 2018), SAC (Haarnoja et al., 2018) and REDQ (Chen et al., 2021) for continuous control tasks and in MaxminDQN (Lan et al., 2020) and EnsembleDQN (Anschel et al., 2017) for discrete control tasks and evaluated on six Mujoco environments and six Atari games. Our results show that MED-RL augmented algorithms outperform their un-regularized counterparts significantly and in some cases achieved more than 300% in performance gains and are up to 75% more sample-efficient.
4. We also show that MED-RL augmented SAC is more sample-efficient than REDQ, an ensemble based method specifically designed for sample-efficiency, and can achieve similar performance to REDQ up to 50 times faster on wall-clock time.

heading: 2 RELATED WORK
text: Ensembles in Deep RL: Use of an ensemble of neural networks in Deep RL has been studied in several recent studies for different purposes. In (Fujimoto et al., 2018; Anschel et al., 2017; Lan et al., 2020) have used an ensemble to address the overestimation bias in deep Q-learning based methods for both continuous and discrete control tasks. Similarly, Bootstrapped DQN and extensions (Osband et al., 2016; Chen et al., 2017) have leveraged ensemble of neural networks for efficient exploration. The problem of error propagation in Bellman backup was addressed in (Kumar et al., 2019) using an ensemble of neural networks. Sample efficiency, a notorious problem in RL has taken advantage from an ensemble (Chen et al., 2021). Recently, SUNRISE (Lee et al., 2020) proposed a unified framework for ensemble-based deep reinforcement learning.
Diversity in Ensembles: Diversity in neural network ensembles has been studied years before the resurgence of deep learning (Brown, 2004). Even though diversity is an important topic in neural networks, most of the studies in this topic revolve around addressing problems in supervised learning settings. More recently there has been a number of studies that have diversity in ensembles to measure and improve model uncertainty. Jain et al. (2020) have proposed a diversity regularizer to improve the uncertainty estimates in out-of-data distribution. Lee et al. (2015) have used Multiple choice Learning to learn diverse Convolutional Neural Networks for image recognition.
Regularization in Reinforcement Learning: Regularization in reinforcement learning has been used to perform effective exploration and learning generalized policies. For instance, (Grau-Moya et al., 2019) uses mutual-information regularization to optimize a prior action distribution for better performance and exploration, (Cheng et al., 2019) regularizes the policy π(a|s) using a control
prior, (Galashov et al., 2019) uses temporal difference error regularization to reduce variance in Generalized Advantage Estimation (Schulman et al., 2016). Generalization in reinforcement learning refers to the performance of the policy on different environment compared to the training environment. For example, (Farebrother et al., 2018) studied the effect ofL2 norm on DQN on generalization, (Tobin et al., 2017) studied generalization between simulations vs. the real world, (Pattanaik et al., 2018) studied parameter variations and (Zhang et al., 2018) studied the effect of different random seeds in environment generation.
Diversity in Reinforcement Learning: Diversity in reinforcement learning is active area of research. (Pacchiano et al., 2020) uses Determinantal Point Processes to promote behavioral diversity, Lupu et al. (2021) have used policy diversity to improve zero-shot coordination in multi-agent setting. (Tang et al., 2021) uses reward randomization for discovering diverse strategic policies in complex multi-agent games. In (Li et al., 2021) proposed CDS that uses information-theoretical objective to maximize the mutual information between agents’ identities and trajectories and encourage diversity. More recently (An et al., 2021) have used diversified Q-ensembles to address overestimation in offline reinforcement learning.
Representation Similarity: Measuring similarity between the representations learned by different neural networks is an active area of research. For instance, (Raghu et al., 2017) used Canonical Correlation Analysis (CCA) to measure the representation similarity. CCA find two basis matrices such that when original matrices are projected on these bases, the correlation is maximized. (Raghu et al., 2017; Mroueh et al., 2015) used truncated singular value decomposition on the activations to make it robust for perturbations. Other work such as (Li et al., 2015) and (Wang et al., 2018) studied the correlation between the neurons in the neural networks.

heading: 3 BACKGROUND
text: Reinforcement learning: We consider an agent as a Markov Decision Process (MDP) defined as a five element tuple (S,A, P, r, γ), where S is the state space,A is the action space, P : S ×A×S → [0, 1] are the state-action transition probabilities, r : S × A × S → R is the reward mapping and γ → [0, 1] is the discount factor. At each time step t the agent observes the state of the environment st ∈ S and selects an action at ∈ A. The effect of the action triggers a transition to a new state st+1 ∈ S according to the transition probabilities P , while the agent receives a scalar reward Rt = r (st, at, st+1). The goal of the agent is to learn a policy π that maximizes the expectation of the discounted sum of future rewards.
Representation Similarity Measure: LetX ∈ Rn×p1 denote a matrix of activations of p1 neurons for n examples and Y ∈ Rn×p2 denote a matrix of activations of p2 neurons for the same n examples. Furthermore, we consider Kij = k (xi, xj) and Lij = l (yi, yj) where k and l are two kernels.
Centered Kernel Alignment (CKA) (Kornblith et al., 2019; Cortes et al., 2012; Cristianini et al., 2002) is a method for comparing representations of neural networks, and identifying correspondences between layers, not only in the same network but also on different neural network architectures. CKA is a normalized form of Hilbert-Schmidt Independence Criterion (HSIC) (Gretton et al., 2005). Formally, CKA is defined as:
CKA (K,L) = HSIC (K,L)√
HSIC (K,K) · HSIC (L,L)
HSIC is a test statistic for determining whether two sets of variables are independent. The empirical estimator of HSIC is defined as:
HSIC (K,L) = 1
(n− 1)2 tr (KHLH)
where H is the centering matrix Hn = In − 1
n 11T .

heading: 4 MAXIMIZE ENSEMBLE DIVERSITY IN REINFORCEMENT LEARNING
text: In this section, we propose MED-RL: Maximize Ensemble Diversity in Reinforcement Learning, a set of regularizers inspired from the Economics and consensus optimization to improve diversity and to prevent the collapse of the representations in the ensemble-based deep reinforcement learning methods by encouraging inequality between the networks during training. This section is organized as follows:
1. We empirically show that high representation similarity between neural network based Q-functions leads to degradation in performance in ensemble based Q-learning methods.
2. we present the Economics theory and consensus optimization inspired regularizers with their mathematical formulation.

heading: 4.1 EMPIRICAL EVIDENCE TO CORRELATE PERFORMANCE AND REPRESENTATION SIMILARITY
text: The work in this paper starts from the conjecture that high representation similarity between neural networks in an ensemble-based Q-learning technique correlates to poor performance. To empirically verify our hypothesis, we trained a MaxminDQN (Lan et al., 2020) agent with two neural networks on the Catcher environment (Qingfeng, 2019) for about 3000 episodes (5× 106 training steps) and calculated the CKA similarity with a linear kernel after every 500 episodes. The training graph along with the CKA similarity heatmaps are shown in Figure 1. Notably at episode 500 (heatmap A) and episode 2000 (heatmap C), the representation similarity between neural networks is low but the average return is relatively high. In contrast, at episode 1000 (heatmap B) and episode 3000 (heatmap D) the representation similarity is highest but the average return is lowest.

heading: 4.2 REGULARIZATION FOR MAXIMIZING ENSEMBLE DIVERSITY
text: In order to maximize the ensemble diversity, we propose to regularize the training algorithm with an additional criteria that favors diversity between the ensembles. In the following, N is the number of neural networks in the ensemble, `i is the L2 norm of the i-th neural network’s parameters, ¯̀ is the mean of all the L2 norms and ` is the list of all the L2 norms.
The first four metrics we consider are based on inequality measures from economic theory. While in economics, inequality is usually considered something to be avoided, in our case we aim to increase inequality (and thus, ensemble diversity).
The Atkinson Index (Atkinson et al., 1970) measures income inequality and is useful in identifying the end of the distribution that contributes the most towards the observed inequality. Formally, it is defined as
A =  1− 1¯̀ ( 1 N N∑ i=1 `1− i ) 1 1− at , for 0 ≤ at 6= 1, 1− 1¯̀ ( 1
N N∏ i=1 `i ) 1 N , for at = 1,
(1)
where at is the inequality aversion parameter used to tune the sensitivity of the measured change. When at = 0, the index is more sensitive to the changes at the upper end of the distribution, while it becomes sensitive towards the change at the lower end of the distribution when at approaches 1.
The Gini coefficient (Allison, 1978) is a statistical measure of the wealth distribution or income inequality among a population and defined as the half of the relative mean absolute difference:
G =
∑N i=1 ∑N j=1 |`i − `j |
2N2 ¯̀ (2)
The Gini coefficient is more sensitive to deviation around the middle of the distribution than at the upper or lower part of the distribution.
The Theil index (Johnston, 1969) measures redundancy, lack of diversity, isolation, segregation and income inequality among a population. Using the Theil index is identical to measuring the redundancy in information theory, defined as the maximum possible entropy of the data minus the observed entropy:
TT = 1
N N∑ i=1 `i ¯̀ ln `i ¯̀ (3)
The variance of logarithms (Ok & Foster, 1997) is a widely used measure of dispersion with natural links to wage distribution models. Formally, it is defined as:
VL(`) = 1
N N∑ i=1 [ln `i − ln g(`)]2 (4)
where g(`) is the geometric mean of ` defined as ( ∏N i=1 `i) 1/N .
The final regularization method we use is inspired from consensus optimization. In a consensus method (Boyd et al., 2011), a number of models are independently optimized with their own taskspecific parameters, and the tasks communicate via a penalty that encourages all the individual solutions to converge around a common value. Formally, it is defined as
M = ‖θ̄ − θi‖2 (5)
Where θ̄ is the mean of the parameters of all the neural networks and θi represents the parameters of the i− th neural network. We will refer this regularizer as MeanVector throughout this paper. For completeness, the algorithm shown in Algorithm 1. Notice that the regularization term appears with a negative sign, as the regularizers are essentially inequality metrics that we want to maximize.

heading: 4.3 TRAINING ALGORITHM
text: Using the regularization functions defined above, we can develop diversity-regularized variants of the the ensemble based algorithms. The training technique is identical to the algorithms described in (Lan et al., 2020; Anschel et al., 2017; Fujimoto et al., 2018; Haarnoja et al., 2018; Chen et al., 2021), with a regularization term added to the loss of the Q-functions. The loss term for i-th Q-function with parameters ψi is:
L (ψi) = Es,a,r,s′ [( Qiψ (s, a)− Y )2]− λI (`i, `) , where Y is the target value depending on the algorithm, I is the regularizer of choice from the list above and λ is the regularization weight. Notice that the regularization term appears with a negative sign, as the regularizers are essentially inequality metrics that we want to maximize. As a reference the modified algorithm for MaxminDQN is shown in Algorithm 1.

heading: 5 EXPERIMENTS

heading: 5.1 ISN’T RESAMPLING AND DIFFERENT INITIALIZATION OF WEIGHTS ENOUGH?
text: The most common question that comes to mind to address the diversity issue is why not initialize the neural networks with different weights and train each network with a different sample from the buffer? This approach has been thoroughly discussed in (Brown, 2004) and have been shown to be ineffective. To re-iterate the findings in (Brown, 2004), we performed a regression experiment in which we learnt a sine function using two different three layered fully connected neural networks with 64 and 32 neurons in each hidden layer with ReLU. The neural networks were initialized using different weights and were trained using different batch sizes (512, 128) and learning rates (1e−4, 1e−3). The Figure 2a shows the learnt functions while Figure 2b represents their CKA similarity heatmap before and after training. The odd numbered layers represent pre-ReLU activations while the even numbered layers represent post-ReLU activations. It can be seen that before training, the CKA similarity between the two neural networks from layer 4 and onward is relatively low and the output being 0% similar while after training, the trained networks have learnt highly similar representation while their output being 98% similar.
This example shows that neural networks can learn similar representation while trained on different batches. This observation is important because in MaxminDQN and EnsembleDQN training,
each neural network is trained on a separate batch from the replay buffer but still learns similar representations.

heading: 5.2 EXPERIMENTAL SETUP
text: Continuous control tasks: We evaluated MED-RL augmented continuous control algorithms such as TD3, SAC and REDQ on the Mujoco continuous control benchmark environments. We compared the results of MED-RL with un-regularized counterparts. We report the mean and standard deviation across five runs after 1M timesteps on six complex environments: Cheetah, Walker, Hopper, Ant, Humanoid and Humanoid-Standup. For REDQ, we evaluated on Cheetah, Walker, Hopper and Ant for 300K timesteps only.

heading: 5.3 COMPARATIVE EVALUATION
text: Continuous control tasks: Tables 1 to 3 show the average returns of evaluation roll-outs for all the continuous control methods. MED-RL consistently improves the performance of SAC, TD3 and REDQ in all the experiments. Even though the focus of this work is to maximize the average return, we find that MED-RL augmented algorithms are more sample-efficient than their un-regularized counterparts (see Figures 3, 4 and 5). For example, it can be seen Figure 3b that baseline SAC reaches the average return of 10K in about 1M environment interactions while all MED-RL variants reach the same average return in approximately 250K environment interaction, therefore, being approximately 75% more sample efficient than the baseline SAC. This improvement in sample-efficiency can be noted in all nearly all the experiments except SAC-Humanoid and TD3-Walker experiments. The training plots for REDQ are shown in Appendix.

heading: 5.4 SAMPLE EFFICIENCY AND COMPUTE TIME
text: Tables 1 to 3 show that MED-RL augmented continuous control algorithms outperform the baseline versions significantly and a visual inspection of Figures 3, 4 and 5 show that MED-RL augmented algorithms are more sample-efficient as well. But are they more sample-efficient than algorithms that are specifically designed for sample-efficiency such as REDQ? To answer this question, we took the bolded results from Table 1, referred as MED-RL in this section, and evaluated the number of environment interactions and wall-clock time it took for MED-RL to reach similar performance as that of baseline REDQ. As shown in Table 4, MED-RL achieves similar performance to REDQ in 50% and 20% few environment interactions on Ant and HalfCheetah environment respectively and have significantly surpassed REDQ on 300K environment interactions. MED-RL does not only improve sample-efficiency but significantly improves compute time. As shown in Table 4, MED-RL achieves similar performance to REDQ up to 50 times faster on wall-clock time. Note it can be argued that REDQ can be parallelized to achieve faster wall-clock time but here we are only comparing standard sequential implementations but that will not address the sample-efficiency issue.
Discrete control tasks: We evaluated MED-RL augmented discrete control algorithms such as MaxminDQN and EnsembleDQN on the PyGames (Qingfeng, 2019) and MinAtar (Young & Tian, 2019). We chose these environments to have a fair comparison since we used the source code provided by MaxminDQN authors. We reused all the hyperparameter settings from (Lan et al., 2020) except the number of neural networks, which we limited to four and trained each solution for five fixed seeds. The results on the discrete control tasks are shown in the Appendix.

heading: 6 CONCLUSION
text: In this paper, we proposed Maximize Ensemble Diversity in Reinforcement Learning (MED-RL), a set of regularization methods inspired from the economics and consensus optimization to improve diversity in the ensemble-based deep reinforcement learning methods by encouraging inequality between the networks during training. We also empirically showed that high representation similarity between the networks of the ensemble could cause degradation in the performance. Our experiments have shown that MED-RL not only improves the average return of ensemble based reinforcement learning algorithms but can increase their sample-efficiency by approximately 75% when compared to their un-regularized counterparts. Additionally we have shown the SAC when augmented with MED-RL can outperform REDQ, an algorithm specifically designed for sample-efficiency, in both sample-efficiency and compute time.
Acknowledgement: This work had been supported in part by the National Science Foundation under grant number CNS-1932300

heading: A TRAINING PLOTS OF REDQ

heading: B RESULTS ON DISCRETE CONTROL TASKS USING MAXMINDQN
text: 0 1 2 3 Environment Interactions (×106)
10
0
10
20
30
40
50
60
Av er
ag e
Re tu
rn
Maxmin N=2 Maxmin N=3 Maxmin N=4 Atkinson N=2 Atkinson N=3 Atkinson N=4
0 0.5 1 1.5 2 Environment Interactions (×106)
0
20
40
60
Av er
ag e
Re tu
rn
Maxmin N=2 Maxmin N=3 Maxmin N=4 Atkinson N=2 Atkinson N=3 Atkinson N=4
0 1 2 3 4 5 Environment Interactions (×106)
0
10
20
30
40
50
60
Av er
ag e
Re tu
rn
Maxmin N=2 Maxmin N=3 Maxmin N=4 Atkinson N=2 Atkinson N=3 Atkinson N=4
0 1 2 3 Environment Interactions (×106)
10
0
10
20
30
40
50
60
Av er
ag e
Re tu
rn
Maxmin N=2 Maxmin N=3 Maxmin N=4 Gini N=2 Gini N=3 Gini N=4
0 0.5 1 1.5 2 Environment Interactions (×106)
0
20
40
60
80
Av er
ag e
Re tu
rn
Maxmin N=2 Maxmin N=3 Maxmin N=4 Gini N=2 Gini N=3 Gini N=4
0 1 2 3 4 5 Environment Interactions (×106)
0
10
20
30
40
50
60
Av er
ag e
Re tu
rn
Maxmin N=2 Maxmin N=3 Maxmin N=4 Gini N=2 Gini N=3 Gini N=4
0 1 2 3 Environment Interactions (×106)
10
0
10
20
30
40
50
60
Av er
ag e
Re tu
rn
Maxmin N=2 Maxmin N=3 Maxmin N=4 MeanVector N=2 MeanVector N=3 MeanVector N=4
0 0.5 1 1.5 2 Environment Interactions (×106)
0
20
40
60
80
Av er
ag e
Re tu
rn
Maxmin N=2 Maxmin N=3 Maxmin N=4 MeanVector N=2 MeanVector N=3 MeanVector N=4
0 1 2 3 4 5 Environment Interactions (×106)
0
10
20
30
40
50
60
Av er
ag e
Re tu
rn
Maxmin N=2 Maxmin N=3 Maxmin N=4 MeanVector N=2 MeanVector N=3 MeanVector N=4
0 1 2 3 Environment Interactions (×106)
10
0
10
20
30
40
50
60
Av er
ag e
Re tu
rn
Maxmin N=2 Maxmin N=3 Maxmin N=4 Theil N=2 Theil N=3 Theil N=4
0 0.5 1 1.5 2 Environment Interactions (×106)
0
20
40
60
80
Av er
ag e
Re tu
rn
Maxmin N=2 Maxmin N=3 Maxmin N=4 Theil N=2 Theil N=3 Theil N=4
0 1 2 3 4 5 Environment Interactions (×106)
0
10
20
30
40
50
60
70
Av er
ag e
Re tu
rn
Maxmin N=2 Maxmin N=3 Maxmin N=4 Theil N=2 Theil N=3 Theil N=4
0 1 2 3 Environment Interactions (×106)
10
0
10
20
30
40
50
60
Av er
ag e
Re tu
rn
Maxmin N=2 Maxmin N=3 Maxmin N=4 VOL N=2 VOL N=3 VOL N=4
Catcher
0 0.5 1 1.5 2 Environment Interactions (×106)
0
20
40
60
80
Av er
ag e
Re tu
rn
Maxmin N=2 Maxmin N=3 Maxmin N=4 VOL N=2 VOL N=3 VOL N=4
PixelCopter
0 1 2 3 4 5 Environment Interactions (×106)
0
10
20
30
40
50
60
70
Av er
ag e
Re tu
rn
Maxmin N=2 Maxmin N=3 Maxmin N=4 VOL N=2 VOL N=3 VOL N=4
Asterix
B.1 RESULTS IN TABULAR FORM
Figure 6 shows the training curves for the all the six environments and Tables 5 to 7 represent the results in the tabular, similar to our observations for the continuous control experiments, MED-RL augmented MaxminDQN outperformed un-regularized MaxminDQN significantly on average return and sample-efficiency metrics. Notably, in Seaquest and SpaceInvader environments where MED-RL MaxminDQN with two neural networks achieved 400% and 300% increase in performance. Note that the baselines shown in Figure 6 are strong baselines. For example, in the MaxminDQN paper, the best performance of baseline MaxminDQN on Asterix and SpaceInvader environments is around 20 and 50 respectively while we have achieved an average reward of 35 and 135 respectively.

heading: C RESULTS ON DISCRETE CONTROL TASKS USING ENSEMBLEDQN
text: 0 1 2 3 Environment Interactions (×106)
10
0
10
20
30
40
50
60
Av er
ag e
Re tu
rn
Ensemble N=2 Ensemble N=3 Ensemble N=4 Atkinson N=2 Atkinson N=3 Atkinson N=4
0 0.5 1 1.5 2 Environment Interactions (×106)
0
20
40
60
80
Av er
ag e
Re tu
rn
Ensemble N=2 Ensemble N=3 Ensemble N=4 Atkinson N=2 Atkinson N=3 Atkinson N=4
0 1 2 3 4 5 Environment Interactions (×106)
0
10
20
30
40
50
60
Av er
ag e
Re tu
rn
Ensemble N=2 Ensemble N=3 Ensemble N=4 Atkinson N=2 Atkinson N=3 Atkinson N=4
0 1 2 3 Environment Interactions (×106)
10
0
10
20
30
40
50
60
Av er
ag e
Re tu
rn
Ensemble N=2 Ensemble N=3 Ensemble N=4 Gini N=2 Gini N=3 Gini N=4
0 0.5 1 1.5 2 Environment Interactions (×106)
0
20
40
60
80
Av er
ag e
Re tu
rn
Ensemble N=2 Ensemble N=3 Ensemble N=4 Gini N=2 Gini N=3 Gini N=4
0 1 2 3 4 5 Environment Interactions (×106)
0
10
20
30
40
50
60
Av er
ag e
Re tu
rn
Ensemble N=2 Ensemble N=3 Ensemble N=4 Gini N=2 Gini N=3 Gini N=4
0 1 2 3 Environment Interactions (×106)
10
0
10
20
30
40
50
60
Av er
ag e
Re tu
rn
Ensemble N=2 Ensemble N=3 Ensemble N=4 MeanVector N=2 MeanVector N=3 MeanVector N=4
0 0.5 1 1.5 2 Environment Interactions (×106)
0
20
40
60
80
Av er
ag e
Re tu
rn
Ensemble N=2 Ensemble N=3 Ensemble N=4 MeanVector N=2 MeanVector N=3 MeanVector N=4
0 1 2 3 4 5 Environment Interactions (×106)
0
10
20
30
40
50
60
Av er
ag e
Re tu
rn
Ensemble N=2 Ensemble N=3 Ensemble N=4 MeanVector N=2 MeanVector N=3 MeanVector N=4
0 1 2 3 Environment Interactions (×106)
10
0
10
20
30
40
50
60
Av er
ag e
Re tu
rn
Ensemble N=2 Ensemble N=3 Ensemble N=4 Theil N=2 Theil N=3 Theil N=4
0 0.5 1 1.5 2 Environment Interactions (×106)
0
20
40
60
80
Av er
ag e
Re tu
rn
Ensemble N=2 Ensemble N=3 Ensemble N=4 Theil N=2 Theil N=3 Theil N=4
0 1 2 3 4 5 Environment Interactions (×106)
0
20
40
60
80
Av er
ag e
Re tu
rn
Ensemble N=2 Ensemble N=3 Ensemble N=4 Theil N=2 Theil N=3 Theil N=4
0 1 2 3 Environment Interactions (×106)
10
0
10
20
30
40
50
60
Av er
ag e
Re tu
rn
Ensemble N=2 Ensemble N=3 Ensemble N=4 VOL N=2 VOL N=3 VOL N=4
Catcher
0 0.5 1 1.5 2 Environment Interactions (×106)
0
20
40
60
80
Av er
ag e
Re tu
rn
Ensemble N=2 Ensemble N=3 Ensemble N=4 VOL N=2 VOL N=3 VOL N=4
PixelCopter
0 1 2 3 4 5 Environment Interactions (×106)
0
20
40
60
Av er
ag e
Re tu
rn
Ensemble N=2 Ensemble N=3 Ensemble N=4 VOL N=2 VOL N=3 VOL N=4
Asterix
C.1 RESULTS IN TABULAR FORM
Figure 7 shows the training curves for the all the six environments and Tables 8 to 10 represent the results in the tabular, similar to our observations for the continuous control experiments, MED-RL augmented EnsembleDQN outperformed un-regularized EnsembleDQN significantly on average return and sample-efficiency metrics. Notably, in Copter, SeaQuest and SpaceInvader environments where MED-RL EnsembleDQN with two neural networks achieved 200%, 350% and 250% increase in performance.

heading: D T-SNE VISUALIZATIONS
text: To visualize the impact of the regularization, Figures 8 and 9 shows t-SNE (van der Maaten & Hinton, 2008) visualization of the activations of the last layer of the trained networks. Figure 8a show the network trained for the Catcher environment, while Figures 8b and 9, the network trained for the PixelCopter environment. The upper row of the figure shows the original, unregularized models, while the lower row a regularized version. For all combinations, we find that the activations from the original MaxminDQN and EnsembleDQN versions do not show any obvious pattern, while the regularized ones show distinct clusters. An additional benefit of t-SNE visualizations over CKA similarity heatmaps is that the CKA similarity heatmaps are useful to show representation similarity between two neural networks, but they become counter intuitive as the number of neural networks increases.

heading: E PLOTTING THE GINI INEQUALITY
text: We measured the L2 norm inequality of the baseline MaxminDQN and EnsembleDQN along with their regularized versions. We trained baseline MaxminDQN and EnsembleDQN with two neural networks along with their Gini index versions with regularization weight of 10−8 on the PixelCopter environment on a fixed seed . Figure 10 represents the L2 norm inequality of the experiments along their average return during training. Notably, despite each neural network being trained on a different batch, the L2 norm of the baseline MaxminDQN and EnsembleDQN are quite similar while the L2 norm of the regularized MaxminDQN and EnsembleDQN have high inequality.
F IMPLEMENTATION DETAILS AND HYPERPARAMETERS
For our implementation of MaxminDQN and EnsembleDQN, we used the code provided by the MaxminDQN authors that has implementations of different DQN based methods github.com/qlan3/Explorer). For the baseline experiments, we used most of the hyperparameter settings provided in the configuration files by the authors except the number of ensembles which we limited to four.
F.1 COMPUTING INFRASTRUCTURE
All the experiments were performed on a Kubernetes managed cluster with Nvidia V100 GPUs and Intel Skylake CPUs. Each experiment was run as an individual Kubernetes job with 11 CPUs, 16GB of RAM and 1 GPU (if needed). This configuration allowed us to run experiments without any interference from other applications which was important to accurately measure the wall-clock time.
Future Work: Even though the focus of this paper was on empirical testing, there are several different research questions that needs to be addressed, for example how do we select the best regularizer for a particular environment or an algorithm with some N numbers of networks in the ensemble. Another interesting line of research is to study how different inequality distributions effect the diversity of the ensembles.

Review summaries:
MED-RL This paper studies foster diversity in ensemble of DRL networks by regularization methods. The paper is an empirical one and compared five ensemble methods with and without their diversity algorithm in six Mujoco and six Atari games, and showed some results.  The algorithm proposed is a modification of MaxMinDQN by Lan et.al. with a regularization. 

The paper considers a problem of ensemble-based deep RL methods that ensembles of critic networks converge to the same point in the representation space. To address it, the paper proposes a regularization technique that forces representations of a critic network to be dissimilar from those of other critic networks in the ensemble. It is empirically shown that this regularization technique improves the sample efficiency and asymptotic performance of baseline algorithms.