Preventing Value Function Collapse in Ensemble  Q-Learning by Maximizing Representation Diversity
Authors: Hassam Sheikh
Ladislau Boloni
Venue:  (2021)
Keywords: Ensemble Q-Learning
Representation Diversity
Reinforcement Learning

Abstract: The first deep RL algorithm, DQN, was limited by the overestimation bias of the learned Q-function. Subsequent algorithms proposed techniques to reduce this problem, without fully eliminating it. Recently, the Maxmin and Ensemble Q-learning algorithms used the different estimates provided by ensembles of learners to reduce the bias. Unfortunately, these learners can converge to the same point in the parametric or representation space, falling back to the classic single neural network DQN. In this paper, we describe a regularization technique to maximize diversity in the representation space in these algorithms. We propose and compare five regularization functions inspired from economics theory and consensus optimization. We show that the resulting approach significantly outperforms the Maxmin and Ensemble Q-learning algorithms as well as non-ensemble baselines.

Under review as a conference paper at ICLR 2021
PREVENTING VALUE FUNCTION COLLAPSE IN ENSEMBLE Q-LEARNING BY MAXIMIZING REPRESENTATION DIVERSITY
Anonymous authors Paper under double-blind review
ABSTRACT
The first deep RL algorithm, DQN, was limited by the overestimation bias of the learned Q-function. Subsequent algorithms proposed techniques to reduce this problem, without fully eliminating Recently, the Maxmin and Ensemble Q- learning algorithms used the different estimates provided by ensembles of learners to reduce the bias. Unfortunately, these learners can converge to the same point in the parametric representation space, falling back to the classic single neural network DQN. In this paper, we describe a regularization technique to maximize diversity in the representation space in these algorithms. We propose and com- pare five regularization functions inspired from economics theory and consensus optimization. We show that the resulting approach significantly outperforms the Maxmin and Ensemble Q-learning algorithms as well as non-ensemble baselines.
INTRODUCTION
Q-learning [Watkins 1989) and its deep learning based successors inaugurated by DQN (Mnih et al. 2015) are model-free, value function based reinforcement learning algorithms. Their popularity stems from their intuitive, easy-to-implementt update rule derived from the Bellman equation. At each time step. the agent updates its Q-value towards the expectation of the current reward plus the value corresponding to the maximal action the next state. This state-action value represents the maximum sum of reward the agent believes it could obtain from the current state by taking the current action. Unfortunately (Thrun Schwartz, 1993 van Hasselt 2010) have shown that this simple rule suffers from overestimation bias: due to the maximization operator in the update rule, positive and negative errors do not cancel each other out, but positive errors accumulate. The overestimation bias is particularly problematic under function approximation and have contributed towards learning sub-optimal policies (Thrun& & Schwartz 1993. Szita & Lörinez 2008 Strehl et al. 2009).
A possible solution is to introduce umderestimation bias in the estimation of the Q-value. Double Q-learning (van Hasselt 2010) maintains two independent state-action value estimators (Q-functions). is calculated by adding observed reward and maximal state-
The state-action value of estimator action value from the other estimator. Double DQN (Hado van Hasselt et al.) 2016) applied this idea using neural networks, and was shown to provide better performance than DQN. More recent actor-critic type deep RL algorithms such as TD3 (Fujimoto et al. 2018) and SAC (Haarnoja et al. 2018) also use two Q function estimators (in combination with other techniques).
Other approaches such as EnsembleDQN (Anschel et al. 2017) and MaxminDQN (Lan et al. 2020) maintain ensembles of Q-functions to estimate an unbiased Q-funciion. EnsembleDQN estimates the state-action values by adding the current observed reward and the maximal state-action value from the average of Q-functions from the ensemble. MaxminDQN creates a proxy Q-function by selecting the minimum Q-value for each action from all the Q-functions and using the maximal state-action value from the proxy Q-function to estimate an unbiased Q-function. Both EnsembleDQN and MaxminDQN have been shown to perform better than Double DQN. The primary insight of this paper is that the performance of ensemble based methods is contingent on maintaining sufficient diversity in the representation space between the f-funtiions in the ensembles. If the Q-functions in the ensembles converge to a common representation (we will show that this is the case in many scenarios), the performance of these approaches significantly degrades.

---

Under review as a conference paper at ICLR 2021
In this paper we propose cross-learned regularizers prevent the collapse of the representation space in ensemble- masseble-bbasd Q-learning methods. Intuitively, these representations capture an inductive bias towards more diverse representations. We have investigated five different regularizers. The mathematical formulation of four of the regularizers correspond to inequality measures borrowed from economics theory. While in economics, high inequality is the metrics to encourage inequality between the representations. The fifth regularizer is inspired from consensus optimization.
negative, in this case
There is a separate line of reinforcement learning literature where ensembles are used to address several different issues (Chen al., 2017; Chua et al. 2018 Kurutach et al., 2018 Lee al. 2020) Osband et al. (016) such as exploration and error propagation but we limit our solution to algorithms addressing the overestimation bias problem only.
To summarize, our contributions are following:
1. We show that high representation similarity between neural network based Q-functions leads to decline in performance in ensemble based Q-learning methods.
2. To mitigate this, we propose five regularizers based on incquality measures from eco- nomics theory and consensus optimization that maximize representation diversity between Q- functions in ensemble based Q P-laerning methods.
3. We show that applying the proposed regularizers to the MaxminDQN and EnsembleDQN methods can lead significant improvement in performance over a variety of benchmarks.
2 BACKGROUND
Reinforcement learning considers an agent as a Markov Decision Process (MDP) defined as a five element tuple (S,A, P)r:7), where is the state space, A is the action space, P: Axxxxxxx [0, 1] are the state-action transition prohabilities,
S x A x S -> is the reward mapping and y y [0, 1] is the discount factor. At each time step t the agent observes the state of the environment 8, E S and selects an action E A. The effect of the action triggers a transition to new state the transition probabilities P. while the agent receives scalar reward R. = r (st, $t+1). The goal of the agent is to learn policy π that maximizes the expectation of the discounted sum of future rewards.
8141 E S according
One way to implicitly learn the policy is the Q-learning algorithm that estimates the expected sum of rewards of state 8 we take the action at by solving the Bellman equation
Q" (st, a,) = E max Q" (8141, a')
d'EA
The implicit policy x can extracted by acting greedily with respect to the optimal Q-function: arg max Q" (s, a). One possible way estimate the optimal Q-value is by iteratively updating it for aEAA
sampled states s, and action a, using
Q" (st, az) + Q" (s), + (Y; Q" (z)) where Y₁ = R₁ max Q" ($141,a') a'EAA
where a is the step size and Y, called the target value. While this algorithm had been initially studied in the context of a tabular representation of Q for discrete states and actions, in many practical applications the Q value approximated by a learned function. Since the emergence of deep learning, the preferred approximation technique is based deep neural network. DQN (Mnih et al. 2015). had demonstrated super-human performance in Atari Games, but required a very large number of training iterations. From this baseline, subsequent algorithms improved both the learning speed and achievable performance, with one of the main means for this being techniques to reduce the overestimation bias of the Q-function.
EnsembleDQN (Anschel et al. (017)) uses an ensemble of N neural networks to estimate state-action values and uses their average to reduce both overestimation bias and estimation variance. Formally,
2

---

Under review as a conference paper at ICLR 2021
the target value for EnsembleDQN is calculated using
N Σ Q-)-)
QE (-) N 1=1
y Rg max QE (#+41, a' EA
More recent, MaxminDQN (Lan et al. 2020) addresses the overestimation bias using order statistics, using the ensemble size N as hyperparameter to tune between underestimating and overestimating bias. The target value for MaxminDQN is calculated using
QM (..))
min Qi Q: i=l.....
MM = Rr max QM ($(41, a'EA
3 RELATED WORK
Techniques to Address Overestimation Bias in RL: Addressing overestimation bias is a long standing research topic not only in reinforcement learning but other fields of science such as economics and statistics. is commonly known as max- meex-operator bias in statistics (D Eramo et al.) 2017) and the winner's
in economics (Thaler 2012 Smith & Winkler 2006). To address this, (van Hasselt 2010) proposed Double Q-learning, subsequently adapted to a neural network based function approximators Double DQN (Hado Hasselt et al. (0016). Alternatively, (Zhang et al.
2017 Lv et al. 2019) proposed weighted estimators of Double Q-learning and (Lee et al. (0033) introduced bias correction term. Other approaches to address the overestimation are based on averaging and ensembling. Techniques include averaging Q-values from previous N versions of the Q-network (Anschel et al. 2017), taking linear combinations of min and max over the pool of Q-values (Kumar et al. 2019). or using random mixture from the pool (Agarwal et al. 2019).
Regularization in Reinforcement Learning: Regularization in reinforcement learning has been used to perform effective exploration and learning generalized policies. For instance, (Grau-Moyal et al. 2019) uses mutual-information regularization to optimize a prior action distribution for better performance and exploration, (Cheng et al. 2019) regularizes the policy T(ass) using control prior, (Galashov et al., 2019) uses temporal difference error regularization to reduce variance in Generalized Advantage Estimation (Schulman et al., 2016). Generalization in reinforcement learning refers to the performance of the policy on different environment compared to the training environment. For example, (Farebrother et al. 2018) studied the effect of L norm DQN generalization, (Tobin et al. 2017) studied generalization between simulations vs. the real world, (Pattanaik et al., 2018) studied parameter variations and (Zhang al. 2018) studied the effect of different random seeds in environment generation.
Representation Similarity: Measuring similarity between the representations learned by different neural networks is an active area of research. For instance, (Raghu et al.) 2017) used Canonical Correlation Analysis (CCA) to measure the representation similarity. CCA find two basis matrices such that when original matrices are projected on these bases, the correlation is maximized. (Raghu al. 2017 Mrouch et al. 2015) used truncated singular value decomposition on the activations to make it robust for perturbations. Other work such as (Li et al. 2015) and (Wang et al. 2018) studied correlation between the neurons the neural networks.
MAXIMIZING REPRESENTATION DIVERSITY IN ENSEMBLE-BASED DEEP Q-LEARNING
The work described in this paper is based on the conjecture that while ensemble-based deep Q- learning approaches aim to reduce the overestimation bias, this only works to the degree that the neural networks in the ensemble use diverse representations. If during training, these networks collapse to closely related representations, the learning performance decreases. From this idea, propose to use regularization techniques to maximize representation diversity between the networks of the ensemble.
3

---

Under review as a conference paper at ICLR 2021
4.1 REPRESENTATION SIMILARITY MEASURE
Let X E IR denote matrix of activations of pi neurons for n examples and Y E R denote matrix of activations of p2 neurons for the same n examples. Furthermore, consider Ki = (1,, and La = (y), U2) where and / are two kernels.
Centered Kernel Alignment (CKA) (Kornblihh al. 2019 Cortes al. 2012 Cristianini al. 2002) is a method for comparing representations of neural networks, and identifying correspondences between layers, not only the same network but also different neural network architectures. CKA is normalized form of Hilbeet-Schhiie Independence Criterion (HSIC) (Gretoon et al. 2005). Formally, CKA is defined as:
HSIC (K,
CKA LK
V /HSIC (K, K) HSIC (L, L)
HSIC is test statistic for determining whether two sets of variables independent. The empirical estimator of HSIC is defined as:
HSIC (K = (n 1)2 tr (KHLH)
where is the centering matrix H, = In 11T
Ma
Episode 500
Vi
1.0
50
0:25
0.31
0.24
0.39
73
0.68
0.79
0.67
0.75
0.40
CKA Similari
Layer
0.6
D
0.59
0.68
0.44
0.74
0.13
0.4
0.68
0.79
0.50
0.76
0.17
0.2
0.95
0.90
0.71
0.70
0.20
0.0
-10
500
1000
1500
2000
2500
3000
Training Episodes
Layer
Heatmap at point
Episode
Episode 2000
Episode 3000
1.0
1.0
1.0
0.20
0.12
0.00
0.00
0.99
0.18
0.22
0.26
0.15
0.82
0.19
0.05 0.0 0.00
0.97
0.55
0.91
0.92
0.96
0.00
0.79
0.85
0.87
0.38
0.25
0.57
0.97
0.94
0.93
0.00
Layer
0.96
0.97
0.79
0.45
0.15
0.57
0.97
0.97
0.97
0.00
0..4
-0.4
0..4
0.68
0.96
0.96
0.96
0.17
0.80
0.90
0.89
0.49
0.20
0.70
0.97
0.94
0.93
0.09
0.2
0.2
0.2
0.97
0.68
0.58
0.55 0.29
0.97
0.80
0.79
0.39
0.15
0.97
0.70
0.58
0.55 0.31
3 Layer
1
4
5
Layer
Layer
Heatmap at point B
Heatmap at point C
Heatmap point D
The training graph and CKA similarity heatmaps of a MaxminDQN agent with neural networks. The letters the plot show the time when CKA similarities were calculated. Heatmaps at and C. have relatively low CKA similarity and have relatively higher average return as compared to heatmaps point B and D that have extremely high similarity across all the layers.
Figure 1:
4.2 CORRELATION BETWEEN PERFORMANCE AND REPRESENTATION SIMILARITY
The work in this paper starts from the conjecture that high representation similarity between neural networks in an ensemble-bat based Q-learning technique correlates to poor performance. To empirically verify our hypothesis, trained MaxminDQN agent with neural networks the Catcher
4

---

Under review as a conference paper at ICLR 2021
environment (Qingfeng 2019) for about 3000 episodes (5 x 10t training steps) and calculated the CKA similarity with linear kernel after every 500 episodes. The training graph along with the CKA similarity heatmaps shown in Figure Notably at episode 500 (heatmap A) and episode 2000 (heatmap C), the representation similarity between neural networks is low but the average return is relatively high. In contrast, episode 1000 (heatmap B) and episode 3000 (heatmap D) the representation similarity is highest but the average return is lowest.
Additionally, in Appendix A. we performed a regression experiment to demonstrate that when two neural networks trained on same data, despite having different architecture, learning rate and batch size can learn almost identical representations. This experiment also demonstrates that random initialization of neural networks enforces diversity is misconception.
4.3 REGULARIZATION FOR MAXIMIZING REPRESENTATION DIVERSITY
In order to maximize the representation diversity, we propose to regularize the training algorithm with an additional criteria that favors diversity in the representation space. In the following, N the number of neural networks in the ensemble, é; is the L norm of the i-th neural network's parameters, i the mean of all the 2 norms and l the list of all the T,2 norms.
The first four metrics we consider are based on inequality measures from economic theory. While in economics, inequality usually considered something be avoided, in our case we aim to increase inequality (and thus, representation diversity).
The Atkinson Index (Atkinson et 1970) measures income inequality and useful in identifying the end of the distribution that contributes the most towards the observed inequality. Formally, it is defined as
d-
for 0 S cat F 1.
N
A
-=1
(3)
for Eat 1,
where eat is the inequality aversion parameter used tune the sensitivity of the measured change. When = 0. the index is more sensitive to the changes at the upper end of the distribution, while the index becomes more sensitive towards the change at the lower end of the distribution when cat approaches 1.
The Gini coefficient (Allison 1978) is statistical measure of the wealth distribution or income inequality among population and defined the half of the relative mean absolute difference:
N 1=
j=1 & Egl
G
2N
The Gini coefficient more sensitive to deviation around the middle of the distribution than at the upper or lower part of the distribution.
The Theil index (Johnston] 1969) measures redundancy, lack of diversity, isolation, segregation and income inequality among population. Using the Theil index is identical to measuring the redundancy in information theory, defined as the maximum possible entropy of the data minus the observed entropy:
Tr
(5)
The variance of logarithms (Ok Foster 1997 is widely used measure of dispersion with natural links to wage distribution models. Formally, it is defined as:
1 Vi(() N
In l, In 2())²
(6)
where g(6) is the geometric mean of f defined
( 1/
5

---

Under review as a conference paper at ICLR 2021
The final regularization method method (Boyd et al.) 2011,, a number of models are independently optimized with their own task- specific parameters, and the tasks communicate via penalty that encourages all the individual solutions to converge around a common value. Formally, it is defined as
is inspired from consensus optimization. In consensus
M = f,1?
We will refer this regularizer as Mean Vector throughout this paper.
4.4 TRAINING ALGORITHM
Using the regularization functions defined above, we can develop diversity-regularize variants the MaxminDON and EnsembleDQN algorithms. The training technique is identical to the algorithms described in (Lan et al., 2020) and (Anschel et al. (2017), with a regularization term added to the loss of the Q-functions. The loss term for 2-th Q-function with parameters 1 is:
C (w:) = Bs..,. [ (s. (s) - XI (L,
where is the target value calculated using either Equation (T) or Equation depending on the algorithm, I is the regularizer of choice from the list above and À is the regularization weight. Notice that the regularization term appears with a negative sign, as the regularizers are essentially inequality metrics that want maximize. For completeness, the algorithm shown in Appendix B
EXPERIMENTS
5.1 TRAINING CURVES
We chose three environments from PyGames(Qingteng 2019) and MinAtar (Young & Tian 2019): Catcher, Pixelcopter and Asterix. These environments were used by the authors in the Max- minDQN (Lan et al. 2020) paper. We reused all the hyperparameter settings from (Lan et al. 2020) except the number neural networks, which we limited four and trained each solution for five fixed seeds. For the regularization weight X, we chose the best value from {10- 10 10 10 $. The baselines were also fine-tuned. The complete list of training parameters can be found Appendix F
60
DQN
BQN
DOON
DOON
Average return
Avera geeraagen PUU
50
VOL-Maamin
Attineca Nacnin
30
40
Thai Ensemble
Average
Theil Bull--nnembtt
Maxmin
20
20
tnuemble
Gni-Masmh
VOL Ensemble
0.5
Training Episodes (xx1
Training Episodes (x109)
Training Episodes (x109)
(a) Catcher
(b) PixelCopter
(c) Asterix
Training curves and 95% confidence interval (shaded area) for the best augmented variants for MaxminDON and EnsembleDON together with baseline algorithms.
Figure 2:
Figure shows the training curves for the three environments. To avoid crowding the figures, for each environment and baseline algorithm (MaxminDQN and EnsembleDQN) we only plotted the regularized version which performed the best. We also show as baseline the original MaxminDQN and EnsembleDON, well as the DQN and DDQN algorithms. For the Catcher environment, both Gini-MaxminDON and VOL-EnsembleDON were able to quickly reach the optimal performance and stabilized after 2 x 10 training steps while the baseline MaxminDQN reached its maximum performance after 3.5 x 10 training steps but went down afterwards. Similarly, the baseline EnsembieDON reached its maximum performance after 4 10 training steps, with the performance fluctuating with continued training. For the PixeICopter environment, VOL-MaxminDON and Theil- EnsembleDON were slower in the initial part of the learning that some of the other approaches, but

---

Under review as a conference paper at ICLR 2021
over time they achieved least double return compared the other approaches. Similarly, for the Asterix environment, Rokinson-MaxminDea
and Theil-EnsembleDOy lagged in training for about 1 x 10 training steps but after that they achieved at least 50% higher return compared to the baselines. Full results together with CKA similarity heatmaps are shown in Appendix C
5.2 T-SNE VISUALIZATIONS
To visualize the impact of the regularization, Figure shows t-SNE (van der Maaten Hinton 2008) visualization of the activations of the last layer of the trained networks. Figure Ba show the network trained for the Catcher environment, while Figure 3b] the network trained for the PixelCopter environment. The upper row of the figure shows the original, unregularized models, while the lower row a regularized version. For all combinations, we find that the activations from the original MaxminDQN and EnsembleDQN versions do not show any obvious pattern, while the regularized ones show distinct clusters. An additional benefit of I-SNE visualizations over CKA similarity heatmaps is that the CKA similarity heatmaps useful show representation similarity between two neural networks, but they become counter intuitive as the number of neural networks increases. More t-SNE visualizations for four neural network experiments are shown in Appendix C.3
B Baseline MoxminDON
Baceline ErsembleDQN
Baseline MaxminDON
Baseline EnsembleDDQN
MeanVector MN
Gini EnsembleDDQ
(a) Catcher
(b) PixelCopter
Figure 3: Clustering last layer activations from Catcher and PixelCopter after processing them with t-SNE map them in 2D. The regularized variants have visible clusters while the baseline MaxminDQN and EnsembleDQN activations are mixed together with no visible pattern.
5.3 STATISTICAL ANALYSIS
What is the impact of the regularization the performance? Similar to the approach taken by (Liu et al., 2022)), we performed a z-score test to rigorously evaluate the improvement of regular- ization over baseline solutions. The z-score is also known as "standard score", the signed fractional number of standard deviations by which the value of a data point is above the mean value. A regularizer's z-score roughly measures its relative performance among others. For each algorithm, environment and neural network setting, we calculated the z-score for each regularization method and the baseline by treating results all the results as populations. For example, find out which EnsembleDQQ with two neural networks is best for the Catcher environment, we took the average reward of 10 episodes for each experiment ((5 + 1) x seeds) and treated as population. Finally, averaged the z-scores to generate the final result presented in Table I In terms of improved performance, all the regularizers have achieved significant improvement over the baselines for all three environments. The z-scores for four neural network experiments shown in Appendix C.4
Is the improvement statistically significant? We collected the 2-scores from the previous section and performed the Welch's r-test with the corresponding z-scores produced by the baseline. The resulting v-values are presented in Table 2 From the results, we observed that the improvement introduced from regularization is statistically significant (p < 0.05) in almost all the cases.
7

---

Under review as a conference paper at ICLR 2021
Table Averaged s-scores for each regularization method.
Reg
Envemble N=2
Maxinin N=2
Ensemble N=3
Muxmin N=3
Catcher
Copter
Asterix
Calcher
Copler
Asterix
Catcher
Copter
Asterix
Catcher
Copler
Asteris
Baseline
-2.125
-2.044
-1.68
-2.143
-2.031
-1.730
-1.931
-2.042
-1.634
-1.702
-2.115
-1.478
Alkinson
0.187
0.353
0,402
0.459
0.379
0.292
0.795
Gini
0.659
0.446
0.349
0.319
0.068
Mean Vector
0.426
0.359
0.372
0.529
0.079
0.315
0.403
0.763
0.227
0.347
0.316
0.174
Theil
0.425
0.358
0,767
0.198
0.576
0.364
0.402
0.284
0.429
0.341
0.499
0.068
VOL
0.433
0,315
0.341
0.522
0,526
0.392
0.397
0.471
0.149
0.332
0.297
0.372
Table P-values from Welch's f-test comparing the 2-scores of regularization and baseline
Reg
Ensemble N=2
Maxmin N=2
Ensemble N=3
Maxmin N=3
Cateher
Copter
Asterix
Catcher
Copter
Catcher
Copter
Asterix
Copter
Asterix
Atkinson
0.002
0.000
0.003
0.000
0.000
0.019
0.000
0.000
0.061
0.003
Gini
0.002
0.000
0.001
0.000
0.000
0.000
0.002
0.003
0.003
0.061
0.000
0.013
MeunVector
0.002
0.000
0.000
0.000
0.008
0.016
0.019
0.000
0.000
0.061
0.000
0.030
Theil
0.002
0.002
0.002
0.000
0.000
0.002
0.019
0.001
0.006
0.061
0.001
0.026
VOL
0.002
0.000
0.000
0.000
0.005
0.001
0.019
0.000
0.017
0.060
0.000
0.008
6 IDENTICAL LAYERS EXPERIMENT
To test the limits of the regularizers, we initialized, each layer of each neural network with the same fixed seed. This initialization enforces maximum representation similarity and is considered the worst case scenario for ensemble based learning methods. We performed this experiment on all three environments and used the same seeds and hyperparameters that were used for the main experiments. The training curves are shown in Figure 4 Notably, the results from the baseline MaxminDQN and EnsembleDOON on both Catcher and PixelCopter environments similar to the main results. For the Catcher environment, both Mini-MaxminDQN and Theil-EnsembleDOg were slow in learning for about 2 x 10t training steps but both solutions were able achieve the optimal performance by the end of training. Similarly for PixelCopter environment, the VOL-MaxminDQN was slow in learning till 1.5 x 10 training steps but was able outperform the baseline results and achieved optimal performance. The complete training plots for these experiments are shown in Appendix D
DON DOON
n0
DQR DOON
DQN BOON
40
Average return
400
Maxmin
Ensemble
60 10000
Ensembll Masmin
Macnio
Ensemble bis
-00
pveregee
30
Average
desemble
20
100
20
20
0.5
15
4 Training Episodes (x109)
Training Episades (x 10f)
Training Episades (x106)
(a) Catcher
(b) PixelCopter
(c) Asterix
Figure 4: Training plots representing the best results from each solution for Catcher, PixelCopter and Asterix environment when the layers of the neural networks were initialized with one fixed seed.
7 CONCLUSION
In this paper we showed that high representation similarity between the Q-functions in ensemble based Q-learning algorithms such MaxminDQN and EnsembleDQN leads to decline in learning performance. To mitigate this, we proposed a regularization approach using five different metrics to maximize the diversity in the representation space of the Q-functions. Experiments have shown that our solution outperforms baseline MaxminDQN and EnsembleDQN standard training settings as well as in scenarios where the parameters of the neural layers were initialized using fixed seed.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. Striving for simplicity off-policy deep reinforcement learning. arXiv preprint 1rXiv:1977005443, 2019.
Paul D. Allison. Measures of inequality. American Sociological Review, 43(6))865-880, 1978. ISSN 00031 htt.hhttp//wwtt www.:wsstrr.... wwt.ss..or//sttoeeeeeeeeeeee
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-DQN: Variance reduction and stabilization for deep reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML-2017), pp. 176-185, 2017.
Anthony B Atkinson et al. On the measurement of inequality. Journal of economic theory, 2(3): 244-263, 1970.
Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3112 2011.
Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via q-ensembles. arXiv preprint 1rXiv:/706.07522, 2017.
Richard Cheng, Abhinav Verma, Gabor Orosz, Swarat Chaudhuri, Yisong Yue, and Joel Burdick. Con- trol regularization for reduced variance reinforcement learning. arXiv preprint arXiv: 19X5v::905.050000 000305 2019.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems, pp. 4754 4765, 2018.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered aligmment. Journal of Machine Learning Research, 13(Mar):795-822 2012.
Nello Cristianini, John Shawe- Shawe-Taylor, Andre Elisseeff, and Jaz Kandola. On kernel-targee aligmment. In Proceedings of the Advances in Neural Information Processing Systems (NIPS-2002), pp. 367- 373, 2002.
Carlo D' E'Ermoo, Alessandro Nuara, Matteo Pirotta, and Marcello Restelli. Estimating the maximum expected value in continuous reinforcement learning problems. In Proceedings of the Conference on Artificial Intelligence (AAAI-2017), 2017.
Jesse Farebrother, Marlos C. Machado, and Michael Bowling. Generalization and regularization in DQN, 2018.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint 1rXiv:1802.09477 2018
Alexandre Galashov, Siddhant Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojeiech Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry in KL-regularized RL. In Proceedings of the International Conference on Learning Representation (ICLR-2019)) 2019.
Jordi Grau-Moya, Felix Leibfried, and Peter Vranex. Soft Q-learning with mutual-information regularization. In Proceedings of the International Conference on Learning Representations (1CLR-2019), 2019.
joCqFX
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In Algorithmic Learning Theory (ALT), pp. 63-77, 2005.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbecl, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv: / 01090 2018.
Hado Hado van Hasselt, Arthur Guez, and David Silver. Deep Reinforcement Learning with Double Q-learning. In Proceeding of Conference on Artificial Intelligence (AAAI-2016), 2016.

---

Under review as a conference paper at ICLR 2021
J. Johnston. H. Theil. Economics and Information Theory. The Economic Journal, 796315):011-02. 09 1969. ISSN 0013-0133, doi: 10.2307/2230396. URL htttps://do..org///
2230396
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Proceedings of the International Conference on Machine Learning (ICML-2019), pp. 3519-3529, 09-15 Jun 2019.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping enror reduction. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS-2019) pp. 11761-11771, 2019.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations, 2018.
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin Q-learning:
Controlling the estimation bias of Q-learning In Proceeding of the International Conference on Learning Represen- tations (ICLR-205 2020), 2020. URL https////ppeeeeveeeeeeee net/ wtr:m/oormmmmim/mmm hott:///oorrnn11.B00000
Donghun Lee, Boris Defourny, and Warren B. Powell. Bias-corrected Q-learning to Control Max- operator Bias in Q-learning- In IEEE Symposium on Adaptive Dynamic Programming and Rein- forcement Learning. pp. 93-99, 2013.
Kimin Lee, Laskin Michael, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. arXiv preprint 04X 2020.
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? In Proceedings of the International Workshop on Feature Extraction: Modern Questions and Challenges at NIPS 2015, volume 44, pp. 196 212, 2015.
Zhuang Liu, Xuanlin Li, Bingyi Kang, and Trevor Darrell. Regularization matters in policy optimiza- tion. arXiv preprint arXiv:1910.09191, 2020. URL htttps:/ hwt..nee//ee.. forum?
id=BllgDertwr
P. Lv, X. Wang, Y. Cheng, and Z. Duan. Stochastic double deep q-network. IEEE Access, 7: 79446-79454, 2019. ISSN 2169-3536.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 February 2015. ISSN 00280836.
weighted truncated-svd canonical correlation analysis. CoRR, 0ob//1511.062665 2015. Youssef Mroueh, Etienne Marcheret, and Vaibhava Goel. Multimodal retrieval with asymmetrically
Efe A. Ok and James Foster. Lorenz Dominance and the Variance of Logarithms. Working Papers 97-22, C.V. Starr Center for Applied Economics, New York University, 1997. URL https r/ideas.repec.
repec. wttppp..rgg/pcvss///eeo. http://vv//saarrr/9999 htmml
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pp. 4026-4034, 2016.
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Ro- bust deep reinforcement learning with adversarial attacks. In Proceedings of the International Conference Autonomous Agents and MultiAgent Systems (AAMAS-2018), pp. 2040 2042, 2018.
Lan Qingfeng. Gym compatible games for reinforcenment learning. https: com/ gan3//ymmgammss 2019.
10

---

Under review as a conference paper at ICLR 2021
Maithra Raghu, Justin Gilmer, Jason Yosinski. and Jascha Sohl-Dickstein Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Proceedings of the Advances in Neural Information Processing Systems (NIPS-2017), pp. 0076-6085, 2017.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR-2016). 2016.
James E. Smith and Robert Winkler. The optimizer's curse: Skepticism and postdecision surprise in decision analysis. Management Science, 52(3):311-32) 2006.
Alexander L. Strehl, Lihong Li, and Michael L.. Littman. Reinforcement Learning in Finite MDPs:
PAC Analysis. Journal of Machine Learning Research (JMLR), 10(00))22113-2444 2009.
István Szita and András Lórincz. The Many Faces of Optimism: A Unifying Approach. In Proceed- ings of International Conference on Machine Learning (ICML-2008), pp. 1048-1055, 2008.
Richard Thaler. The winner's curse: Puradoxes and anomalies of economic life. Cambridge University Press, 2012.
Sebastian Thrun and A. Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the Connectionist Models Summer School (CMSS-1993), January 1993.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation the real world. In Proceed- ings of the International Conference on Intelligent Robots and Systems (IROS-2017), pp. 23-30, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Ma- chine Learning Research, 22579-2605. 2008. http:://// jmlr r prg/papers/v9/
wandermaaten008e html
Hado van Hasselt. Double Q-learning In Proceedings of the Advances in Neural Information 2010.
Processing Systems (NIPS-2010), pp.
Liwei Wang. Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, and John Hopcroft. Towards understanding learning representations: To what extent do different neural networks learn the same representation. In Proceedings of the Advances Neural Information Processing Systems (NIPS-2018), pp. 0004-9993, 2018.
Chris Watkins. Learming from Delayed Rewards. PhD thesis, King's s College. Cambridge, 1989.
Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint 1903.033 2019.
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. A study on overfitting in deep reinforcement learning. arXiv preprint arXiv:/804.00 06893, 2018.
Zongzhang Zhang, Zhiyuan Pan, and Mykel J Kochenderfer. Weighted double Q-learning. In
Proceedings of the International Joint Conferences on Artificial Intelligence (IJCAI-2017), pp. 3455-3461, 2017.

---

Under review as a conference paper at ICLR 2021
A SUPPLEMENTARY MATERIAL
A.1 MOTIVATING EXAMPLE TO DEMONSTRATE SIMILARITY BETWEEN NEURAL NETWORKS
We performed regression experiment in which learnt a sine wave function using two different three layered fully connected neural networks with 64 and 32 neurons in each hidden layer with ReLU. The neural networks were initialized using different seeds and were trained using different batch sizes (512, 128) and learning rates (1e-4, 1e-3). The Figure 5a the learnt functions while Figure 5b represents their CKA similarity heatmap before and after training. The odd numbered layers represent pre- Ure-ReLU activations while the numbered layers represent post-ReLU activations. It can be that before training, the CKA similarity between the two neural networks from layer 4 and onward is relatively low and the output being 0% similar while after training, the trained networks have learnt highly similar representation while their output being 98% similar.
Before Training
After Training
0:55
0.53
Simlarity
0.44
6.86
8.
0.64
0.80
9.46
0.00
Layer
0.95
2.09
0.50
5.76
3.75
0.04
052
0.05
2.54
0.03
5.56
0.85
0.05
0.25
0.4
0.50
3.00
0.59
0.72
0.62
0.97
3.00
8.64
0:65
0.61
0.29
0.2
1.00
5.57
0.27
0:52
0:43
022
1.00
0.57
0.22 0.31
0.0
1
2
1
Layer
Layer
(a) Regression using two different (b) CKA similarity heatmap between different layers of the two neural neural networks
networks used for the regression experiment.
Figure 5: Left: Fitting sine function using two different neural network architectures. The upper function was approximated using 64 neurons in each hidden layer while the lower function used 32 neurons in each hidden layer. Right: Represents the CKA similarity heatmap between different layers of both neural networks before and after training. The right diagonal (bottom left top right) measures representation similarity of the corresponding layers both neural networks. The trained networks have learnt similar representations while their output was 98% similar.
This example shows that neural networks can learn similar representation while trained on differ- ent batches. This observation is important because in MaxminDQN and EnsembleDQN training, each neural network trained on a separate batch from the replay buffer but still learns similar representation similarity (see Figure
12

---

Under review as a conference paper at ICLR 2021
B ALGORITHMS
For completeness, the regularizered MaxminDQN and EnsembleDQQ algorithms are given below
Algorithm 1: Regularized MaxminDQN
The differences between the baseline MaxminDQN and regularized MaxminDON are highlighted Initialize N Q-functions (Q'....: Q^ parameterized by {wh,,..., (NN
Initialize empty replay buffer
Observe initial state
while Agent is interacting with the Environment do
Qmin (s,a) a)
(I...,,) (s, Ya A
Choose action by e-greedy based Qmin
Take action a, observe s
Store transition (s, a, s') in D
Select subset S from [I.....N} NN (e.g., randomly select one to update)
for do
Sample random mini-batch of transitions (SD. aD, TD. TD,
sp)) from D
Get update target: Y. + TD + max Qm (8 a')
Generate list of 2 norms l = |1b1l we..l?....,II 1.
Update Q' by minimizing Esp (0%s (n), -yM)² -XI (li,
end + s
end
Algorithm 2: Regularized EnsembleDQN
The diffcrences between the baseline EnsembleDQN and regularized EnsembleDQN are highlighted
Initialize N Q-functions "Q'...., QN} parameterized by (6/,,..,,N7 Initialize empty replay buffer D
Observe initial state
while Agent is interacting with the Environment do
Qen N
1
EN (s,
Choose action by e-greedy based on Qens
Take action a, observe r, s'
Store transition (s, 8') 3 in D
Select a subset from [I.....N) (e.g., randomly select one to update)
for i E S do
Sample random mini-batch of transitions (sn, an, rD, s'p) from
Get update target: YE +- rp + max Qens (8' a.)
Generate list of L norms l |1.11?..... II...
Update Q' by minimizing Esp.ap.r..fy (0), (sD, YE)² -XI
end
+- s
end
13

---

Under review as a conference paper at ICLR 2021
C ALL TRAINING PLOTS
C.1 TRAINING PLOTS FOR MAXMINDON
60
Average return
30
Maxmin Nec
Average Return
Moxmin Nod
Maxmin
Atsinesn N--2
Andincon M-2
Alkinson Ned
Atddneenn N..3
20
Maxmin Ne3
Maxmiin No2
Maxminn
Average 40
Atfenson
30
20
-10
Training Episodes (x109)
Training Episodes (x10f)
Training Episodes (x10%)
80
Naxmin
Maxmle
Average return
Aver ge return
Moxmin Ne3
Maseeti Hidd
Average Return
Moxmin Nw3
Maxmin Hi-d
60
Giel 2
Giee N-
N
Macmin N..2
30
Macmin
Musmin tews
20
20
Gini Niod
-10
Training Episodes (x 10
Training Episodes (x109)
Training Episodes (x10f)
80
Maxmin
Maxmin H-3
Average return
Maxmin
Returm 30
Macmin Ne44
MeanVector 2
Neanivector Hi.
40
MaanVeecor
MeanVeettrr
Average
Meanieector Rock
Maamin 2in2
30
Maxmin tee
Meantiecto Nad
MeariVector Ne4
Training Episodes (x10f)
Training Episodes (x10%)
Training Episodes (x109)
80
Naxmin 2
Maxmin H-2
Maxesin Huck
Maxmin Nuck
Average return
Average return
60
Macein No44
Thail
50
Maxmin N-44
Thail
Thail
Normin Rox
40
Average
20
20
1.0
Thail Hil
Thail Rea
Training Episodes (x10%)
Training Episodes (x105)
Training Episodes (x10$)
80
Maxmin N
Masmin Hul
Haceein H..
Moxmin Ne3
Average return
Maxmin
50
Maomia
40
Nocmin Noc
Avres overa age
40
Average
100
20
20
VOL Ni-3
VOL Hod
0.5
15
Training Episodes (x10f)
Training Episodes (x10f)
Training Episodes (x10f)
Catcher
PixelCopter
Asterix
Figure 6: All MaxminDON Results. Top to Bottom: Atkinson, Gini, Mean Vector, Theil, Variance of Logarithms
14

---

Under review as a conference paper at ICLR 2021
C.2
TRAINING PLOTS FOR ENSEMBLEDON
80
60
Average return
Averagg age Com
Entemblle
Entambln N
60
Alkinsen
Atlinsen M=2
Atsinson N-3
Addmssc H-3
Alkinsen
Average
Alkinsen Mod
20
Ensemble fies
Entemble N.J
2000
Entemble Ni-4
20
othinssn Hwz
Attiinoon
10
-10
Training Episodes (x10f)
Training Episodes (x10f)
Training Episodes (x109)
60
Average return
50
Average return
Enseibble
Ensemble
Gini 2
Gini M
Ginl N-3
40
Giel N-
20
Ensemble few2
Enzemble 1i-3
Ensemble Nesk
40
26
Hidd
Average
20
20
Ni.d
-100
Training Episodes (x109)
Training Episodes (x10f)
Training Episodes (x10f)
Average return
3.0
age retur
Ensembble Row
50
Entemblle
Ensemble H4
Meanken H
Meanifecor Ro2
Neatieector
40
MeanVeector
20
Ensemble H-2
Enaemble N-3
Bncembble R-d
Aver Average
20
MeariVectrr
Average
20
MoanVuctor
MeanVecter
10
-10
Training Episodes (x109)
Training Episodes (x10f)
Training Episodes (x10f)
80
Entembble
Average return
50
Avera age return
600
Ensemblee Red
Greembll R.d
Thril Now2
50
Ensembil H.
Thei New
Ensemble H-
Thail
Thail
40
40
20
Encemble Noz
Ensumble fow3
Enzemble
20
20
Thai
10
Thai Hidd
-10
Training Episodes (x 100)
Training Episodes (x10%)
Training Episodes (x100)
Entamble Ensemble Rid
Entembll
Ensemble
Average return
Average return
Eresumbbee 64
50
Ensemble H
VOL Roiz
VOL Roc2
Ro-3
30
No-3
Average
Ensumble Tew2
30
Enzembla Ni.3
10
20
VOL Ri4
-10
Training Episodes (x109)
Training Episodes (x10²)
Training Episodes (x10*)
Catcher
PixelCopter
Asterix
Figure 7: All EnsembleDQN Results. Top to Bottom: Atkinson, Gini, Mean Vector, Theil, Variance of Logarithms
15

---

Under review as a conference paper at ICLR 2021
C.3 HEATMAPS AND T-SNE VISUALIZATIONS
Sepureereeeeeentti the CKA similarity heatmaps using a linear kernel of all the two neural network experiments after training averaged over all the five seeds. The point of interest is the right diagonal (bottom left to top right) that represents the representation similarity between corresponding layers. For the baseline experiments, the output layer has more than 96% similarity in almost all the scenarios while for the regularized versions have around 90% similarity in the output layer. This 10% difference provides enough variance in the Q-values to prevent the ensemble based Q-learning methods converging to the standard DQN.
Calccher
Copter
Asterix
Catcher
Copter
Asterix
0.23
0:23
0.30
0.50
0.00
0.0%
0:05
3.04
0.97
0.05
0:33
0.22
8.51
5.66
0.78
0.75
0.70
0.24
0.99
0.01
0.06
5:17
6:12
0.97
0.12
0:15
0.21
0.50
0.98
Baseliiee
0:47
026
0.54
5.57
0.04
3.24
0:65
0.55
0.78
0.22
0.40
0.45
027
024
0.63
0.57
0.52
0.93
9:91
0.06
0.87
0.35
0.59
0.85
2.58
2:40
2.24
0:42
5.09
0.95
0.82
0.02
0.27
0.53
0.82
0.54
5.02
0.98
3.64
0.30
5.38
0.12
0.26
0.78 0.40
0.40
0.89
5.00
0:02
5:11
3:15
0.05
0.04
0:03
0.15
0:32
5.64
0.27
0.77
0:17
0.44
8.89
0.05
0.47
5
0.00
0.92
0.05
3.05
0:51
0.78
0.92
Akkimson
0.70
0.2
0.54
0.12
3.
6:64
0.74
0.70
0.24
0.02
0.41
3.43
0.02
0.47
6:67
0.22
0.67
0:25
0.48
0.37
0.81
0.30
0.33
0884
3:42
3:43
8:83
0.54
0.03
0.81
0.79
0.22
0.52
0.63
3:43
0:53
0.93
0.64
0.52
0.09
0:23 0.22
0:25
0.88
0.98
0:05
3.55
3:55
0.89
0.08
0.03
0:24
0:27
0.68
0.50
0.35
0.23
0.54
0.38
0.08
0.07
3:11
0:05
0.52
0:05
3.09
0.33
0.91
2.79 0.85
0.69 0:53 0:14
6.32
0:54
537
0:75
0.14
0.52
0:63
0.59
0:82
2.32
0:70
0.73
0.46
0.23
0:42
0.62
0.82
3.79
0:55
5.07
0.44
8.52
0:00
5.57
0.34
Gini
0.52
5.25
5:42
0.15
0.97
0.37
0.63
0.07
4:42
0.54
0.92
0:33
0.48
0.14
0.06
0.81
0.30
0.14
0.93
3:48
2.05
8:85
0:56
2.09
0.08
0.32
0.73
0.39
0.70
0.69
3:42
0:46
0.92
0.35
0.09
0.28 0.24
0.08
0:32
0.89
0.05
5.03
3:55
0.59
2.57
0.02
5:36
0:33
5.52
2.25
0.25
0.20
0.24
0.92
0.36
0.13
8:15
0:48
675
0.09
3.09
0:23
0.91
MeanVector
2.75 0.72
0.47
0:53
6.39
0.75
3.54
2.53
0.00
2.27
8:63
0.89
0:83
0.30
5.75
0.52
0.4
0.63
0:30
0.23
0.27
3.20
3:70
5.00
0.14
0:59
0.95
0.00
0.25
0.88
0.27
0:42
5.20
0.0%
0.97
0.20
0.67
0:25
0.99
3:59
0.10
3.94
0.78
0:32
0.90
30
0.00
9.56 0.95
0.79
0.35
0.90
3.99
0:38
85
0.26 0.22
0.99
0.03
3.55
3.39
0.92
2.04
0.02
0.24
0:25
5.62
0.28
0.24
0.29
0.23
0.92
0.98
0.06
3.10
0:05
3.67
0.05
3.09
5.20
0.91
0.90
0.62
0:72
0:22
0.54
8:65
3.68
2:08
0.09
0.48
0.51
0:82
0:34
0.51
0.57
0.75
0.00
0:25
0.51
0.88
3.52
0.54
5.07
0.45
8:53
0.95
0.50
0.22
Theil
0.08
0:32
5.42
0.05
0.64
0.30
0.65
0:42
0.59
0.89
0.48
0.10
0:32
0:51
6.64
8:55
3.54
0.63
0.94
0.85
0.36
0.66
0.71
0.48
0.93
8.84
0:57
0.24...2
8.47
0:1
6.9
0.10
0.09
0.02
6:13
0:21
0.99
5.20
0.76
0.25
0.90
0.06
0.67
0:08
3.92
0.05
5:35
0.99
0:91
0.82
0.61
0:1
6:34
8:28
6.65
8.69
6.10
5.47 6.6)
0.91
0.82
0:33
5.80
0.86
0.80
0.57
0:21
6.53
0.90
3.60
0.8
5.66
0.45
6.63
0.85
0.65
0.31
VOL
0:34
0.79
0.06
0:60
044
0.70
45
9.49
0.09
0.49
0:63
0.93
0.18
0.58
0.85
0.81
0.65
0.22
0.63
0.93
9.
0.81
6.66
0.50
9:69
0.95
0.52
0.16
0.70
033
0:08
9:52
0:83
0.52
0.63
0.56
0:39
0,62
0.60
3.39
0.93
96
6.5
(a) EnsembleDQQ
(b) MaxminDQN
Figure& 8: Heatmaps representing the CKA similarity of 2 neural network experiments.
16

---

Under review as a conference paper at ICLR 2021
Figure 9 represents the t-SNE visualizations of the baseline and regularized solutions trained with four neural networks on the PixelCopter environment. This visualization is consistent with the visualizations shown in Section (5.2 where the baseline activations cluttered without any pattern while the Theil-MaxminDON and Theil-EnsembleDON activations have visible clusters.
Baseline MaxminDQN
Baseline EnsembleDQN
Theil-MaxminDON
Theil-EnsembleDON
Figure 9: Clustering last layer activations from PixelCopter after processing them witht-SNE to map them in 2D
C.4 7-SCORE TABLE FOR FOUR NEURAL NETWORK EXPERIMENTS
Table 3: Averaged z-scores for each regularization method with four neural networks
Reg
Ensemble N=4
Maxmin N=4
Catcher
Copter
Asterix
Catcher
Capter
Asterix
-2.096
0.463
Gini
0.409
0.307
0.126
0.340
0.433
0.000
Mean Vector
0.452
0.685
0.019
0.362
0.206
0.351
Thoil
0.341
0.669
0.596
0.363
0.842
0.384
VOL
0.409
0.374
0.141
0.363
-0.167
0.252
Table P-yalues from Welch's 1-test comparing the z-scores of regularization and baseline
Reg
Ensemble N=4
Maxmin N=4
Catcher
Copter
Asterix
Carcher
Copter
Asterix
Atkinson
0.002
0.005
0.004
0.044
0.018
0.012
Gini
0.002
0.005
0.005
0.045
0.024
0.056
MecnVector
0.002
0.000
0.016
0.044
0.010
0.020
Theil
0.001
0.001
0.007
0.044
0.005
0.014
VOL
0.002
0.003
0.006
0.044
0.024
0.030
17

---

Under review as a conference paper at ICLR 2021
D IDENTICAL LAYERS EXPERIMENT (ALL TRAINING PLOTS)
D.1 TRAINING PLOTS FOR MAXMINDQN
Aver return
30
Aver Average reture
Maxmin Red
Maxmin
40
Moxmin Rod
Maxmin
Hosmin
Ankinson M--2
Atdinscn M-2
Masrin Rwd
Alkimson Ite
Atddneenn N
Masmin Wi-d
40
Adkineonn Mid
20
N-22
Average 20
20
10
-10
Training Episodes (x10%)
Training Episodes (x10f)
Training Episodes (x10%)
80
Average return
50
Mosmin N-2
Average return
Maxmin Midd
Gle Hoz
20
Moxmin Nw3
Gins No
Maxmin Hidd
Hasmin Red
Girt H-3
Masmin Re-d
40
Hed
20
15
20
-10
Training Episodes (x 1x9)
Training Episodes (x109)
Training Episodes (x109)
Maxmin
Maxmin
40
Maxmin H-3
Average return
Average return
Maamin Hed
Macmin N-44
Maomin R--3
Maxmin Noo
Meantruct No2
Meant/act
20
MaanVector
MeanVeeter
Masmin
40
Moantrectoo
Meartiester
Average
20
26
10
2.0
-100
Training Episodes (x109)
Training Episodes (x109)
Training Episodes (x10%)
Maxmin Maamin Hua
40
Maxmie H-2
Average return
Average retura
Masmin N-44
Average Return
Maxmin Ne-4
Maxmin Muck
40
Maxmin Hi-2
Theil
20
Thail
Masmin Rwd
Haamin
40
30
20
1.0
Training Episodes (x10ff)
Training Episodes (x105)
Training Episodes (x10$)
80
Masmin
Masein Hua
Average return
Average return
Haamin Nidd
Maxmin
40
Moxmin Rt3
Masnin
40
Masmin R--2
VOL Ho
VerkencoOfLop
Maimin Rud
40
Average
VOL.
10
VOL.E
20
10
0.5
155
Training Episodes (x110f)
Training Episades (x109)
Training Episodes (x109)
Catcher
PixelCopter
Asterix
Figure 10: All MaxminDQN Results. Top to Bottom: Atkinson, Gini, Mean Vector, Theil, Variance of Logarithms
18

---

Under review as a conference paper at ICLR 2021
D.2 TRAINING PLOTS FOR ENSEMBLEDQN
40
Average return
50
Ensembill
Average return
Ensamble
Entambln Ni-4
Alkinson H=2
60
Albinson
Atlinsen M-2
Anfonson Mi-
Atsinsonn N--3
Addmssc H-3
Atkinsen Hedd
40
Atkinsen
Average 20
Alkinses Mad
20
20
1.
-10
Training Episodes (x 109)
Training Episodes (x10f)
Training Episodes (x100f)
Ensemble
Average return
Ensemble Nowd
Gind H-22
Average return
600
Gini H-2
20
Gisl 2
Giel N-3
Gind Ne3
25
Giti H
-00
Average
20
Ginl No4
25
-10
Training Episodes (x109)
Training Episades (x10f)
Training Episodes (x10f)
Aver return
30
Ensemmble
Ensembble Rood
Average return
Ensembie Rord
Ensorbale
Average Return
Ensemble N-44
N
Maant//ccton M
Heanifector N..2
MeanVector Ni.2
Wearthect Mew
Moantructon New
Meantfectrr Rtd
MoanVuctor
10
20
10
-10
Training Episodes (x10%)
Training Episodes (x10f)
Training Episodes (x105)
Ensemble Rew
Enveemble
Encemble Ensemble Revk Thail N-2
40
Ensemble No
Ensembble
Average return
Encembble Ri-d
Avera Average age return
600
Encemblle bi Hi.d
Theil Mt-2
30
Thei New
Theil Med
Thail
-00
Average
1.0
-10
Training Episodes 10F)
Training Episodes (x109)
Training Episodes (x100f)
Ensemble
80
Entamble 2
Ensemble Rwd
Ensemble Ni-3 Insembtee Roa
Ensemble Nez
Average return
40
Ensembbee Nood
VOL
Average return
60
VOL No
Ensemble H-44
Varinccofl-ag N-2
VartincoOfLag
-0
Average
20
15
10
0.5
Training Episodes (x109)
Training Episodes (x10²)
Training Episodes (x10*)
Catcher
PixelCopter
Asterix
Figure 11: All EnsembleDQN Results. Top to Bottom: Atkinson, Gini, Mean Mcan Theil, Variance of Logarithms
19

---

Under review as a conference paper at ICLR 2021
E IMPLEMENTATION DETAILS AND HYPERPARAMETERS
For our implementation of MaxminDQN and EnsembleIQQN we used the code provided by the MaxminDON authors that has implementations of different DQN based methods jeithub.com//gaan3/EXppooo
For the baseline experiments, we used most of the hyperparame- ter settings provided in the configuration files by the authors except learning rates which we limited to [le - 3, le 4, 3e - 5] and limited the number of ensembles to four. The complete list of hyper- parameters each environment is shown Table The values in bold represent the values used for the reported results.
Table 5: List of hyperparameters used for the experiments.
Hyperparameter
Catcher
PixelCopperr
Asterix
Learning
[1e
3, le 4,
[e
3, le 4, 3e
5]
[1e
3, le
1, 3e
Batch size
332 64
[32, 128, 1024]
[32,
Buffer Size
le4, 1e7]
[le4, 1e6]
[le5, 2eb]
Exploration Steps
[1e3, 2e3
Hidden Layer Size
[64, 64]
[64, 64]
[64, 64]
Gradicat Clip
-1
Discount Factor ?y
0.99
0.99
0.99
Regularization Weight
[Ie 5, 1e 6)
[Ie 6, le 7, 1e 8)
(Ie 5, 1e 6)
For the identical layer experiment, no hyperparameter tuning was performed and we reused the hyperparameters from the main results. In terms of number of experiments, we ran 190 experiments: (5 regularizers x 5 seeds x 3 ensemble settings x 2 algorithms) 40 baseline experiments for each environment totaling 570 runs for all environments after hyperparameter tuning. The same number of experiments were performed for the identical layer experiment which sums up to 1140 runs where each run took 11 hours of compute time on average.
F PLOTTING THE GINI INEQUALITY
We measured the 2 norm inequality of the baseline MaxminDQN and EnsembleDQN along with their regularized versions. We trained baseline MaxminDQN and EnsembleDQN with two neural networks along with their Gini index versions with regularization weight of 10 on the PixelCopter environment on a fixed seed. Figure represents the L norm inequality of the experiments along their average return during training. Notably, despite each neural network being trained on a different batch, the _ norm of the baseline MaxminDQN and EnsembleDQN are quite similar while the 2 norm of the regularized MaxminDQN and EnsembleDON have high inequality.
0.30
80
Maxmin
Ensemble
0.25
Average Return
Gini-Maxmin
Gini Index
60
Gini-Ensemble
Maxmin
0.15
Ensemble
40
Gini-Maxmin
Gini-En Ensemble
0.10
20
0.05
0.00
0.5
1.5
1.5 Training Episodes (x10%)
0.5
2
Training Episodes (x10%)
Figure 12: Left: Piot representing the 2 norm inequality between the two neural networks using Gini index trained PixelCopter environment. Right: Plot representing the average return during training.
20


Reviews:
[{"review_id": "n2uWgpPY_yD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to improve ensemble-based methods such as Maxmin\nand Ensemble Q-learning by regularizing the representations for the\nensemble members to increase diversity. The motivating intuition is\nthat on their own, these methods can still collapse and once they do\nthe benefits of the ensemble are lost. The authors use a variety of\nmeasures of inequality to provide regularization (which increase that\ninequality of representation among ensemble members).\n\nAlthough the hypothesis seems plausible, I don't think the results\nsufficiently support it.\nIn particular, much of the motivation focuses on over-estimation, but\nthis was never evaluated explicitly. I think the experimental results\ncould be strengthened in a number of ways: more domains, single set of\nhyperparameters, further empirical analysis of the effects of the\ndifferent inequality measures used for regularization.\n\nPros:\n\nInteresting idea and seems effective based on experimental results.\nHighly relevant, as ensembles are one of the most commonly used and\nbest methods for a number of use cases in and outside of RL.\n\nCons:\n\nPer-domain hyperparameter tuning\nSmall set of domains\nLimited understanding of  why some regularization methods work better\nthan others.\n\nComments/Questions:\n\nSection 4.1-4.2\nFigure 1 did not convince me about the similarity measure the way I\nthink the authors wanted it to. Just looking at the Heatmaps, I really\nstruggled to see why A and C should be better than B and D. The only\nview on these I could come to was that if you only look at the\ndiagonal components (or center block) of the heatmaps then the\nconnection works. But, why are these the similarities that matter?\nOverall this section (4.2) does not lay the groundwork of motivating\nthe correlation between performance and representation similarity the\nway it was intended to. It seems like this could have been validated\nmuch more directly, by controlling the representations used first, and\nthen measuring performance afterwards.\n\nSection 5.1\nHyperparameters appear to have been tuned per-domain, which makes me\nsceptical about the results as a whole. A more typical approach would\nbe to use a single set of hyperparameters over the set of domains\nconsidered. One reason for this is that doing so penalizes methods\nthat are sensitive to hyperparameter tuning, which should be\nconsidered as part of the algorithm itself.\n\nRegularization does seem effective, but these are very limited\nresults. Why not increase to the full set of MinAtar games?\n\nSection 5.2\nWhat do the colors represent?\nAs this section attempts to give us some idea of the impact of\nregularization (beyond simply performance) I would have really liked\nto get more intuition for the ways the regularizers are affecting the\nrepresentation more directly. The t-snes are nice to show something is\nhappening, but don't give us any insight into how or why. How should\nwe understand the differences in the different methods used? Why\nshould one be preferable over another?\n\n\nTypos:\n\nPage 2, \"The implicit policy \\pi can extracted\" (can be extracted)\nPage 7, \"we calculated the z-score for each regularization method and\nthe baseline by treating results all the results\" (remove the first\n'results')", "extra_scores": {}, "extra_text": {}}, {"review_id": "Y_9vXodaaPo", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes methods to induce diversity in the networks of ensemble-based Q-Learning methods. This is achieved my maximizing a variety of measures of inequality based on the L2 parameter norms of individual networks in an ensemble. This is motivated by the benefit of having diversity in the learned features, which itself is motivated by observations on the CKA of some ensembleDQN networks.\n\nStrengths:\n- The high-level motivation of this work is sound. Diversity within an ensemble is undeniably desirable\n- The proposed methods do improve performance on interesting benchmarks\n\nPossible improvements/Weaknesses:\n- There are missing steps in the chain from motivation to method, but these steps can easily be done and verified by measuring the appropriate quantities systematically.\n- It's not clear exactly what the method is doing, it would be good to actually measure correlations between feature similarities and performance (which the authors claim exists without measuring)\n- It's not clear if the method is truly benefiting from feature diversity, or if the regularization induces some totally different effect, simple baselines and using a variety of feature similarity metrics could address this\n\nI gave this paper a score of 5; while the method is somewhat interesting, the authors need to **quantitatively** show that the regularizations they propose have the effects they claim they have. Otherwise, this is just a \"my number is bigger than your number\"-paper, which isn't very valuable for our scientific understanding of deep RL.\n\n\nDetailed comments:\n- \"The first deep RL algorithm, DQN\", that's not very historically accurate. Martin Riedmiller published Neural Fitted Q Iteration in 2005, Gerald Tesauro published TD-Gammon in 1995, and I don't even know if they're really the \"first\". Even if we're stricter about the notion of \"deep\", in 2010 Lange & Riedmiller successfully trained deep autoencoders to pretrain features used for NFQ, in 2012 Hess & al used RBMs and MLPs to model policies. Perhaps \"landmark\" or \"remarkable\" would be more accurate.\n- The authors emit the hypothesis that \"representation similarity between neural networks in an ensemble-based Q-learning technique correlates to poor performance. To empirically verify our hypothesis...\", but that hypothesis is never verified, showing qualitative plots of two networks with high CKA is very different than making a clear measurement of correlation. For example, a figure with CKA on the x axis and performance on the y axis for repeated runs and a clear correlation line and r^2 value would be a quantitative statement testing the hypothesis. \n- \"that random initialization of neural networks enforces diversity is a misconception\". It's known that different similarity metrics like CCA or maximum matching tend to find that two networks trained with different initializations _do_ converge to different features, while CKA finds they do _not_. It might be valuable to test these metrics in the RL setting as well, I suspect the conclusions/motivation of this paper might differ somewhat.\n- Figure 3 is quite misleading, in the bottom row the individual networks are trained to have different norms, thus it is extremely likely that the activations of the last layer are also going to have different norms. This is the first thing the t-SNE is going to pick on, and in that sense Figure 3 most likely isn't showing that different features have been learned, and it's also likely that if the networks learned similar features with different scales t-SNE wouldn't be able to show it.\n- The argument the authors make about feature diversity is based on CKA, but the methods are based on L2 norms of parameter vectors.\n  1. The link between the two is not made (and not obvious to me)\n  2. Deep ReLU networks are known to be invariant under several reparameterizations, including some rescalings (see Dinh et al, https://arxiv.org/abs/1703.04933). It's not clear to me that two networks that have different L2 norms have significantly different features. It's in fact quite easy to take a trained network and then finetune it only with L2 or L1 without significant loss in performance (presumably the same features are thus kept, just scaled; this is what pruning, quantization, and DNN compression are based on). I thus don't see how diversity in features follows from diversity in L2 norms.\n  3. A crucial missing baseline seems to be to train an ensemble of networks with a varying L2 regularization loss on each network of the ensemble. Another interesting baseline would be to vary the learning rate of each network, since having smaller or larger parameters may in the end only affect the effective learning rate of a particular network.\n- The results of 5.1 and Fig 2 are interesting, but\n  1. it would be interesting to see the average performance of diversity-regularized methods, not just the best ones\n  2. it seems like a downside that no single diversity method is systematically the best, but perhaps it reflects something about the environments used, and could be interesting to investigate.\n- The conclusion claims that \"high representation similarity between the Q-functions in ensemble[s] [..] leads to a decline in learning performance\". The paper only seems to have 4 data points to that effect, Figure 1. This is not a very strong claim.\n- The conclusion claims that the proposed method can \"maximize the diversity in the representation space\", but this is not measured, only hypothesized. The claims that this paper really makes are:\n  - CKA can measure diversity, and ensembles with high CKA perform poorly (this is suggested by Figure 1 and 8, but not measured precisely), thus low diversity is bad\n  - Having variety in L2 norms in an EnsembleDQN setup is good\n One big missing step is to verify that diversity in L2 norms induces diversity as measured by CKA. Another missing step as I mention earlier is that to show that high diversity (as measured by CKA) induces good performance.\n- Appendix F should be much larger, and contain a plots of all 5 diversity measures, as well as, ideally, individual or group statistics of the L2 norms these regularizations are acting upon.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xn3Klt-U0Ij", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Q-learning is known to have overestimation bias. Approaches like EnsembleDQN and MaxminDQN try to use different estimates from ensembles of learners to reduce the bias. The authors study a specific observation and try to tackle it by regularization technique to maximise the diversity of representation space. Five different regularization functions are evaluated in the paper. And experiments show that the proposed regularization helps on the diversity and outperform MaxminDQN and EnsembleDQN. Note that the reviewer is not very familiar with methods to introduce diversity in representation, but based on educated guess, the proposed method look interesting.\n\nPros:\n+ The reviewer really likes how the paper goes from Section 4.2, illustrating the misconception of using different random initialization of neural networks could enforce diversity, then going to 4.3 to list the potential regularization metric which could be borrowed from economic theory and consensus optimization, including Atkinson Index, Gini coefficient, etc. \n+ Results in Section 5.1 and Figure 2 demonstrate the excellent performance of the proposed method;\n+ The t-SNE visualizations in Figure 3 are very clearly demonstrating the effect the regularization has;\n\nCons:\n- From results in Figure 2, the best perform regularization methods keep changing, e.g., for (a) Catcher, they are Gini-Maxmin and VOL-Ensemble and for (b) PixelCopter they are VOL-Maxmin and Theil-Ensemble. It would be great if the authors could keep the algorithm fixed and then show the performance across different games. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "QRoreoXnJWd", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper presents different regularizers that promote representation diversity in order to prevent value function collapse and lead to better performance. This is mainly based on a conjecture and is verified with an experimental study.\n\nGeneral comments:\nThe paper is well written and presents the different concepts quite well. I think the experimental section is well organised. However the number of environments is quite small. I do think that the contribution would be much stronger with a more exhaustive study. At the moment, it is quite hard to have a definite answer on how much the regularizers proposed are useful. One thing that could strengthen the experimental section is to provide the number of weights used for each methods. I assume that the ensemble networks used N times more weights that the simple DQN. If so, would it be possible to compare with a network N times bigger for DQN?. It is mainly to compare for a fixed computational budget how much the proposed methods are better.\n \nFinally, overestimation may not be all bad in particular for exploration. It could be a way to allow for some optimism in the face of uncertainty. It would be nice to show if this intuition is wrong and provide more evidence why overestimation is wrong and lead to bad outcomes. This could be done with a broader experimental study.\n\nRating: I am ready to change my rating if some or all of my comments are answered. This review is an educated guess and I am not at all an expert in the field of overestimation in statistics, economy or reinforcement learning.  \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper tackles the Q-value overestimation problem by proposing a regularization technique to maximize diversity in representation space, preventing ensemble "collapse", in order to improve the efficacy of techniques such as Maxmin and Ensemble Q-learning. Reviewers praised the originality of the method and the interesting connections drawn to economic theory, and seemed to agree that the method is somewhat effective. On the other hand, R3 pointed out that hyperparameters are tuned per-domain, the number of domains considered is small (echoed by R4), and criticized the paper for failing to truly validate its central hypothesis experimentally (echoed by R1).  R4 raised the issue of unfair comparisons in between ensemble and non-ensemble methods, while R1 raised a multitude of criticisms ranging from ahistorical attributions to confusing figures, which I will not exhaustively repeat here. Based on the reviews, and the fact that the majority of reviewers' concerns remain entirely unaddressed (the authors only responded to R2), this manuscript is not a candidate for acceptance at this time.

Decision: Reject