Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: MED-RL
This paper studies foster diversity in ensemble of DRL networks by regularization methods. The paper is an empirical one and compared five ensemble methods with and without their diversity algorithm in six Mujoco and six Atari games, and showed some results. 
The algorithm proposed is a modification of MaxMinDQN by Lan et.al. with a regularization. 
main_review: 
The claim in abstract that “members of the ensemble can converge to the same point either the parametric space or representation space during the training phase “. This is not true for DRL. Neural networks converge to different solutions given the initialization is different and multiple local minima. 

Questions:
1. Why these environments were chosen? 

2. I get your Sec. 4.1. However, this is far from compelling. The claim of your conjecture is pretty big and here you have a small experiment about only one algorithm at only two time points. 

3. In Alg 1, what is I()? Is it a general function that you have a few alternatives in Sec 4?

4. Sec 5.1: how do you initialize the networks in reaching the conclusion that “each neural network is trained on a separate batch from the replay buffer but still learns similar representations. “. Did you try another initialization method? How come the cross-diagonal of the after-training not exactly being ones?  Is the CKA non-symmetric? Why do you say the similarity is high? 0.15 between layer 6 and 7 is pretty low. 

Ensembles in Deep RL: Most the cited papers used ensembles for the critic (Q function). There is also ACE algorithm that uses ensembles from actors. Building an ensemble from actors can help agents explore at the option level. 

ACE: An Actor Ensemble Algorithm for Continuous Control with Tree Search
https://arxiv.org/abs/1811.02696

The results are not clearly separatable. Many lines are hard to see their statistical importance especially only 5 runs were performed. 

Minor:
Lan Qingfeng. Gym compatible games for reinforcenment learning. 
The author name for this paper reversed first name and last name. 

summary_of_the_review: The paper had a claim that initialization of different networks is not effective in proving diversity. This is not well supported. It can give a wrong message to the literature with the small and insufficient studies. this is my main concern.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a collection of diversity metrics to improve ensemble diversity, which substantially improves learning efficiency for a variety of RL methods.
main_review: This paper is clearly written and easy to follow. I'm always happy to see simple effective techniques that can substantially improve the performances of existing methods in challenging testbeds, which looks promising to me. 

**Question**
My biggest question is that all the criteria are evaluated w.r.t. the l2 norm of the whole network parameters, which is very counter-intuitive to me. Note that the paper simply uses the CKA metric to evaluate layer similarities for the purpose of visualization and interpretation. Wouldn't it be a more natural choice to use some metrics that at least involve CKA or layer-wise information? For example, can we simply optimize CKA directly in the pseudo-code instead of using $\mathcal{I}(l_i,l)$ as the auxiliary objective? Could the authors provide some in-depth analysis and justifications on why simply the l2 norm is preferred? Or, is it possible to provide some more experiments that use CKA directly?

**Possible Improvement**
From the additional diversity training objective highlighted in the pseudo-code, it is conceptually related to the diversity learning literature in population-based-training, where a population (corresponding to ensemble to some extent) of policies are trained and each policy not only optimizes its reward objective but also optimizes a diversity metrics for the purpose of effectively covering diverse strategic modes. So it would be appreciated if the authors can provide some discussions in the related work section. Some of the references are listed here for the purpose of helping the authors to survey the literature more easily: https://arxiv.org/abs/2002.00632, http://proceedings.mlr.press/v139/lupu21a.html, https://arxiv.org/abs/2103.04564, https://arxiv.org/abs/2106.02195 
summary_of_the_review: In general, although the paper can be still improved, the general results and suggested techniques are promising.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers a problem of ensemble-based deep RL methods that ensembles of critic networks converge to the same point in the representation space. To address it, the paper proposes a regularization technique that forces representations of a critic network to be dissimilar from those of other critic networks in the ensemble. It is empirically shown that this regularization technique improves the sample efficiency and asymptotic performance of baseline algorithms.
main_review: # Strong Points of the Paper

- S1. It is nice that one can gain a considerable sample efficiency and performance improvement by such a simple technique.
- S2. It is intriguing that the performance of ensemble-based deep RL methods highly (negatively) correlates with the representation similarly among ensembles of neural networks.
- S3. I highly appreciate that the paper provides many experimental results with different regularizers, baseline algorithms, and the number of ensembles.

# Weak Points of the Paper

I give detailed comments about these points in the next section.

- W1. From Figure 1, I am not so convinced that the performance of ensemble-based deep RL methods highly (negatively) correlates with the representation similarity.
- W2. It is unclear how hyper-parameters and seeds are chosen.
- W3. Some comparison seems to be unfair.

# Detailed Comments

As W1 above, what is meant by "high" and "low" CKA similarity? In heatmaps at point A and C, (near-)diagonal elements have low values, but other elements typically have higher values. In contrast, heatmaps at point B and D show the opposite tendency, and it is not so convincing to me that there is a negative correlation between performance and layer x-y similarity. Furthermore, only showing results at 4 points is not convincing to me. I trust the authors, but it is known that there is an implicit bias that causes cherry-picking without any intention. Can you plot curves of layer x-y similarity along with the performance?

As for W2 above, how did you choose hyperparameters and seeds? There are some places where the authors say "seeds are fixed". However, does this mean that you fixed random seeds, ran experiments with different hyperparameters, and pick up the best results? If you did this, there may be a maximization bias. In addition, running experiments with only 5 seeds is probably insufficient (Henderson et al. 2018). Please run experiments with more random seeds.

As for W3, it seems to me that REDQ's performance in this paper is significantly lower than that of original paper. Since the authors of REDQ's paper opensourced their code [here](https://github.com/watchernyu/REDQ), I don't think reporting re-implemeted REDQ's result with a lower performance is fair. (Or you use it, and I just missed that?)

# Questions

- Q1. As for proposed indices, it seems to me that you can easily maximize some (maybe all?) of them, especially when ReLU is used. For example, in case of Gini coefficient, you can multiply weights of the second to the last layer by a huge constant, and then, multiply the weights of the last layer by a small constant. By choosing the constants carefully, you can get as high $L^2$ norm as you like without affecting the output and causing a high Gini index. Why doesn't this occur? Maybe I am missing something?
- Q2. Why don't you simply minimize the sum of CKA? It is possible to backpropagate through it, right? Figure 1 indicates that low CKA implies high performance, so it is a very natural idea to minimize the sum of CKA.

# Minor Comment

Would you replace Figure 2(a) with a lighter one? It takes a long time to load the figure on my PC, and I can't print out the PDF probably because of it. I guess it is because the figure is a vector format, and there are too many points in the figure. Maybe replacing it with JPG or PNG would resolve this issue.

# References

Deep Reinforcement Learning that Matters; Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger. AAAI 2018

summary_of_the_review: Figure 1, which is intended to demonstrates the negative correlation between representation similarity and performance, is not convincing to me. Furthermore, the experiment conditions seem to be insufficient or not explained well.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, it is proposed to regularize the parameters of each Q function in ensemble so that they are not similar to each other. Through various experiments, it is shown that RL methods that use ensembles of Q-functions (e.g., REDQ, Minmax-DQN, etc) can improve its sample efficiency by using the regularization. 
main_review: strengths: 
+ A simple but general regularization method to improve the performance of Q-function ensemble-based RL methods is proposed. 
As I mentioned in the summary of the paper, a method to regularize the similarity of the parameters of the Q-functions is proposed in the paper. 
The similarity of parameters between Q functions is evaluated based on five measures (e.g. Gini coefficients), and then added to the loss for the Q functions. 
Since we only need to add a similarity score to the loss, the proposed method can be easily introduced into the existing ensemble-based RL methods. 
+ The proposed regularization method is shown to be effective in various RL methods and environments.  
The proposed regularization method is shown to be effective in five different RL methods (SAC, TD3, REDQ, Maxmin-DQN, and Ensemble-DQN). 
In addition, the regularization method is evaluated in continuous control (6 Mujoco envs.) and discrete control (6 Atari envs.).  



weaknesses: 

- There is no theoretical justification for introducing the regularization method based on the similarity measures.  
  - For example, it is not clear if Q-functions converge to the optimal one (under reasonable assumptions) when they are trained with the regularized method.  
  - Also, there is no theoretical discussion of how the use of similarity measures affect overall performance improvement.  
  - Nevertheless, I regard the study presented in the paper as an empirical one, and thus do not think that the lack of theoretical justification is a severe weakness of the paper. 

- The quality of the presentation is not sufficiently good: 
  - Connections among the sections in the paper are somewhat unclear.  
    - For example, in Sections 3 and 4.1, the similarity of the Q functions (neural nets) is measured by the similarity of their activation units' outputs. However, in Section 4.2 and later, the similarity of Q functions is measured by the similarity of their parameter values. In sum, the criterion used for evaluating similarity differs among the sections.  
    - In addition, Algorithm 1 in Section 4 describes the Maxmin-DQN variant that uses the proposed regularization method. However, this algorithm (and Maxmin-DQN) do not appear in the later sections (except Appendix). 

  - Experiment setups are not clearly explained. In particular, the setup in Section 5.4 is very unclear, e.g.,:  
    - Steps taken by MED-RL to reach REDQ performance:  How this number of steps is counted? How MED-RL's performance is evaluated? (is it MED-RL's return in a single test episode? or its average return over multiple test episodes?)  
    - Wall clock time REDQ (in mins): Is this the time for REDQ to complete 300k interactions with an environment? If so, what if REDQ's performance converges before 300k interactions? 
    - REDQ baseline: Is the REDQ's learning curve the one shown in Figure 5? If so, why is it much worse than the one shown in the original REDQ paper?  

Minor comment:  
The figures in the pdf of the paper (e.g. Figures 3 and 4) take a very long time to be rendered.
This is probably because all the data points in the graphs are output in vector format.
If you want to put graphs with many data points on the same page, it is better to rasterize the graph. 


summary_of_the_review: I'm leaning to recommend reject. 
I acknowledge the significance of the finding that it is useful to keep the divergence among the Q-function. 
I also acknowledge the authors' effort to demonstrate the usefulness of the regularization method in various RL methods and environments. 
However, the clarity of the current version of the paper does not meet the threshold for publication, and a non-trivial revision is needed. 

----- Update after author's revision 20211125 ------------------------------  
The authors have improved the explanation of experimental setups, but the connections between the sections still have not been sufficiently improved. 
In particular, some of the other reviewers are concerned about the mismatch between the similarity criterion (CKA) used in Section 4.1 and the similarity criterion used in later sections. 
I have seen their discussions with the authors, and think that their concern about the mismatch is not sufficiently resolved. 
I think that replacing the analysis based on CKA with an analysis using criteria based on parameter values (e.g., equation (4)) would make the discussion in the paper more consistent.  

Overall, I acknowledge that the paper was improved to some extent in the revision, and thus slightly improve the score (WR->WA). 
However, as mentioned above, I have still concerns about the clarity of the paper and cannot strongly champion the paper to accept.
review_title: null

Meta-review: This paper concerns ensemble methods in deep reinforcement learning, examining several such methods, and proposes to address an important issue wherein ensemble members converge on a representation of approximately the same function, either by their parameters converging to an identical point or equivalent points that give rise to the same function. The authors propose a set of regularization methods aimed at improving diversity, and benchmark these augmentations on five ensemble methods and a dozen environments.

3 of 4 reviewers generally praised the method's simplicity and generality, and found the experiments convincing. Reviewer a9sA describes it as "clearly written and easy to follow", although others found clarity lacking in parts. There was agreement among these 3 reviewers that this was an interesting problem to tackle. Reviewer TfGq notes that this method lacks theoretical justification or guarantees, but that as a largely empirical paper this is perhaps of secondary importance. Reviewers 6miY and a9sA had questions about the precise choice of metrics, hyperparameters and seeds; the resulting discussion cleared up many of these concerns.

The most critical reviewer, i4M1, disputes the existence of the phenomenon at all, saying that "Neural networks converge to different solutions given the initialization is different and multiple local minima." The remainder of i4M1's criticisms seem centered on the choice of environments and the number of seeds (also raised by other reviewers). The issue of seeds has been addressed partially and the authors have committed to strengthening their results in this regard.

Reviewer i4M1's statement on the convergence of neural networks to different minima matches a bit of dated folk wisdom about neural networks, but the AC disputes this. The authors have cited a study from before the DL era properly began that identifies this issue and Section 5 addresses these criticisms directly. In practice, modern neural networks, especially with non-saturating activations, tend to be surprisingly consistent across random seeds when trained against the same data stream, and more recent work posits that the loss landscape is less riddled with local minima than with saddle points (see e.g. Dauphin et al, 2014). _Equivalent_ minima are of course common due to scaling and permutation symmetries but SGD has a well documented preference for low norm solutions in the former case, and the authors' have chosen methods that would at least conceivably overcome these issues, by focusing on summary statistics of the representations rather than their precise values (and indeed, CKA is designed with these concerns in mind).

Despite i4M1's incredulity I am inclined to agree with the majority of reviewers and view the paper as a worthwhile contribution to the body of knowledge (purely empirical though it may be) on both NN ensemble methods and DRL ensembles in particular. The introduction of measures from economics is clever and original, and the results are promising. A more exhaustive study on the entire Atari57 benchmark but can appreciate the resource problem this poses, and find that the suite of considered environments, combined with the augmentation of 5 different DRL ensemble methods, strikes a good balance. I concur on the issue of seeds and would encourage authors to include as many as possible for the camera ready, but on balance would recommend acceptance.
Decision: Accept (Poster)