Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Motivated from the idea that minimal representations — that encode information relevant to a task and nothing more — are likely to generalize well, the paper proposes an approach, titled Model Invariance (ModInv) to learn representations. The underlying idea is to use multiple diversified predictors solving auxiliary objectives on top of a shared representation/encoder. Different predictors are diversified in terms of their training dynamics — different initializations and learning rates. The approach relies on the hope that a shared representation that can lead to optimal performance for a diverse set of predictors is likely encoding some notion of implicit invariance to changes in predictors, thereby, introducing some notion of minimality. The authors conduct experiments on reinforcement learning and self-supervised (in computer vision) settings and claim that ModInv provides strong performance improvements in both. Based on further ablations,  the authors demonstrate how sensitive ModInv is to the number of predictors and the diversity in terms of training dynamics (in terms of learning rate), and whether ModInv is complementary to recent augmentation strategies in reinforcement learning.
main_review: I will now highlight the strengths and weaknesses of the paper. The strengths and weaknesses will highlight the quality, clarity, and significance of the proposed approach.

**Strengths**

1. The paper is generally well-written and easy to follow. With the exception of a few points covered under weaknesses, the authors do a good of introducing the proposed approach and walking the reader through different experimental settings.

2. On the DMC suite, ModInv seems to offer improvements over Dreamers, CURL, DBC in most cases (excluding RAD settings, please refer to the weaknesses). In vision settings, when using a linear predictor, ModInv seems to offer significant improvements on CIFAR-10 when compared to a recent self-supervised approach (SimSiam). 

**Weaknesses**

1. I’m not sure I completely follow the thread of motivation introduced in the paper. ModInv is right now motivated from a sparsity/minimality point of view (in the context of how they’ve been described in the submission) and the reasoning for the particular instantiation relies on introducing invariance across multiple predictors which subsequently reduces redundancy in the learned representations. I’m not sure if this hop of reasoning works. It’s not entirely clear to me that being resilient to changes in downstream predictors will necessarily reduce redundancy in the learned representations. Can the authors comment on this? Additionally, I would suggest the authors also discuss [A], where the information bottleneck interpretation of self-supervised learning w.r.t. a downstream task has been studied, in related work and situate their proposed approach accordingly

2. The paper suffers from a few clarity issues. Highlighting them here. For instance, the ModInv structure implies that each predictor head is trained independently (a given sample only trains one head), implying that there is some underlying routing of the sample to a predictor head is going on. It’s unclear from the draft right now how this routing is performed. As in, given a training sample, how do the authors decide which head to feed the corresponding representation? If this routing is random, can the authors comment if they investigated other routing strategies? In Section 5.2, the authors mention that mean correlation across feature dimensions decreases more steadily when using ModInv. However, the draft doesn’t provide any supporting evidence (a table/plot) for this. More specifically, evidence demonstrating the evolution of mean correlation when using vanilla, ModInv, and an agent trained with mean correlation loss would help solidify this point even more. This is particularly important since the bulk of the motivation for ModInv relies on this being a desirable and observed behavior.

3. I think the experimental evidence in support of the proposed approach is somewhat weak. Firstly, ModInv + Aug outperforms RAD in only one setting (Reacher, Easy at 500k; Table 1). Did the authors use the same set of augmentations being used in RAD? If yes, then the utility of ModInv is heavily diminished. Particularly since ModInv involves using extra parameters. Results on computer vision benchmarks seem strong only for one setting — CIFAR-10 with a linear predictor. ModInv is competitive or worse on STL-10 and when a non-linear predictor is used, ModInv seems to consistently hurt performance in most settings. Furthermore, the experiments in Table 4 seem to suggest that ModInv outperforms Mean Correlation Loss in only one (out of two) settings. Given these observations, I am not entirely confident of the utility of the proposed approach. Additionally, I would suggest the authors appropriately adjust the claims regarding performance improvements. An ablation mentioned in Section 4.2.2 states that using individual SimSiam models with different learning rates is worse than using different predictor heads. The paper would benefit if this ablation was done in a more comprehensive manner, i.e., across both RL and vision settings at different stages. Since ModInv relies on using multiple predictor heads and using different models is a natural alternative, including such results seem necessary and would definitely improve the paper.

4. [Minor Points] Can the authors comment on how the learning rates and initializations for different predictor heads in all settings were chosen? Given that diversifying predictor heads is a big part of ModInv (as indicated from the ablations in Table 5), specifying these details seem important. In Section 6.1, the authors suggest that ModInv is similar to DIB in terms of instantiation. It’s unclear to me how that’s true. Can the authors comment on this?

[A] - Self-supervised Learning from a Multi-view Perspective
summary_of_the_review: The points highlighted under strengths and weaknesses form the basis of my rating. In particular, the weaknesses influence my current rating of the paper the most. While the paper is generally well-written, I think the issues highlighted under weaknesses in terms of motivation, clarity issues and lack of strong experimental evidence makes me less convinced of the utility of ModInv. I would encourage the authors to address the first three points under weaknesses. Addressing those will greatly help in reconsidering my rating of the paper. The minor points highlighted under 4 are addressable, I think.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose MODINV, a new algorithm to learn minimal and compressed representations.
MODINV uses several predictors with different initializations and learning but all built on top of the same learnt representation.

After explaining the algorithm and its relation to minimal representation the authors evaluate it on vision and RL tasks.

On vision tasks the authors show improvement on CIFAR-10 by adapting their method on top of SimSiam in the specific case of a linear predictor. On RL they evaluate on control tasks from pixel inputs.

The authors also provide an experiment to show that adding a loss to discourage correlation within features helps to improve RL performance.
main_review: Strengths:
* The idea in itself is interesting and can be applicable on top of most of self supervised algorithms.
* The authors evaluate their method on two very different sets of tasks (RL and Vision).
* The paper is well-written easy to read.

My main concern with the paper is its experimental soundness. I would strongly encourage the authors to resubmit a new version with more thorough experiments in order to back the (potentially interesting) claims that they make.

Weaknesses:
* From the elements currently provided in the paper, I don't see why the authors' algorithm would lead to minimal representation, at least in the information-theoretical sense as defined in the paper they cite: Shamir et al., 2010. I would expect either a citation or some experiments to support this claim.
  * For instance: "The common representation acts as a implicit invariance loss which ensures that only the optimal representation remains at convergence. " -> I need either a citation or evidence for that.
  * Similarly: "since there is only one common representation to support all the predictor heads, this representation must be robust to any spurious correlations as training progresses.". 

In particular I don't see why each predictor would not learn to filter out unnecessary information in the representation. Using the paper's example of the pen: knowing the position of a distractor object is not necessary to solve the task, yet all the predictors could learn to filter out this useless information. In this case I don't see why the proposed algorithm would incentivize to remove this information from the representation.

* The authors claim "Our experiments show that [...] the mean correlation decreases much more steadily when we use MODINV." 

Are these results in the paper? If not, adding them would benefit the paper a lot. The current Table 4. only shows that encouraging decorrelation of features enables better RL. It does not show that using ModInv provides better decorrelation of representation features.

* The author choose to evaluate in the specific case of a linear predictor. They claim :
  * "2-layer predictor has been speculated to be using a lot of 'lucky' initialization".
  * "testing in this setting offers a much more robust evaluation option".

These claims lack either citations or experiments.

Evaluating with a linear predictor is problematic for me as this case is not the standard regime. SimSiam is tuned for the 2-layer predictor regime and does not work well in the linear regime.I would expect the authors to either provide improvement in the 2-layer regime or convincing experiments or citations to support this choice.

* Can you clarify : "Unlike the reinforcement learning case, where any of the modules after the representation (state and reward prediction heads) are not used actively in the algorithm, in the vision case the projector and predictor both are used to align the backbone representation (Tian et al., 2021). "
For me it seems that in both cases the predictor and projector are used as tools to learn a good backbone representation. Even in the vision case the predictor and projector are thrown away when the network is actually used (for classification for instance).

Typo: "the phenomenon that that neurons that have"
summary_of_the_review: I think a lot of claims are not justified either by experiments or citations. Especially the main claim that the authors' algorithm learns a minimal representation.

I also found the vision experiments not convincing: on CIFAR-10, they used SimSiam in the linear prediction case instead of the usual 2-layer predictor in which SimSiam performs best.

From the elements provided in the paper, I could not tell whether the proposed algorithm was indeed helping to learn minimal representations, or that such a representation would generalize better compared than one learnt without the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Inspired by the information bottleneck principle, the authors propose to learn minimal representations by independently training multiple predictive heads on top of a shared representation, such that the resultant predictor is more robust to spurious correlations. To test this proposal, the authors provide extensive set of experiments in RL from pixels and vision benchmark tasks. The ablation studies help make the case too.
main_review: ## Strengths

- The paper is well-written and easy to understand. The intuitions and connections to relevant related works have been established meaningfully.
- The ablations are very well done, and reasonably support many of the claims.

## Weaknesses

I have two main concerns to be fully confident of the gains claimed.

- The proposed scheme lies very close, in principle, to a family of approaches called the [Hyperparameter Ensembles](https://arxiv.org/abs/2006.13570) (HE). HE induce diversity by training using different hyperparameters. The obvious difference is the fact that the authors here only train the predictor heads in an HE manner instead of the full network. Nevertheless, approach is very closely related.

  Here, the authors claim the benefits from an information bottleneck perspective, but the intuitions only remain justified empirically. In that sense, I would then believe that the gains are due to the same reason deep ensembles work, than for an information-theoretic reason of learning minimal representations.

  The fact that adding more than $K=3$ heads leads to diminishing returns may also hint to qualitative similarity to deep ensembles (where an ensemble beyond $\sim 5$ leads to diminishing returns).

- Another intuition that the authors highlight is that forcing the same representation induces an information bottleneck. This is turn should nudge the network away from learning spurious correlations. This claim is hand-wavy and is not supported either empirically or theoretically (perhaps even a toy example would suffice to qualitatively guarantee that it is doing what the authors claim it to be doing). My impression here is again that it is not that the minimal representations are learned that help avoid spurious correlations and hence better generalizations. It is instead the same reason that deep ensembles work, i.e. finding a set of good minima which induce a reasonable functional diversity. Deep ensembles are not very robust to spurious correlations either (though better than a single network), and it is unlikely that a very similar training routine proposed in this paper does something very different.

Nevertheless, I do want to acknowledge that the setting of using RL with pixels and the achieved gains is still a potentially novel intersection. But the authors approach the problem from an information bottleneck perspective, the robustness claims from which are not entirely supported.

## Minor

- Highlighting best results in Table 1 would make it easier for the reader.
- Hyperlinks do not work in the main submission.


summary_of_the_review: The main theme of my concerns is that it is not that the training has learned minimal representations but just that it is the diversity of predictor heads that provides the gains. Without sufficient empirical or theoretical evidence, I think the main claim of the paper of being robust to spurious correlations due to the implicit information bottleneck is weak.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes using an ensemble of predictors or heads on top of a shared representation to improve performance. The ensemble of predictors/heads are made "diverse" by using different learning rates. Some performance gains are shown in RL settings, while some stability gains are shown in a particular vision setting.
main_review: **Strengths:**
- Simple method -- just train 3 heads with different learning rates.
- No increase in wall-clock time
- Clear writing

**Weaknesses:**
- **No evidence for the main claims**
    - *”Common representation acts as an implicit invariance objective to avoid the different spurious correlations captured by individual predictors”:* There is no evidence of this, i.e. that the proposed method better deals with (or avoids) certain spurious correlations.
    - *Each head “captures different spurious correlations”:* Again, there is no evidence for this claim.
    - *”[...] diversify the learning of heads so they capture different spurious correlations”* Again, no evidence, e.g. an analysis of what correlations are captured by each head.
    - *”[...] This in turn leads to better generalization performance”:* There is no evidence that the proposed method achieves better generalization performance by avoiding spurious correlations. If in-distribution test-set performance is what is meant by generalization performance, then I think this needs to be made clear. Nonetheless, to justify the original statement, it would still need to be shown that this improvement is due to the avoidance of spurious correlations.
- **Questionable claims**:
    - *Training independently*: the predictors are given the same samples in the same order, and even share the same (backbone) representation. “Trained independently” is a bit of a stretch…
    - *”Strong performance boosts in both [RL and vision settings]”:* It seems that the proposed method ModInv only improves performance when additional training stability is needed (perhaps unsurprising since it's an ensemble method).
      - *RL:* With data augmentation to already stabilize training, ModInv only improves performance on 1 of 6 tasks over RAD, decreasing performance on 4 of 6 tasks (500k steps).
      - *Vision*: ModInv only helps when using a linear predictor on CIFAR-10. When using a nonlinear predictor on CIFAR-10 (which is the standard and improves performance with or without ModInv) or any predictor on STL-10, ModInv actually hurts performance.

**Other comments:**
- **Missing results**: Section 5.2 -- where are the results mentioned that show the mean correlation decreasing more steadily with ModInv?
summary_of_the_review: While the method is admirably simple and the writing clear, the main claims of the paper are either questionable or without any supporting evidence. Thus, in my opinion, this paper falls well short of acceptance for a top-tier conference like ICLR. If evidence can be provided in support of the main claims, I would be happy to revise my score.
review_title: null

Meta-review: Although all reviewers had many positive comments on the paper, and the authors engaged nicely in the discussion period, at the moment there is a consensus among the reviewers that the central claims of the paper (related to minimal representations / information bottleneck) are not adequately supported by the current experiments. In particular, there were concerns that performance gains could be due to diversity of predictors, rather than minimal representations, which would need to be addressed. It's suggested that the reviewers take all of these comments and discussion into account when preparing a revised version of the paper.
Decision: Reject