Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper posits that compressing the representation during learning improves the robustness of said representation. To demonstrate their argument, they derive an objective for SimCLR and BYOL that explicitly compresses the representation using the Conditional Entropy Bottleneck. The addition of this compression improves the performance for both BYOL and SimCLR on ImageNET. They also demonstrate that their method improves the robustness of the encoder and its representation to perturbation of the input data.
main_review: The paper presents a principled and novel approach to learning a compressed representation in the context of contrastive methods. While the method is well detailed for both SimCLR and BYOL, it could be helpful to have a pseudo-code of the objective function detailing how the computation is done in practice. Other than that, the paper is very well written. The argument is concise and easy to follow.

The experiments convincingly show the benefits of the approach proposed. While the authors demonstrate that their method performs better than the state-of-the-art of ImageNet, they also provide an ablation demonstrating that the improvement is due to their modifications. They also demonstrate that their method improves robustness on various benchmarks.

Overall, the paper provides a good demonstration that the compression of latent representations improves robustness. 

### Question / comments
* L107: I am confused about the correspondance between $exp(\dfrac{1}{\tau} r_{yk}^\top r_x)$ and $b(z|y)$. Isn't $y_k$ the *negative* views and $y$ the *positive* view. I.e., shouldn't the former be a correspondance with $b(z|y_k)$?
* While the authors claim that they will release their code upon acceptance, they could have released an anonymized version.
* The practical realization of $e$, $b$, $c$ and $d$ were not clear to me. The confusion could be resolved by presenting a pseudo-code of the practical implementation of the method.
* It is not clear to me why a model Lipschitz constant generally relates to model robustness. The authors present some works that relate "types of model robustness" to the model's Lipschitz constant. What are the types of model robustness? How do these works relate to your model?
limitations_and_societal_impact: The authors discuss some limitations of their work.

The authors present a generic discussion of the societal impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Contrastive learning approaches learn salient representations that will be effective for various downstream tasks, while not all learned information is relevant to the task target. Given this insight, the authors aim to learn compressed representations by adding information compression objectives to existing algorithms. 

They applied conditional entropy bottleneck (CEB) objective to SimCLR and BYOL to compress away the irrelevant information from the learned representations. In addition, they discovered that enforcing the information compression entails the encoder with bounded Lipschitz constant which is closely related to robustness. Extensive experiments confirm their hypotheses that adding compression regularizer can improve the performance in terms of accuracy and robustness to domain shift.
main_review: They present their work with reasonable motivations. Overall, I think this paper would be interesting to the NeruIPS audience.

I have some technical concerns:
- It is unclear to me what are the architectures and outputs of decoders $d(.), c(.)$. Are they supposed to predict the inputs $x, y$?
- Is it possible to insert decoders $d(.), c(.)$ into Figure 1&2 just like you did for $e(.), b(.)$. I believe that would help people to get a better understanding of your work.
- I like the diagrams in Figure 1&2. But I would suggest to spend a few sentences to explicitly explain the work flows. For example, I think you can make them more friendly for readers if you can explain what are the red dash lines implying. 
- Although the absolute difference is small, I would suggest to report both mean and standard deviation. For instance, in table 1, the last these accuracies are just marginally better than baselines, if std dev is available that would help review to clearly see the significance of experimental results.
limitations_and_societal_impact: the authors have adequately addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper shows how to use the Conditional Entropy Bottleneck (CEB)[1] in a self-supervised fashion.  Thus resulting in a new method for training approximately minimal self-supervised  (SSL) representations, i.e., a new member of the SSL information bottleneck family (although extremely similar to [2]). Specifically, they show how to use CEB in conjunction with SimCLR and BYOL. Furthermore, they provide good arguments to suggest that their proposed objective will learn smoother encoders than standard self-supervised models, which can help robustness. Through extensive experiments,  they show significant performance gains of the proposed methods compared to SimCLR and BYOL.

- [1] Fischer, Ian. "The conditional entropy bottleneck." Entropy 22.9 (2020): 999.
- [2] Federici, Marco, et al. "Learning robust representations via multi-view information bottleneck." arXiv preprint arXiv:2002.07017 (2020).
main_review: **Post-rebuttal**: I've updated my score given that the authors (seem to) have addressed all my major concerns (related work / mathematical mistakes / practical details) in the updated manuscript.

----------

**Overall**: This paper has interesting ideas with strong experimental results. It will likely be of interest to the SSL and IB sub-communities. I nevertheless do not think that the current version is ready for publication given that (1) it fails to discuss very related work; (2) has mathematical mistakes; (3) it's very hard to understand what authors do in practice. Fortunately, all these problems can be easily solved and I am happy to increase my score if the authors make the necessary changes. 

**Strengths**:
- **Significant empirical gains and thorough experimental section**. The experimental results show significant gains against very strong baselines. These results are even more impressive given the thorough evaluation protocol. 
- **Interest to the community**: The link between the proposed CEB and smoothness of the learned objective is interesting and to my knowledge novel. Such a smoothness argument might be useful beyond robustness. Furthermore, although SSL+IB has already been considered (see "related work" below), the final objective used with vMF variational family is novel and has many potential advantages compared to previous work (which I hope the authors will discuss in the final version).
- **Thorough appendices**: The supplemental material is thorough and very useful.

**Weaknesses**:
- **Missing important related work**: The discussion of related work is insufficient. Most importantly, multi-view IB [1] provides a loss that is extremely similar to your C-simlcr objective (the only real difference seems to be the variational family) and provides experiments in robustness and standard SSL just as you. More generally, the paper reads as if it is the first (although the authors do not explicitly say so)  to investigate the use of "compressed"/"low information" representations in SSL, which is definitely not the case. The paper also fails to cite previous work that uses/notice the link between von Mises-Fisher distribution and contrastive learning with normalized representations. This definitely diminishes the novelty of the paper, but I do not find that to be an issue as long as it is correctly discussed. Authors should take advantage of their "related work section" to discuss the advantages/limits of the proposed extension of CEB compared to previous work. More details can be found in my "related work" section below.
- **Some mathematical mistakes/imprecisions** currently there are many mathematical issues that obscure some parts/equations of the paper. The biggest issues are the following (see "suggestions" for other issues):
     -  Equation (5) has many mistakes, to the point that I don't even know what you were trying to show and how many mistakes there are. The mistakes that are obvious are: (1) what do you mean by lower bounding a conditional entropy by a distribution? surely you meant $E[-\log d(y|z)]$; (2) you say that you drop $H[Y]$ because that's a constant, but that's only true if you have some minimization, do you mean taking min on both sides? (3) you say that this is the InfoNCE bound but you are removing $\frac{1}{k}$ from the denominator, is that on purpose? If so say that it's InfoNCE up to $log K$. I actually think there are deeper issues, but I'm not even sure what you aiming for due to the previous issues. Here's what I think you meant by using InfoNCE bound: $I[Z,Y] \geq I_{NCE}$ so $H[Y] - I[Z,Y] = H[Y|Z]  \leq H[Y] - I_{NCE}$ so $\inf_{e(z|x)} H[Y|Z]  \leq \inf_{e(z|x)} - I_{NCE} =  \inf_{e(z|x)} - \log K - E[ \log d(y|z)]$ so $\inf_{e(z|x)} H[Y|Z]  \leq \inf_{e(z|x)} E[- \log d(y|z)]$. But if that is the case you are: (1) missing a negative sign; (2) used lower bound instead of an upper bound; (3) should really just use the standard cross-entropy bound instead of InfoNCE to get the same result.
    - I think that the “distance" measure that you are using is not an actual metric (it’s only a pseudo metric) because it does not satisfy $d(x_1,x_2) = 0 \iff x_1=x_2$, indeed, you can have $e(z|x_2) = e(z|x_1)$ for different $x_1,x_2$.  As a result, you cannot formally talk about Lipchitz continuity and the results from the paper you cite in line 191 might not hold. I would suggest adding a footnote explaining that you are actually using a pseudometric so you are not exactly considering Lipchitz continuity but hypothesize that this provides similar robustness guarantees. I actually don’t think that it makes much difference but this has to be flagged.
- **Hard to understand what is done in practice and missing important implementation details**: it's actually quite hard to get what is done in practice because there are many moving components and these are scattered in different places of the paper (e.g. (5) and (11) are the equations that are needed for C-SimCLR but separated by much text and equations that is not used for the model). It would help a lot to have clear algorithms for C-SimCLR and C-BYOL. Other non-trivial and important details are missing, e.g., how you sample (with gradient flows) from the vMF which is known to be very hard in high dimensions.

**Related work**
- **Multi-View Information Bottleneck**: As previously stated [1] is extremely similar to your paper on many fronts. As a result,  I think that this paper should be cited in your introduction where you should explain what you are contributing which they didn't. Furthermore, the exact difference needs to be discussed in the text. 
- **Self supervised learning + information bottleneck**:  besides [1] there are other (albeit less related) papers that consider "compressed"/"low information" representations in SSL. For example, Barlow Twins[2] can be seen from an SSL IB point of view  (see their Appx. A).  [3] provides some theory for this type of SSL+IB. [4] provides different losses for learning compressed SSL representations (although this seems to be online since March so contemporary work).  The only similar work that you cite is InfoMin Aug, which arguably is the least related.
- **Von Mises-Fisher + cosine similarity**: the relation between vMF and contrastive loss with cosine similarity has already been pointed out/used by others, e.g., [5-7]. Please add some discussion about those or at least acknowledge some of them around line 113.
- **Minimality and generalization**. Your experimental results clearly show that compressed / low-information representations help downstream performance. There’s a large body of work that proves/suggests that this will be the case and I think that these should be cited.   E.g., [8] proves that a minimal representation ensures a generalization gap of 0 (i.e. tight but only for optimal representations), while [9,10] provides upper bounds (although loose) on the generalization gap based on $I[Z, X]$. 

**Suggestions / specific issues:**
- I would say very early on that you are conditioning on a second view instead of the label (as usual in CEB). This was not clear to me when reading until I arrived at section 2.2.
- The use and meaning of $r_x,r_y$ vs $z_x,z_y$ is unclear. Can you please clarify around lines 102-103 how $r$ relates to $z$? From section 2.2. it seems that $r$ refers to parameters of the density of $z$ but this should be clearer much sooner. Also, in equation (6) should it be $L_{nce}(r_x,r_y)$?
- I find section 2.3 hard to follow. I would suggest simplifying this section, although to be honest, I’m not sure how, because there are many moving components in BYOL.  
- Section 3.2. is missing some motivation/explanation as to why compression should help in standard iid setting  (i.e. it’s missing a paragraph like at lines 271-273 for robustness). As I said in “related work” I would cite the body of work that proves [8] or suggests [9,10] that low information representations provide strong guarantees on generalization to give an intuition as to why this makes sense and why you consider the experimental setting 3.2. 

**Questions**
- Right now in equation (11) it seems that you are estimating the KL of the vMF distributions using samples, have you considered using an upper bound on the KL between vMF such as in [11]? This would essentially maximize $r_x^Tr_y$ which is much simpler and already symmetric. 
- How do you sample in practice from the vMF with a reparametrization trick in high dimension? Implementations of vMF usually suffer from numerical issues as the Bessel function is very hard to evaluate in high dimensions (e.g. TensorFlow's implementation does not work for more than 5 dimensions).  E.g., [12] had to come up with a clever parametrization trick for vMF, is that what you are using?

**Minor Suggestions**
- Equation (3): this should be an equal sign instead of define
- Line 81: say that d(y|z) is also a variational approximation
- Line 82: bounded -> upper-bounded
- Line 88: cannot take gradients -> cannot easily take gradients
- Line 107: $r_{y_k}$ -> $r_y$
- Line 107: corresponds to $b(z|y)$ ->corresponds to the unnormalized  $b(z|y)$       (or use $\frac{1}{C}exp$instead of exp, where $C$ is normalising constant).
- Line 114: n-dimensional -> (n-1)-dimensional     (or change pdf to n+1 in 115)
- Line 123: “relax” makes it sound as if the resulting $I_{NCE}$ is not a proper lower bound anymore, but that’s not the case, right? If it’s still a lower bound I’d use “extend”.
- Equation 15: I don’t think the first equality is correct: both densities have different domains and even different graphs. I think you mean that $e(z|x)=e(w \cdot \phi |x)$ where  $e(w ,\phi |x) = e(w | x) \cdot e( \phi |x)$.
- Line 158: “and since C is a constant.” I don’t understand what you mean here?
- Line 161: Continuity -> continuity
- Equation 18: say that L is maxed over L(z)
- Line 175: isn’t -> is not 
- I would replace all $\min$ with $\inf$ (e.g. equation 4,9,10,11)

**References:**
- [1] Federici, Marco, et al. "Learning robust representations via multi-view information bottleneck." arXiv preprint arXiv:2002.07017 (2020).
- [2] Zbontar, Jure, et al. "Barlow twins: self-supervised learning via redundancy reduction." arXiv preprint arXiv:2103.03230 (2021).
- [3] Sridharan, Karthik, and Sham M. Kakade. "An information-theoretic framework for multi-view learning." (2008): 403.
- [4] Dubois, Yann, et al. "Lossy Compression for Lossless Prediction." arXiv preprint arXiv:2106.10800 (2021).
- [5] Wang, Tongzhou, and Phillip Isola. "Understanding contrastive representation learning through alignment and uniformity on the hypersphere." International Conference on Machine Learning. PMLR, 2020.
- [6] Hasnat, Md, et al. "von mises-fisher mixture model-based deep learning: Application to face verification." arXiv preprint arXiv:1706.04264 (2017).
- [7] Kumar, Sachin, and Yulia Tsvetkov. "Von mises-fisher loss for training sequence to sequence models with continuous outputs." arXiv preprint arXiv:1812.04616 (2018).
- [8] Dubois, Yann, et al. "Learning optimal representations with the decodable information bottleneck." arXiv preprint arXiv:2009.12789 (2020).
- [9] Shamir, Ohad, Sivan Sabato, and Naftali Tishby. "Learning and generalization with the information bottleneck." International Conference on Algorithmic Learning Theory. Springer, Berlin, Heidelberg, 2008.
- [10] Vera, Matías, Pablo Piantanida, and Leonardo Rey Vega. "The role of information complexity and randomization in representation learning." arXiv preprint arXiv:1802.05355 (2018).
- [11] Diethe, Tom. "A Note on the Kullback-Leibler Divergence for the von Mises-Fisher distribution." arXiv preprint arXiv:1502.07104 (2015).
- [12] Davidson, Tim R., et al. "Hyperspherical variational auto-encoders." arXiv preprint arXiv:1804.00891 (2018).
limitations_and_societal_impact: The limitation and societal impact are generally well discussed. Although, as previously stated, I would like to see a comparison (and thus potential limitations of the current method) compared to previous SSL + IB papers.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work adds a conditional entropy bottleneck (CEB) objective to two existing frameworks for self-supervised learning, namely SimCLR and BYOL. The motivation for adding CEB is to encourage information compression in the representations learned from these models which can make the representations more robust. The authors measure the performance of SimCLR and BYOL with added CEB objective empirically on standard benchmarks such as linear evaluation protocol on ImageNet with different amounts of training data. They provide evidence that adding CEB outperforms other existing self-supervised learning models on these benchmarks, including vanilla SimCLR and BYOL. 
main_review: ### Strengths
- While the information theoretic CEB objective is not novel, to the best of my knowledge, incorporating it into SimCLR and BYOL has not been done before and is a well-motivated idea. This approach can be applied in other settings. 
- The experimental setup is exhaustive. Мodels with the proposed modified objective are evaluated on popular benchmarks (including the standard ImageNet linear classification protocol and linear classification on versions of ImageNet designed to evaluate robustness and generalization) and show strong empirical performance when compared to other existing methods. 
- The paper provides a link between Lipschitz continuity and CEB compression in the case of SimCLR.
- This is a well written paper and it is organized clearly. 

###Questions & Comments
1. How many seeds were used for results in Tables 2-5? What was the standard deviation?
2. What is the full range of values that were used for $\beta$? 
3. I think it would be beneficial for the main text to clearly indicate the best values for $\beta$ for both SimCLR and BYOL in all the different experiments.

###Nitpicks
- Equation (6): $z_x$ and $z_y$ should be replaced by $r_x$ and $r_y$, the sum in the denominator should be over lower case $k$ (not capital $K$)
- Line 155: in “distribution of ||z||” should use LaTeX, $\||z\||$

limitations_and_societal_impact: Yes, they have.
review_title: null

Meta-review: This paper applies conditional entropy bottleneck (CEB) objective to two existing frameworks for self-supervised learning: SimCLR and BYOL.  The paper clearly demonstrates the advantages of this approach and is well-grounded in the existing theory on CEB. Adapting CEB to SimCLR and BYOL seems to be novel. Interestingly, presented approach can be applied in several similar settings, thus the results can be potentially impactful also beyond the main scope of the paper.
Decision: Accept (Poster)