Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The paper "On the duality between contrastive and non-contrastive self-supervised learning" suggests a unifying perspective on two families of self-supervised learning. One family is the contrastive methods (such as SimCLR) that repel samples from each other through negative pairs. Another family is the covariance-regularizing methods (such as VICReg) that repel samples from each other by enforcing a unit spherical covariance matrix of the embedding. The two competing families have so far been seen as conceptually distinct. This paper argues mathematically and shows empirically that they are approximately equivalent.
strength_and_weaknesses: I am a big fan of unification papers. There are several new self-supervised methods appearing every year each claiming SOTA performance, but every method makes its own design choices and so conceptual comparisons can be difficult. This paper shows that many design choices are not essential, and is able to transform SimCLR into VICReg and vice versa while using the same network architecture. The authors also show that simply tuning some of the SimCLR hyper-parameters increases its performance on ImageNet to the SOTA level. I think this is a very welcome contribution and a very strong accept.

All the experiments were run on ImageNet for 100 epochs (only two final runs were made for 1000 epochs). Given that ImageNet optimization is costly, I was wondering if the authors could also demonstrate some of the results e.g. on CIFAR-10 where running 1000 epochs is not a problem. Or is CIFAR-10 "too simple" for the purpose of this study?
clarity,_quality,_novelty_and_reproducibility: see above
summary_of_the_review: Very strong paper suggesting a unifying perspective on various self-supervised learning methods.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper links two different views of self-supervised learning, namely contrastive and non-contrastive self-supervised learning. This is done by interpreting the non-contrastive methods as dimension contrastive ones, and regular contrastive methods as sample contrastive. The authors use this insight to demonstrate relationships between the two types of self supervised loss, and demonstrate how one can derive one type of loss from the other.
strength_and_weaknesses: Strengths:

- This paper promotes an interesting link between two variants of self-supervised loss which are often used in practice. Linking these two variants is a very original and interesting line of research, and the fact that the authors demonstrate that the two are very closely related is very useful to provide a better understanding of these techniques. 

- The theoretical part of the paper is useful, in that it shows that each of the two variants of self-supervised loss can be used to derive bounds for the other one. This is done in Equations 6 and 7 in the main paper. While the upper bounds in these equations are quite loose (for example, in Equation 6 the upper bound has a difference of the order $N^2 - N$), I believe that it should be argued that the lower bounds are more important. In Equation 6, normalizing the embeddings means we are optimizing $L_c$, and the lower bound tells us that $L_{nc}$ cannot be arbitrarily large either (similarly in Equation 7). I think this is an important connection between the two styles of self-supervised learning.

- The experimental part of the paper is also insightful, in that it supports the proposition of the authors that the two styles of self-supervised learning are closely related, so with sufficient tuning both should achieve similar performance. This is in line with the author's theoretical results. Moreover, in the Appendix the authors also compare the relationship between the losses in practice. These experiments serve to fully support the close connection between the two variants of self-supervised learning.

Weaknesses:
- The authors demonstrate how several well-known self-supervised learning methods fall in their framework of sample contrastive and dimension contrastive methods. To fully complete this comparison, I believe that methods such as MoCo, BYOL and DINO should also be included in this discussion (in Proposition 3.2), since they are also very well-known. Knowing how these methods align in this framework would be useful (mainly because BYOL and DINO also make use of a second teacher modle during training).

- As a minor issue, for the sake of clarity I believe that in their definitions of $L_c$ and $L_{nc}$, the authors should include the alignment term between the representations, even if it is the same across the two. This would make it easier for the reader to recognize that the main difference across methods is the regularization term used in each.

Minor issues:
There is a typo in Section 5.1, third line: “strateoes” -> “strategies”

clarity,_quality,_novelty_and_reproducibility: This paper provides an overall novel view of self-supervised learning, which may be useful to provide further insights in how the methods used in this field can be linked to each other. The paper is very clear and easy to understand, and I find it very good in quality overall. The authors also provide details to reproduce their experiments.
summary_of_the_review: I believe that this is a solid work, and barring some minor issues I have with the presentation I believe that it should be accepted in the conference. The framework proposed by the authors can be used to unify several different methods used in self-supervision, and this insight can be potentially used to further improve upon existing methods, by easily considering both contrastive and non-contrastive variations.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper makes an attempt at bridging the two broadly classified approaches towards self supervised learning, namely contrastive and non-contrastive. Recognizing that the methods can be viewed as sample contrastive and dimension contrastive respectively, the work categorizes popular contributions in the field under these regimes and attempts to consolidate the approaches by drawing
equivalence. Towards this aim, the paper (i) introduces two criteria characteristic of the regimes and demonstrates the
equivalence between them, (ii) attempts to empirically close the performance gaps between popular approaches under the regimes, given the equivalence, (iii) introduces methods that interpolate between the two regimes to isolate components of the loss functions for targeted study, (iv) attempts to validate the theory by improving the performance of SimCLR to match VICReg's and the robustness of VICReg to embedding dimensions like SimCLR.
strength_and_weaknesses: Strength:
The paper conducts many experiments towards substantiating the claims. The contribution is novel, to the extent of my knowledge, making a better understanding of the self-supervised learning methods. The work has potential for further theoretical research. 

Weaknesses:
1. The presentation is not clear, which needs significant revision. More details are given in the Clarity section.

2. It seems this paper only considers SimCLR and VICReg, while the title tries to cover the whole contrastive and non-contrastive learning methods.

3. Lack of focus on invariance criteria.
Although the claim is made that different similarity criteria are equivalent from an optimization point of view, experimental evidence in support of the claim is missing. I am not convinced that the differences between contrastive and non-contrastive learning methods only rely on the objective in Definition 3.2. I don't think different contrastive learning methods can be simply represented using definition 3.2, e.g., how can you show SimCLR can be written with Definition 3.2? Thus, I think there are many other factors in different contrastive learning methods, an using the designs in this paper cannot cover these.

4. Lack of KNN evaluation.
The paper speaks of the equivalence of the two contrastive regimes and illustrates the equivalence through the gram and covariance matrices, besides empirical evidence with downstream performance. We see that the off diagonal elements in specifically the gram matrix are significantly lower than their diagonal counterparts. But this begs the question of whether the performance of dimension-contrastive methods are at par with sample-contrastive methods for KNN based evaluation of features. KNN performance based experiments will be helpful towards reinforcing the claim of the equivalence of the two regimes.

5. Claim that the small batch size issues with contrastive methods are misleading (Section 5, Clearing up misconceptions).
The paper adequately demonstrates the merits of hyperparameter tuning and use of a better design choice for the projectors may alleviate the problems with small batch sizes, but it is in contrast with the work of [1] where they identify a systematic bias in the updates in SimCLR. This leads us to believe that a better landscape of hyperparameters or projector architectures may be available for large batches as well, reinstantiating the problem with smaller batch sizes in sample contrastive methods, which may require more experiments in those settings. Furthermore, keeping the architecture consistent for fair comparison, we still witness a drop in VICReg performance as the embedding
dimensions decrease (Figures 1 and S4, Table S5). 

[1] Yuan, Zhuoning, et al. "Provable stochastic optimization for global contrastive learning: Small batch does not harm performance." International Conference on Machine Learning. PMLR, 2022.
clarity,_quality,_novelty_and_reproducibility: Clarity:
The paper needs significant clarifications in many places. For example, the focus of the paper is on SimCLR and VICReg, however, there is not even a formal description for these methods. The authors are not supposed everyone ones these methods. Even they do, you also need to describe them so that they match your notation.

Also, as mentioned above, how Definition 3.2 can include all contrastive and non-contrastive learning methods is totally unclear and very confused. This part should also be carefully revised.

Quality and Novelty:
The overall quality of the paper is fair. The paper provides insightful theory and experiments for making contributions towards understanding self-supervised learning. Experiments that substantiate the equivalence of the several invariance criteria, justifying the focus
on only the contrastive criteria may be required. Further experiments towards the nature of embeddings learned and performance without classification heads may strengthen the claims.

Reproducibility:
The authors have provided the complete set of hyperparameters and models/architectures as well as pseudo-codes and learning rate procedures for reproducibility, without the official implementation.
summary_of_the_review: The paper attempts to answer the fundamental question of the parallels between sample and dimension contrastive approaches towards self-supervised learning. In that regard, the authors conduct many experiments supporting their theoretical motivations and provide a reproducibility statement for verification. However, the presentation of this paper is rather unclear, and the paper lacks some experiments that may serve to strengthen the claims, and may contain some erroneous in statements regarding the sensitivity of these approaches to batch size or feature dimensions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: 1. This paper interprets the covariance regularization-based non-contrastive methods as dimension-contrastive methods and interprets the traditional contrastive methods as sample-contrastive methods. It then proves that the dimension-contrastive criterion is equivalent to the sample-contrastive criterion when both the rows and columns of the embedding matrix are normalized. It also bounds the difference between these two criteria when the embedding matrix is not doubly normalized.  
2. In the experiments, this paper shows that the performance of several SOTA sample-contrastive and dimension-contrastive methods can be closed. This paper showed several design choices have a negligible impact on performance by studying the interpolation between VICReg and SimCLR. Along the way, it also improved the robustness to embedding dimensions of VICReg and improved the performance of SimCLR to match VICReg. It also showed that the normalization strategy does not have a significant influence on performance. 
strength_and_weaknesses: Strengths:
This is a very insightful paper that provides a unifying view of the contrastive and non-contrastive self-supervised learning methods. From a theoretical perspective, they proved that the sample-contrastive criterion is equivalent to the dimension-contrastive criterion up to the normalization of the embedding matrices. In the experiments, they further empirically verified that the performance of several SOTA sample-contrastive and dimension-contrastive methods can be closed. I believe this unifying view of contrastive and non-contrastive self-supervised learning can help us gain a deeper understanding of these methods. And many techniques and advantages of one family can be transferred to the other, as demonstrated in this paper. It might also be possible to improve the performance by designing a self-supervised learning algorithm that exploits both sample-contrastive and dimension-contrastive criteria. 

Weaknesses:
For the theoretical part, it's proved that the sample-contrastive criterion is equivalent to the dimension-contrastive criterion when the embedding matrix is doubly normalized. However, when the embedding matrix is not doubly normalized, the bounds in Lemma 3.4 and Corollary 3.4.1 is pretty loose. So I wonder if it's possible to prove tighter bounds in some more restricted theoretical settings. 
clarity,_quality,_novelty_and_reproducibility: Clarity:
In general, this paper is well written. Some minor comments/questions:
1. DCL seems undefined when it first appears in Prop 3.1.
2. In Prop 3.1, what's exactly SimCLR and DCL's criteria? It might be good to give a precise definition. 
3. In the last paragraph of page 3, what's the meaning of "more than two vectors cannot be the pairwise opposite for SimCLR and DCL's criterion"?

Quality:
The theory is pretty solid and the experiments are extensive. 

Novelty:
I think a unifying view of contrastive and non-contrastive methods is pretty novel. 

Reproducibility:
I think both the theoretical analyses and the experimental results are reproducible. 
summary_of_the_review: This is a very insightful paper that provides a unifying view of contrastive and non-contrastive methods. It interprets them as sample-contrastive methods and dimension-contrastive methods and proved they are equivalent when the embedding matrix is doubly normalized. In the experiments, they showed that the performance of several SOTA sample-contrastive and dimension-contrastive methods can be closed and many techniques originally developed for one family can be transferred to the other. I believe this unifying view can help us have a deeper understanding of self-supervised learning and guide the community to develop better algorithms.
review_title: null

Decision: Accept: notable-top-5%