Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper provides a theoretical study of the popular self-supervised learning (SSL) techniques VICReg, SimCLR and BarlowTwins. The main results are closed-form optimal representations for each method as a function of the training data and the sample-relation matrix (which indicates which samples are from the same class, e.g. this could be constructed as the matrix that indicates which samples are augmentations of the same image). The paper further provides simplified versions of these expressions in linear settings and uses them to show equivalence between the SSL methods and various spectral methods, and provides a study on downstream task performances.
strengths_and_weaknesses: Strengths

1. The paper studies the important problem of theoretically understanding increasingly popular SSL methods.

2. To my knowledge, no other work has derived such expressions concerning SSL optimal solutions, or made the connections between SSL and spectral embedding methods.

3. The scope of the paper is comprehensive: all of the most popular SSL techniques are analyzed. Clearly a lot of work went into writing this paper.

Weaknesses

1. It is difficult for me to understand the insights of this work because the writing is so dense. Often expressions are introduced without sufficient background and conclusions drawn without sufficient explanation and/or motivation. E.g. $\mathcal{L}\_{var}$ and $\mathcal{L}\_{cov}$ in Theorem 2 and equations (13) and (14) - intuitively how do these equations arise and why are they useful? The figures are especially information-overloaded and difficult to parse. The claims relating to downstream task performance are not clear. Overall, it should be abundantly clear what the key messages are from this paper, but they are lost in the density of the work.

2. There is insufficient discussion of other theoretical studies of SSL (only a brief mention in the introduction).

3. Some of the statements and proofs are not rigorous, as it is not formally defined what it means for one method to recover another. The proof of Theorem 3 requires an assumption that is not made in the theorem statement. Other proofs can use more formality (only English text is given and the reader is left to fill in the details) and the proof of Lemma 1 is missing.

questions: If the authors can provide additional explanation of Figures 2, 3, and 4 that would be helpful.
limitations: The limitations of the theory are not discussed, and an assumption is missing (see above). There are no potential negative societal impacts.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a general framework that leverages spectral methods to unify the representative self-supervised learning (SSL) methods such as VICReg, SimCLR, BarlowTwins. The authors provide a detailed theoretical analysis of different SSL frameworks and demonstrate the properties of their learned representations.  
strengths_and_weaknesses: ### Strength:

Unifying existing self-supervised learning methods with spectral methods looks interesting. It also provides new insights into the community to help understand how SSL works. Both the motivation and the technical details in the experiment designs seem sound. The authors have cited most of the relevant papers and well summarized previous works. 

### Weakness:

Although the paper provides an insightful view of understanding the mechanism of SSL, most of the analysis is only valid in the pre-assumptions, some of which might have a gap in practical usage. For example, the properties of VICReg are based on the linear network assumption, while in practice, the encoder backbones are often non-linear. As the authors claim this paper aims to provide some guidelines to practitioners, these gaps could limit their insights to them. 

The example in Eq (4) and (5) is a little bit hard to follow. It is not obvious see the logical relation between these two formulations. 

The presentation quality is fairly good but needs further polishing. It feels like the paper is finished in a rush, and some paragraphs are not clear to the readers. Some abbreviations are not explained with the full name or corresponding citation when first appearing in the text, e.g., DA, DN, which might confuse the readers.   

The experiment/simulation settings are not specified in the paper, although most validations are conducted on toy examples. For example, what datasets are used for these proof of concepts? 

Most of the discussions are based on the assumption that the downstream task is classification, while in practice, SSL methods are widely used in various settings, including detection, segmentation, generative modeling, etc. The insight to practitioners might be limited in the scope of the classification. 

Minors: 

The notation $h$ in Eq (3) is not used; simCLR -> SimCLR

Line 91: loose -> lose; 

Line 98: variosu -> various
questions: I am generally satisfied with the paper, but have an open question based on the analysis:

As these methods have different properties, while some of them are complementary, e.g., VICReg prevents dimensional collapse, while SimCLR could have a dimensional collapse, is there any way to propose a solution that complements each other's advantages? 
limitations: The authors have discussed the limitation of the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper explores the links between self-supervised losses, metric learning and spectral embedding methods. It specifically investigates which representations are learnt when the embedding is chosen to be linear, and carefully investigates how the choice of loss affects their rank. 
strengths_and_weaknesses: **Strengths**

Though many connections between metric learning, SSL and spectral embedding methods have been previously explored [1, 2, 3], this paper is a welcome addition. Explicitly writing out the optimal representations in the linear regime is useful. Explicitly showing how the choice of SSL loss impacts the rank of the learnt representation (in comparison of the rank of the downstream information encoded in G) is  a welcome contribution to recent investigations on the "dimensional collapse" of SSL representations. 


**Weaknesses: better referencing known terminology and problems**

Given the authors are connecting SSL to known Spectral Embedding methods, it would add significant clarity to draw on known terminology as well. For example, VICReg is composed of three terms that are eponymously described as 'variance-invariance-covariance' with a rather new vocabulary of 'dimensional collapse' and 'semantic' similarity. On a simpler level, the VICReg loss is simply Robust Nonlinear PCA, where Terms 1 and 2 'whitens the signal' and Term 3 encourages robustness to perturbations (e.g. jitter or rotation) encoded in G. Specifically,
- Term 1 removes the scaling ambiguity of PCA (by setting the variances to 1)
- Term 2 encourages orthogonal representations
- Term 3 encourages robustness to user-specified transforms (e.g. rotation)

Relating this to known terminology (PCA) is useful because it *explains* the observations made by the authors, when they say
- "the optimum is not unique" or "there exist many local minima": this is due to well-known indeterminacies for PCA (e.g. permutation, offset) and is not a new problem
- Fig 2, gamma = 0 vs. gamma = 0.1 is essentially PCA vs. Robust PCA. Expectedly, PCA has known indeterminacies (Panel 3) and Robust PCA blurs the landscape (Panel 3) and smoothens the latent space (Panel 2) as it desensitizes the loss to local perturbations. 

It seems like these observations are framed as a new problem of VICReg when they in fact inherit from PCA. 


Abbreviation: SSL = Self-Supervised Learning.

[1] Agrawal, Ali, Boyd, 2021. Minimum Distortion Embedding. 

[2] Lee, Lei, Saunshi, Zhuo, 2021. Predicting What You Already Know Helps: Provable Self-Supervised Learning.

[3] Pfau, Petersen, Agarwal, Barrett, Stachenfeld, 2019. Spectral Inference Networks: Unifying Deep and Spectral Learning. 




questions: **Clarification**
1. Figure 3 is intruiging. Indeed, VIC is composed of three terms: the "variance" and "covariance" whiten the representation, while the "invariance" term incorporates "downstream" task information (e.g. an image and its rotation should have the same label). However, Figure 3 shows that the downstream task can be perfectly when the "invariance" term barely intervenes (gamma = 0.001, just enough to make Z full rank). In other words, this suggests that a (nonlinear) PCA or ICA would be enough to solve a downstream task, which is counter-intuitive to SSL which posits that injecting downstream task information via pairwise relations (matrix G) is essential. Could the authors comment on that?
2. The spectral analysis VICReg and Barlow Twins are presented as separate contributions, but isn't Barlow Twins subsumed by VICReg (setting the 'variance' term to zero)?
3. The authors reference "BarlowTwins, DINO, BYOL, and SimSIAM" but DINO and BYOL do not seem to appear subsequently in the analysis: could the authors clarify?
4. It is not very clear to me how equation (4) is built and subsequently used.
5. "The authors write "VICReg" recovers [...] Linear Discriminant Analysis (LDA)": can the authors specify is that is when the whitening is enforced (Theorem 2) or relaxed with alpha and beta (equation 1)?
6. The sentence "This already brings a contrast from non-contrastive methods showing that contrastive methods learn to produce signals such that the graph estimate is close to the known graph" seems like it is over-reaching. I am not sure there is a clear definition of "contrastive" and "non-contrastive" methods in the first place: it could be possible after all to reformulate VICReg as a contrastive loss with implicit negative samples. Also, SimCLR learns a graph estimate (G_hat in equation 15) while VICReg does not (equation 1), but that is a choice in the loss. Perhaps it is clearer to compare VICReg with SimCLR directly, without extrapolating to "non-contrastive" vs. "contrastive"?
7. [1] reformulates PCA and Kernel PCA using a Laplacian given in Section 3.3. "Principal Component Analysis", matrix Lij. This corresponds to solving the "variance" and "covariance" terms of VICReg in this submission, as they are enforced in Theorem 2. Could the authors comment on that? Does that connect with the authors' equation 9 which combines the Laplacians of the PCA terms ("variance" and "covariance") with the Laplacian of the downstream task ("invariance" to G)?




 **Minor Typos or Formulations**
- "We can directly remind the VICReg" -> recall
- "variosu" -> various
- "there exists many local minimum" -> there exist many local minima
- "to solve the task at hand perfect" -> perfectly
- "Prior to moving" -> Before moving
- "unify SSL methods under the realm" -> under the helm
- "similar pairwise distance (usually l2) than" -> similar [...] to
- "we ought ot" -> to
- "poitn" -> point


**Related Work**

I believe there is very relevant work that could be referenced by the authors. 
- [1] unifies many embedding methods (e.g. PCA, spectral methods, MDS, UMAP, Isomap) under the helm of pairwise metric learning
- [2] already draws a link between SiamSiam and nonlinear CCA
- [3] connects spectral methods (over a Kernel, e.g. Covariance or Graph Laplacian) with SSL objectives, and possibly provides theoretical justification for tricks of the trade (e.g. 'stop-gradient' for so-called 'Non-Contrastive' SSL to avoid collapse). 


Abbreviation: SSL = Self-Supervised Learning.

[1] Agrawal, Ali, Boyd, 2021. Minimum Distortion Embedding. 

[2] Lee, Lei, Saunshi, Zhuo, 2021. Predicting What You Already Know Helps: Provable Self-Supervised Learning.

[3] Pfau, Petersen, Agarwal, Barrett, Stachenfeld, 2019. Spectral Inference Networks: Unifying Deep and Spectral Learning. 

limitations: I have no potential negative societal impact to raise.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a unfied framework for self-supervised learning with the help of spectral manifold learning .the authors show that VICReg, SimCLR, BarlowTwins are specical cases of their proposed framwork. They points out the prerequisite about the production of optimal self-supervised representations for downstream tasks.
strengths_and_weaknesses: Strengths
1. this paper is well written and easy to follow
2. the idea of specral manifold learning for unifying SimCLR(NNCLR), BarlowTwins and VICReg seems to interesting for me
3. this paper is technically sound

Weakness
1. I think the authors overclaim their contributions of this paper. From my view, they have only consider VICReg, BarlowTwins and SimCLR in this paper. Although They have mentioned DINO in this paper, they do not reveal the connection between DINO and others.

questions: 1. how the loss of DINO can be reformulated into the way of Eq. (5)?
1. can the sprectral manifold learning be applied on DINO so as to unify it into your proposed framework?
if the authors can address my concerns, I will raise my rate.
limitations: No
review_title: null

Meta-review: metareview: This paper focuses on providing some theoretical intuition/understandings of popular self-supervised learning (SSL) methods. The authors develop closed-form optimal representations for various method as a function of the training data and the sample-relation matrix. The authors also provide further intuition by developing simplified versions of these expressions in linear settings which they use to show an equivalence of sorts between SSL and various spectral methods and how it affects downstream tasks. Overall the reviewers were positive and thought the paper had nice insights. They did raise some concerns about the quality of exposition and various detailed technical issues. Most of the technical issues seems to have been addressed by the authors in their response. I concur with the reviewers. The paper has nice insights and therefore I recommend acceptance. I do however recommend that the authors further polish the paper for the camera ready version by addressing the issues raised by the reviewers especially about the exposition.
confidence: Certain
Decision: Accept