Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a simple and effective method(VICReg) that prevents the collapse in self-supervised learning with two regularizations terms. The variance term maintains the variance of the embedded vector along each dimension independently. The covariance term prevents the network encode similar information to the same dimension in the embedded space.
main_review: Pros:
VICReg achieves comparable results with the SOTA on several downstream tasks.
VICReg does not require some training constraints on the architecture, which is more applicable.
The paper is clearly written and well organized.

Cons:
The covariance term is borrowed from the Barlow Twins method, which decreases the originality and significance.
The improvement of Variance Reg and covariance Reg on other self-supervised methods is marginal. It would be better if VICReg can be plugged into different methods.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: Summary:
This paper proposes a new self-supervised learning method. It makes self-supervised learning very elegant: it does not require techniques such as weight sharing between the branches, batch normalization, feature-wise normalization, output quantization, stop gradient, memory banks, etc., and achieves results on par with the state-of-the-art results on several downstream tasks. It does not require that the inputs be of the same nature. This opens the door to the use of non-contrastive self-supervised joint-embedding for multi-modal signals, such as video and audio. I really like this paper. Still, I have some concerns about this paper.

main_review: 

(Positive) This paper correctly understands the critical problem in self-supervised learning, i.e., the main challenge is to prevent a collapse in which the encoders produce constant or non-informative vectors.

(Positive) This paper makes self-supervised learning very elegant: it does not require techniques such as weight sharing between the branches, batch normalization, feature-wise normalization, output quantization, stop gradient, memory banks, etc., and achieves results on par with the state-of-the-art results on several downstream tasks.

(Positive) This paper correctly understands the critical problem in BYOL, i.e., the dynamics of learning and how they avoid collapse, is not fully understood.

(Negative) There is a tradeoff for Variance, Invariance, and Covariance. I am afraid that different hyper-parameter will result in very different performances. If so, the proposed method will not be robust. Especially when I see Coefficients λ and μ are 25. Could the authors provide some dense results/visualization curves beyond Table 5?

(Positive) The following property is fascinating: it does not require that the inputs be of the same nature. This opens the door to the use of non-contrastive self-supervised joint-embedding for multi-modal signals, such as video and audio. I really like this property very much!
Importantly, this good property also indicates: The ability of VICReg to function with different parameters, architectures, and input modalities for the branches widens the applicability of joint-embedding SSL to many applications, including multi-modal signals.

(Negative) Unfortunately, in most of our experiments, this paper uses a Siamese net architecture in which the two branches are identical and share weights.

(Negative) Although I like the above property very much, finding a real-world case meeting the property is difficult.  Because if we cross-modal learning, this is supervised learning but NOT unsupervised learning. Please refer to OpenAI's CILP. Could the authors provide a real-world case meeting the above property with an experimental result?

(Negative) In the overall paper, the authors consider BYOL and MOCO very different. Specifically, BYOL is regarded as a distillation method, while MoCo is considered as a contrastive learning method. I am not convinced by this categorization. If the authors think deeply, they will find that BYOL and MoCo are almost the same, except that MoCo has negative samples. The EMA networks (or mean teachers)  in MoCo and BYOL are exactly the same. If BYOL is categorized into knowledge distillation, MoCo could also be considered knowledge distillation, distilling knowledge from both positive and negative samples. Actually, I tend not to categorize methods like BYOL into knowledge distillation because the teacher is just a mean teacher with dynamic changes during the training. Specifically, SimSiam even has no mean teacher, and the learning target is just a copy of the student networks without BP.

(Negative) Is the following sentence correctly written?
Line 107-108: These methods are effective, but there is no clear understanding of how t but suffer from a lack of explainability regarding the way collapsing solutions are avoided.

 (Neutral) I think this paper may over-attribute some contributions from Barlow Twins. Actually, it borrows the decorrelation mechanism of the Barlow Twins. But it includes an explicit variance-preservation term for each variable of the two embeddings and thus does not require any normalization.

(Positive) The description of both motivation and the method are clear and easy to follow.

(Negative) Would you please provide an empirical analysis of gamma in Eqn (1), which is a constant target value for the standard deviation?

(Positive) The following observation is good:
"Using the standard deviation and not directly the variance is crucial. Indeed, if we take S(x) = Var(x) in the hinge function, the gradient of S with respect to x becomes close to 0 when x is close to x ̄. In this case, the gradient of v also becomes close to 0, and the embeddings collapse."

(Negative) The following results are not good: "We report the performance in Table 2, VICReg performs on par with most concurrent methods, and better than Barlow Twins, across all classification tasks, but is slightly behind the top-3 on detection tasks. "
The explanation is not persuasive: "This could be explained by the fact that VICReg learns representations that are more invariant to transformation, but eliminates more low-level information about the images than the other methods."

(Neutral) According to Table 3, the proposed method should have used symmetric loss (i.e., Eqn. (8)). Please clarify.

(Negative) The following sentence is confusing:
Line 315: This hurts the accuracy by 1.1%. Who is compared with who?

(Negative) Two online networks increase both the memory and the running time.

(Positive) I strongly agree with the following observation (in my experiments, I also have these observations): "Using multi-crop with VICReg did not yield any performance improvement and showed signs of overfitting. More generally, multi-crop does not seem to help VICReg, Barlow Twins [9], SimSiam [7] nor BYOL [6], but yields performance improvements with SwAV [5], SimCLR [12] and MoCo [3], which might be related to the fact that these methods are contrastive."

(Negative) Provide evidence for the following claim: "Self-supervised learning is also trained on data, which may also have a bias.
Using the method described here is not likely to mitigate the usual issues with machine-learning systems due to biases in the data or the model architecture."

(Negative) The authors claim that multi-cropping yields performance improvements with SwAV [5], SimCLR [12] and MoCo [3], which might be related to the fact that these methods are contrastive. This is not true. For example, DINO and the Triplet loss method are not in line with this claim. They don't belong to contrastive learning, but Multi-crop learning does help them. Maybe the gain is related to loss function but not contrastive learning.
DINO: Emerging Properties in Self-Supervised Vision Transformers
Triplet Loss: Towards Solving Inefficiency of Self-supervised Representation Learning

(Negative) Because the proposed method can handle two different networks with different architectures and different weights. I would like to see such an example. Especially when one network is ResNet-50 and another network is a larger network than ResNet-50 (e.g., ResNet-101 or a strong ViT). Could the linear evaluation result of ResNet-50 be very high in these cases? It would be interesting to see such a result because it implicitly contains a knowledge distillation effect that distills the knowledge from the larger network.

limitations_and_societal_impact: 

Please refer to and respond to the "negative" items in the above main review.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose a self-supervision technique with a more flexible structure requirements such as  not requiring batch norm, normalization, weight sharing or quantization.  Their model uses 2 encoders and a loss function with 3 variance and covariance regularization terms.  The authors mention that compared with the previous works, their main novelty lies in one of their variance terms which adjusts the variance of each input dimensions and, in return expected to mitigate collapsing problem. The authors claim that not sharing weights and having regularization terms separate on each encoder branch make their model better suited for multimodal inputs and tasks. They show results mainly on ImageNet classification, object detection tasks, with an ablation study on their and other baseline methods by varying regularization terms.
main_review: ##### *Originality:*

- The idea of trying to prevent collapsing and using decorrelations, also working on a more flexible model exists in the recent works. The authors' main contribution lies in the details of the formulation of their loss. For example, if we compare with Barlow twins method, we can say that authors' invariance term corresponds to changing the cosine metric to euclidean for the diagonal covariance terms(so the same idea but different metric here). They kept covariance regularization separate for each embedding model as another difference. Exactly using hinge loss to control standard deviation dimension-wise is novel as of my knowledge. 
- Overall, the work is novel in the sense that they collected all these details in one model, but I see these modifications as training tricks rather than a new loss function for self supervision, due to a lack of any theoretical analysis or at least a proof of concept with simpler cases(My opinions about the intuition part are below). 

- The  authors did a great job in the supplementary to compare and separate themselves from the previous works.(Also Figure 2 is very informative) I believe they could be mentioned in the main paper in a better, more organized way. 

- The related work was adequate, figures and comparisons in the supplementary section could be a good part of a survey paper in my opinion. 

- I appreciate the authors efforts for their detailed and transparent ablation analysis (Table 3,5,6 ), a good example for our field.

&nbsp;
##### *Quality:*

- The authors both present their weaknesses and strengths in their analysis with sufficient experimental detail, it is admirable, but they could provide more intuition why other methods do better than theirs. 

- The claims could be better supported. Some examples and questions(if I did not miss out anything)
     - Why using normalization is a problem for a network or a task (it can be thought as a part of cosine distance)? How would Barlow Twins perform if their invariance term is replaced with a euclidean distance? 

     - Your method still uses 2048 as the batch size, I would not consider it as small. For example, Simclr uses examples in the same batch and its batch size changes between 256-8192. Most of the methods you mentioned need even much lower batch size. 

     - You mentioned not sharing weights as an advantage, but you have shared weights in your results, except Table 4 in which the results degraded as you mentioned. What stops the other methods from using different weights? It should be possible even though they have covariance term between the embeddings, how much their performance would be affected compared with yours? 
My intuition is that a proper design might be sufficient rather than separating variance terms. 

     - Do you have a demonstration or result related to your model collapsing less than other methods? In line 159, you mentioned gradients become 0 and collapse; it was a good point, is it commonly encountered, did you observe it in your experiments?

     - I am also not convinced to the idea that the images and their augmentations need to be treated separately, they can be interchangeable. 

    - Variances of the results could be included to show the stability of the algorithms since it was another claim in the paper(although "collapsing" shows it partly, it is a biased criteria since the other methods are not designed for var/cov terms).

    - How hard is it to balance these 3 terms?

    - When someone thinks about gathering two batches from two networks and calculate the global batch covariance in this way; it includes both your terms and Barlow Twins terms. Can anything be said based on this observation, about which one is better and why?
   
&nbsp;

##### *Significance:*

- Currently, the paper needs more solid intuition or analysis  or better results to make an impact in my opinion. The changes compared with the prior work are minimal. Most of the ideas and problems in the paper are important, but they are already known.

- The comparisons with the previous work is valuable to the field, they could maybe extend their experiments to the more of the mentioned methods or other variants.  



limitations_and_societal_impact: The authors did a great job in presenting their work's limitations, their results in general not being better than the previous works and their extensive analysis(tables). If they did a better job in explaining the reasons/intuitions in a more solid way, or include some theory if there is any, I would be inclined to give an accept.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a Variance-Invariance-Covariance regularization (VICReg) for self-supervised learning. The total loss function is the weighted sum of three terms: 1. distance between two views of the same input batch; 2. hinge loss on the variance of the embedding variables, encouraging them to stay away from zero; 3. loss on the covariance between different embedding variables, encouraging them to be close to zero, and hence preventing a representation collapse. The proposed model does not require the embedding branches to be identical, nor does it require mining of contrastive pairs. Experiments are conducted to demonstrate the effectiveness of the proposed regularization.
main_review: ** Pros **
1. The paper is well written and well organized. It is easy to read.
2. The proposed variance term is easy to understand and implement.
3. Experiments and ablation study has been conducted to verify the effectiveness of each component of the proposed VICReg.

** Cons **
1. The novelty and contribution of the proposed work seems to be limited: the authors have stressed multiple times that VICReg does not require the two branches to be identical. But, if I understand correctly, the prior work Barlow Twins also does not have such restriction.
2. Also, the work seems to be built on the idea of Barlow Twins, with the only addition of the variance term in the proposed VICReg.
3. The fact that the proposed work does not require batch-wise or feature-wise normalization does not strike me as extremely important. After all, in table 6, the accuracy drop after standardizing the embedding is only 0.2%
4. The choice of hyperparameter also seems a bit willy-nilly -- why is $\lambda = \mu$? Is there any specific reason for such choice?
5. The method has only comparable accuracy compared to Barlow Twins, which makes me wonder what is the major gain from using variance regularization.
6. In 5.1, the authors mentioned that the performance of VICReg is on par with state of the art without using asymmetric networks architectural tricks such as the momentum encoder of BYOL. Is there going to be performance gain when such asymmetric network tricks are used? After all, the fact that the proposed architecture does not need to be symmetric is a highlight of the work.

**Review after author response**
I appreciate author's response addressing my concerns (2-6). The preliminary experiment on comparing Barlow Twins and the proposed work is also promising, but more study on the performance gain is warranted. I am raising my rating to 5.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: 3 of the 4 reviewers voted to reject this paper based on the lack of significant differences from prior work, especially Barlow Twins, and the lack of SOTA numerical results. One concern I share with a number of the reviewers is that what is "simple" is somewhat subjective. E.g. it is not obvious that normalization is especially complex or replacing it with a (co)-variance regularizer is necessarily "simpler". While one reviewer points to potential significant gains with multi-modal data, this is a hypothesis not supported by specific claims in the paper as it stands, and therefore that cannot be considered in the current review process. As a result, I am recommending rejection. However, the approach will be a good paper with some revisions (e.g. additional results on multi-modal data).
Decision: Reject