Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a Variance-Invariance-Covariance regularization technique for self-supervised learning. The loss function used in the paper consists of three terms: the invariance term encouraging samples with different view to have similar embedding; the variance term, which is a hinge loss on the variance of the embedded variables (this is the main contribution of the paper, and the authors claim that it helps to avoid variance collapse); and a covariance term which borrows from the previous work Barlow Twin. The proposed method has greater flexibility for siamese architecture design, such as not requiring batch-normalization and weight-sharing, which the authors claim opens the door for multi-modal signal embedding. Experiments and ablation study have been conducted to demonstrate the performance of the proposed components.
main_review: Strengths:
+ The authors did a very good job in explaining the background and presenting the paper. The main idea is conveyed very clearly.
+ The idea of adding a variance term to the total loss to avoid representation collapse is interesting, intuitive and novel.
+ A great number of experiments compared with prior methods with detailed set up have been conducted.
+ Ablation analysis has also been conducted, showcasing the effects of different components.
+ A study on multi-modal signal representation learning is presented, demonstrating the importance of not requiring architecture or weight sharing in two branches.

Weakness:
- It seems that the main contribution, which is the variance term, plays a somewhat insignificant role in Table 1 and Table 2. In fact, compared to Barlow Twins, which does not have the variance term, the proposed method in many cases actually underperforms.
- Not requiring shared weight between different branches is a feature of Barlow Twin as well. Can the authors provide an explanation on the inferior performance of Barlow Twin in Table 3 and Table 5?
- The authors mentioned that using standard deviation instead of the variance in the hinge loss is important. Can a toy numerical example be provided to showcase the presence of representation collapse when variance is used?
summary_of_the_review: The paper is easy to understand, and has its contribution and novelty. Many experiments have been conducted, but theory is a bit lacking. I am willing to increase my rating if the authors can respond to my comments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper combines three objective functions for the self-supervised visual pre-training on ImageNet.

(1) The alignment between the two different views of an identical image, which is very common for existing methods;

(2) The covariance term to bring the off-diagonal coefficients of the features' covariance matrix to zero, which is modified from the Barlow Twins;

(3) The variance term that defines a hinge function on the standard deviation of embeddings along the batch dimension for every specific dimension of the feature projections . To the best of the reviewer's knowledge, such objective function is firstly applied for the visual pre-training in this paper, although the same measure has been used to analyze the model collapse problem (e.g., in the paper of SimSiam), but not be designed as a specific pre-trained loss function.
main_review: Strengths:

1. The paper is well-written and easy to follow;

2. The method is simple and achieve comparable performance for both linear evaluation and downstream transferring;

3. The authors provide a clear and detailed discussion to compare this work with the previous methods.


Weaknesses:

1. The reviewer does not feel very excited about the work. In fact, the three loss functions are not very novel. As the reviewer mentioned in the summary, the covariance term is just directly modified from the Barlow Twins. The same measure of the variance term has been used in some previous works (e.g., SimSiam) to analyze the model collapse problem, while it is not designed as a pre-trained loss function.

2. In the table 1, the comparison with previous methods might not be very fair. In particular, some compared methods such as MoCo v1/v2, SimSiam and InfoMin are just pre-trained for 800 epochs, while the proposed model is pre-trained for 1000 epochs. Besides, some of the previous methods do not use LARS optimizer and warmup strategy that are applied in this work.

3. While the proposed method is simple, however, the computation time of the covariance matrix is quadratic in terms of the feature dimension, which slow the pre-training significantly. 

4. Although the authors have provided detailed discussions to illustrate the differences of this work with previous works in terms of the design details, however, can the authors elaborate theoretically on the advantages of the variance and covariance terms against the whitening operation in W-MSE?

5. Besides ResNet-50, it will be more beneficial to the community if the authors can compare the proposed method with the MoCo v3, by showing the performance with the Transformer backbone.
summary_of_the_review: Overall, the reviewer tend to vote for accept for this work since the proposed method is simple and it has conducted thoughtful experiments to demonstrate the effectiveness.

The reviewer encourages the authors to speed up the proposed method, make the comparison with previous methods fairer and try to test the method on different architecture.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper propose a new self-supervised method. New loss is designed to explicitly avoid collapsed solution.
main_review: 
Advantages:
1. Authors give an explicit loss function to deal with the collapsed solution problem, which is understandable and explainable compared with BYOL and SimSiam. And the design of minimizing standard deviation for each dimension is insightful.
2. The application of minimizing variance and covariance to other methods, especially SimSiam, is interesting, which can help people understand the mechanism of how negative-free methods work.
3. Well-written and easy to follow.

Comments:
1. The invariance term and covariance term seems a decouple version of BarlowTwins. So I thought the main difference is the variance term.  However from the results, it seems that VICReg does not bring extra improvements compared with BarlowTwins. It is not clear that what kind of problem authors aim to solve. If the variance term is the key, it will be better to show the std of BarlowTwins features, and give more analysis of why the combination of variance-invariance-covariance is advantageous.
2. Authors emphasize that one of the advantages of VICReg is it does not require the weight sharing. It is indeed the VICReg can work without siamese network design, but the property maybe not a exclusive advantage of VICReg. According to my understanding, SimCLR, Barlow Twins can also work with two different architectures.  I thought authors should also compare with these method in the setting of non-shared architectures.
3. About the ESC-50 experiments. It is not clearly that why VICReg perform much better than BarlowTwins in this experiment. And I can not find details in the paper that whether BarlowTwins also use the multi-modal data. Because I believe that Barlow Twins can also work with different architectures, so it is important to figure out why VICReg perform better.
4. Table 4 shows the effect of variance term and covariance term on different method, but missing BarlowTwins. I believe the effect of variance term on BarlowTwins is a key experiment to compare. 
summary_of_the_review: The variance-invariance-covariance framework is insightful, but the experiments are not so convincing.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a novel objective function for self-supervised representation learning. The objective function consists of three terms, the invariance, the variance, and the covariance terms. The invariance term drives representations to be invariant to input transform, the variance term ensures each dimension of the representation has enough variability, and the covariance term inhibits co-adaptation of dimensions. The proposed objective function shows competitive performance to existing self-supervised learning techniques.
main_review: # Strengths

- The overall exposition of the paper is clear and easy to follow.
- The proposed method is simpler than the previously proposed self-supervised learning techniques. It is agreeable that the variance and covariance terms prevent the collapse of representations.
- The ability to handle the heterogeneous encoding networks seems to be a meaningful improvement.
- The proposed method requires a moderately sized batch of 2048.

# Weaknesses

- It is unclear that the collapse of representations, the main problem tackled by the paper, is the major bottleneck in self-supervised learning. The experimental results presented in Table 1 and Table 2 are okay, but not pushing the boundary of self-supervised learning. 
- While Table 3 and Table 5 showed that VICReg is more suitable for using heterogeneous encoders, the necessity of heterogeneous encoders is not demonstrated very clearly, because the setting is not practical. The performances reported in Table 3 are far from the state-of-the-art, and in Table 5, the shared weight setting performs best. A more natural setting, such as representation learning for multi-modal data as in VSE [1], should be investigated.
- The contributions of the variance term and the covariance term are not well analyzed. Table 4 is supposed to show the contributions, but it lacks CovReg column so that the conclusion from the table is somewhat vague. Additional efforts for illustrating the effect of the variance and the covariance terms will make the paper more persuasive.
- The difference from Barlow Twins needs to be elaborated in detail. Otherwise, the proposed method is conceived as a minor improvement over Barlow Twins. I found that the definition of the covariance term is meaningfully different from that of Barlow Twins, but it is not emphasized.

[1] Faghri, Fartash, et al. "VSE++: Improving visual-semantic embeddings with hard negatives." arXiv preprint arXiv:1707.05612 (2017).
summary_of_the_review: I vote to reject because the contributions of the paper are not well demonstrated in the paper.
review_title: null

Meta-review: This paper presents a self-supervised learning method for the multi-modal setting where each modality has its own feature extraction mapping, and i) the extracted features shall be close for paired data,  ii) in the feature space each view has close to diagonal covariance, while iii) the scale for each feature dimension is constrained away from zero to avoid trivial features. The presentation is clear and the reviewers do not have major confusion on the methodology. There have been some discussions between the authors and reviewers, and most questions on the empirical study have been addressed by the authors with additional experiments. The remaining concern is on the novelty (difference from prior SSL methods especially Barlow-Twins) and significance.  I think that while it is relatively straightforward to extend methods like Barlow-twins to the multi-modal setting, I do see the value of empirically demonstrating the effectiveness of an alternative loss to the currently pervasive contrastive learning paradigm, and hence the paper is worth discussion in my opinion. In the end, the method resembles classical multi-modal methods like canonical correlation analysis, in terms of the objective (matching paired data in latent space) and constraints (un-correlated feature in each view, and unit-scale constraint for each feature dimension); such connections shall be discussed.
Decision: Accept (Poster)