Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new self-supervised learning objective that aims to further adapt the previous decorrelation-based [1] contrastive learning method [Bardes, 2021], which largely alleviates the trivial solutions in SSL. Compared with prior arts, the proposed Zero-ICL/FCL are constructed from mainly the aspects of instance and feature-wise. In particular, Zero-CL requires no negative samples; feature-wise FCL discards the redundancy term by feature-wise whitening, and the proposed ICL prevents the collapse of contrastive learning effectively. Quantitative evaluations validate that Zero-CL leads to on-par performances with previous state-of-the-art results.


main_review: This work presents a good extension on the basis of the previous regularization-based SSL method [1], and further, proposes Zero-CL from the instance and feature-wise aspects. The empirical experiment shows only incremental performance gain over the prior arts like Barlow Twins, yet the proposed Zero-CL has an obvious complexity advantage as it's not dependent on the feature dimension. 

Some technical doubts regard the Zero-CL:
1. It seems the parameter \lambda regulates the weight of feature/instance terms, is there any intuition behind the setting of lambda=1?

2.  According to what is shown in Figure 3, L_{fea} has a consistent Acc@1 trend with Barlow Twins (larger dimension yields better results) but there does have a seemingly optimal hidden-size (1024 as shown in the Figure). Does this indicate that there's a conflict between the instance/feature-wise objective? In such a case, how could one decide the best-hidden size besides empirical experiences?

[1] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906, 2021.
summary_of_the_review: - The paper is very constructed and has a clear motivation, comprehensive ablative studies, and good analytic discussions.

- The only concern from the reviewer is whether the complexity advantage is a sufficient contribution over the previous regularization method [1], given that Zero-CL shows only incremental performance gain. 

- The exhibited theoretical analysis of Zero-CL with previous efforts gives direct and clear comparisons.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper is concerned with preventing the model collapse in the self-supervised learning scenario using contrastive losses. The main addition is an adaptation to the existing formulations that performs instance or feature whitening, avoiding the use of negative examples than can be expensive to store and to compute the similarity. Experiments on the selected datasets show that the method generally matches the current top-performing methods.
main_review: Pro:
+ The proposed formulation can be integrated into various methods as a plug-and-play component.
+ The approach is clean and well backed-up both theoretically and experimentally.
+ In-depth ablation study

Cons/Quest:
- How frequent is the mode collapse to trivial solutions in practice for the competing methods? Most of this approaches are quite stable in practice even without using negative samples (i.e. SimSiam).
- What is the actual advantage in wall-clock training time? Does the lower complexity in theory translates well in practice?
- In Table 1, what is the explanation behind the fact that the proposed approach lags behind as the number of iteration increases?
- Can this approach scale well to larger models too?
Minor: 
- Section 2, a few paragraphs don't start with a capital letter.
summary_of_the_review: Overall the idea looks clear and efficient, it's not clear however how problematic for the current approaches the collapse is and some additional clarifications could strengthen the work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel contrastive loss by whitening the embedding vectors in two ways: along the instance dimension and along the feature dimension. The results are comparable to recent works.


main_review: This loss function is somewhat new. However, the justification is relatively weak. The work is good, though at this stage, the authors have only been able to show it gives satisfactory results. It is unclear whether this is just an incremental work among the recent advances.

This formulation is still using other samples as negative samples. The negative samples are unrelated to the positive samples  in the loss function. This is nice but in theory it is still quite similar to the recent contrastive learning approaches.

Typo: eq 9 Z -> H.
summary_of_the_review: The paper is satisfactory.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the collapse problem during self-supervised contrastive learning. ICL and FCL methods are proposed to decorrelate instances and features. Zero-Contrastive Learning can discard negative pairs with advantages in negative free, reducing correlation of different features to zero and retaining information during transformation. Promising experimental results are demonstrated on CIFAR and ImageNet.
main_review: + This paper dedicates to whitening transformation with a ZCA-based solution. It is intuitive to apply the whitening to both instance and feature dimensions, which makes the objective function of ICL and FCL consistent.
+ Paper is well presented and easy to follow.

- The key concerns came from the symmetric framework. Despite the fancy and consistent solution, the problem is down to the different properties of feature dimensions and instances. For image classification, we believe the ideal feature representation should have decorrelated non-redundant dimensions. However, the correlation between instances is inevitable - the inner-class correlation should be encouraged. Is a pure orthogonal matrix that pushes every instance unique the best solution? Should it be a "block-based" matrix with ones filled in the same class?
- Another big concern is about the claimed contribution in complexity. It is argued the second term in Eq(13) results in square order complexity. And the improvement is made by the replacement of the whitened embedding. I am not super clear why this can reduce the computational complexity.


- Eq (9) seems an error: should it be Z_{i,d}^{A,Ins}\dot Z_{i,d}^{B,Ins}?
- Decorrelating for redundancy removal can often result in low-rank effects (it is equivalent to implicit feature selection or sampling) in the transformed matrix. It would be good to discuss and potential properties and problems resulting from the low-rank issues.
- In the literature review, the background, properties and advantages of ZCA compared to PCA and Cholesky should be extended to clarify the motivation of using ZCA.
- Eq(4) the "1" should be in bold to denote an identity matrix?

summary_of_the_review: Overall, it is a good paper. I am keen to understand why the symmetric framework can be consistently applied to features and instances given they have different properties in the contexts of image classification. I am also not clear about the complexity reduction, which seems to be claimed as a very important contribution in this paper.

Minor issues include some typos in equations and the discussion of low-rank property and ZCA background.
review_title: null

Meta-review: The initial reviews for this paper were somewhat diverging, however the paper did not receive any significant negative criticism to push it towards below the acceptance threshold. The reviewers have found some minor issues about the paper. Following the reviewer recommendations, the meta reviewer recommends acceptance.
Decision: Accept (Poster)