Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper provides an analysis of different whitening losses used in Self-supervised learning, seeking to interpret some empirical observations, e.g. the connection between whitening losses and the asymmetric methods (BYOL/SimSiam), why PCA does not work, and why whitened outputs are not good representations. The paper also proposes channel whitening with random group partition (CW-RGP), which is shown to be an effective whitening loss. CW-RGP is evaluated on image recognition (ImageNet1k and 4 other benchmarks), object detection (VOC/COCO), and instance segmentation (COCO).
strengths_and_weaknesses: Strengths

i) the proposed random group partition is technically sound;

ii) the performance of the proposed CW-RGP method looks promising on some benchmarks e.g. COCO object detection/ instance segmentation;

iii) an ablation study on the batch size is provided;

iv) the writing is clear in general.


Weaknesses

i)  The interpretations about “why PCA does not work” and “why the whitened output is not good” are not convincing. To me, the explanation could be much simpler and more intuitive: the batch whitened outputs rely on the batch statistics: an image may have different whitened representations when computed in different mini-batches.

When using PCA, the descriptor of an image relies on the eigenvectors (U in L136), which may change dramatically across mini-batches. This explains why BW-based approaches prefer large batch sizes. It explains the experimental results shown in Fig. 4. It also explains why whitened outputs are not good representations, i.e. experiments in Fig. 3.

Note that the predictors in the asymmetric methods (L197), on the other hand, do not rely on batch statistics, which I believe is a key difference.

ii) The comparisons in Table 2 may not show the full picture, e.g. baselines like BYOL/SWAV may be significantly under-trained. Here, the batch size for BYOL/SWAV is 4096. When trained for fewer epochs (e.g. 200 epochs), a large batch size may hurt the performance as it leads to significantly fewer training iterations. It would be better if baselines like BYOL/SWAV-batch-size-512-epoch-100/200 are also included.


iv) Channel whitening has been proposed before in [47] for the same task. As [47] has been published in ICLR 2022. I’m not sure if [47] could be considered a concurrent work. Compared to [47], the new content is the random group partition. This extra design may not be enough for NeurIPS. Overall, I believe [47] should at least be included as a baseline, and the ablation on the random group partition should be included.

Minor issues

i) L18-19 “two networks are trained …[8]”, I think there is only one network 

ii) L286: “rand” → “random”

iii) Table 1, Simsim → SimSiam

iv) Table 1, references are included for some baselines (SimCLR, BYOL), but not all (e.g. Shuffled-DBN, W-MSE)

v) Table 2 & 3, it would be better if references are included.

questions: i) When evaluating the performance of the whitened representation (L167, Fig.3) I wonder if the whitening matrix (\Phi(Z)) is estimated per mini-batch or over the whole training set?

ii) L308-309, are “d=2” and “d=4” here referring to “g=2” and “g=4”?

limitations: Limitations are discussed in the main paper. Potential negative societal impacts are not discussed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper gives out a thorough investigation into whitening loss applied in self-supervised learning. Based on their analysis, they proposed a channel whitening method named CW-RGP. From their experiments on variuos datasets, CW-RGP gets the SOTA in most cases.
strengths_and_weaknesses: Strengths:

1, The writing is nicely done and the paper is organized well for understanding.

2, The experiments are quite sufficient, including the analysis experiments and comparison experiments.

3, The understanding for whitening loss is convincing and there are also some new indicators proposed for future study.

Weakness:

Overall, I didn't see much disadvantages in this paper, I'm just curious about several points below.

1, The barlow twins and VICReg methods are not present in all tables. In the related work, the authors have classified them as soft whitening, so I think some comparison with them should not be neglected. 

2, Although, Barlow Twins might get even better results than CW-RGP (from my experiments before), I don't think it can be very important. This paper mainly studies the whitening loss, thus I think the comparison with whitening loss based method (such as W-MSE) is more important.
questions: I would wonder whether the Barlow Twins method can be better. 

Then grouping the experiments based on their techniques is easier to show the contribution, especially this paper belongs to the whitening loss branch. 

Finally, the comparison methods from Table 1,2,3 are a bit different, actually I was finding the W-MSE results in Table 3. Can you explain a bit?
limitations: None
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: - Self-supervised learning approaches need to avoid collapse to a trivial representation. This has been tackled by various approaches in literature including use of negatives, use of asymmetric networks etc. Use of whitening loss has been explored by some recent works. 
- This paper studies this whitening loss and various variants of the whitening transformation used in practice. 
- The paper investigates some issues with the transformations used and a new random group partition based channel whitening approach to prevent collapse. The approach works well for large batch sizes which is shown through experiments on datasets like ImageNet and transfer to COCO. 

strengths_and_weaknesses: Strengths:
- The paper seeks to analyze various approaches used for whitening of feature representations used for SSL. The paper is very well written. 
- The paper does a good job at explaining some of the preliminaries.
- Previous approaches have been analyzed extensively through experiments based on public repositories. 
- The paper decomposes the whitening loss and connects it to common SSL approaches. 
- Obtains state-of-the-art results on standard benchmark datasets. 
- I think the paper will be of significance to the wider ML/SSL community. 

Minor Weaknesses, suggestions and questions :
- Since one the Section 3.3 is one of the most important contributions of this paper, I would suggest having some more details & intuitions behind the explanations. Especially : L185-196. This could perhaps be included in the supplementary material. 
- Can the centering+sharpening operation used in DINO & equipartion constraint used in SwAV be looked at through a similar analysis ?
- L280: To ensure $\frac{d}{g} > m$, do you modify the last layer of the projection head ? 
- How does the proposed approach compare to state-of-the-art on training time ? 
- Do you have some thoughts on why the gap in performance decreases at 100 ep vs 200 ep (Table 2) ? What about W-MSE 4 @ 200 ?
- I agree with the motivation in L263-266. I think from a practical standpoint, it is be a better idea to report required GPU memory & GPUs per approach in Table 2 in addition to the batch size. 

questions: Please refer to the section on "Strengths And Weaknesses" for questions and comments. 
limitations: - The authors have adequately discussed limitations of the proposed approach. 
- While the work is more at a fundamental level, I do urge the authors to include some discussion on potential negative impacts.
review_title: null

Meta-review: metareview: This paper studies the impact of whitening losses used in recent Self-supervised learning (SSL) methods. It shows that the symmetric whitening loss can be decomposed into two asymmetric losses, explaining important behaviour experimentally observed (*e.g.* why some whitening transformations -*e.g.* PCA- are not always effective, and why whitened output is not always a good representation). The authors proposed a channel whitening with random group partition (CW-RGP) with good experimental performances.

The paper initially received mixed reviews: 2 borderline reject and 2 accept recommendations. The main reviewers concerns related to the soundness of the proposed analysis of the whitening loss, the fairness of the experiments, and the positioning and comparison to other recent baselines (*e.g.* [47], DINO or SwAV). The rebuttal did a good job in answering the reviewers' comments, and there was a consensus that the paper should be accepted after the rebuttal. 

The AC's owns reading of the submission confirmed that the paper is a solid contribution for NeurIPS. It considers that the proposed analysis of whitening loss is valuable for the community, and that the proposed CW-RGP is meaningful and experimentally validated. Thus, the AC recommends acceptance. 
The reviewer's consensus for acceptance has been reached given the authors' feedback including promises to improve the clarity and insights of the analysis, and regarding the positioning with recent baselines and new experimental results. Therefore, the authors are highly encouraged to carefully include these improvements into the final paper.

confidence: Certain
Decision: Accept