Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Summary:
This paper proposes a regularization loss by resorting to rotation-invariant kernels. The method promotes embedding distribution to be close to uniform distribution in order to avoid collapse in self-supervised learning.


strength_and_weaknesses: Strength:

- This paper introduces a generic regularization loss based on kernel trick and might be of interest of the community to better understand self-supervised method from a kernel perspective.

- The introduced loss family uniforms several mainstream self-supervised methods. This includes uniformity-based and information-maximization methods as a special case of the proposed framework under different kernel choices. 

- The method is able to reduce computation time and memory efficiently with the help of kernel trick while it remains competitive on downstream tasks


Weakness:
- W1:  Why L is only constrained to be 2 and 3? What happens when L becomes bigger? Is there any relevant ablation studies on this? 

- W2: Is it possible to explain the relationship between different kernel choices and the eventual feature quality? How does different kernel choices affect downstream task performance? I asked this because it seems they all have the same goal of pull the embedding distribution to be close to uniform distribution. How can we know that which choice of kernel is the best among all other kernels? Also, it seems you don't achieve SOTA performance on all settings.

- W3: It might be helpful to give a brief introduction about kernel mean embeddings as preliminaries to improve readability for those who are not familiar with this topic.

clarity,_quality,_novelty_and_reproducibility: The paper is well written. The idea reflected in the paper seems to be novel to me. 
summary_of_the_review: Given the novelty, technical and theoretical contribution of the paper, I recommend acceptance of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a generic regularization loss on self-supervision learning of image representations. The regularization is proposed to enforce the uniformity of transformed representations hypersphere. The author show that the proposed loss family accommodates several previous self-supervision learning methods. Moreover, the author emphasized that the proposed regularization with truncated kernel can significantly save memory and runtime. Experiment results show the advantages of applying SFRIK in self supervision learning.
strength_and_weaknesses: Pros:
The author proposed SFRIK for avoiding representation collapse in invariant representation learning. 
The author did thorough study on the truncated kernel and drew connection between the proposed method and information maximization.
The author conducted experiments for both representation quality and runtime complexity measurement.



Cons:
1. The author proposed a family of losses which are powered by different choice of kernels. However, the choice of RBF kernel and generalized distance kernel have already been studied in existing works. as the author mentioned in table 1. Hence the novelty of the proposed method shrunk to the adaptation of truncated kernel.
2. The author mentioned that the proposed SFRIK can help saving computation and memory complexity comparing with VICReg due to the kernel trick. However, applying RBF kernel should provide the same saving. As the author mentioned adopting RBF kernel is nearly equivalent to the AUH method. The saving on computational complexity shouldn’t be considered as the novelty of SFRIK.
3. In the experiment section. Table 2 shows AUH method performs quite competitive to the proposed method. However, in Table 3, the author show the performance of applying RBF kernel is much worse than the proposed truncated kernel. The author may add some explanation of the discrepancy if AUH works equivalent as applying RBF kernel.
4. The motivation of applying the truncated kernel was not clear enough. Though this kernel function was not investigated in self-supervision learning. The author may explain more on why the adoption is necessary.
clarity,_quality,_novelty_and_reproducibility: The clarification of the novelty part ma need some revision.


summary_of_the_review: Overall, the paper needs some clarification of the motivation and novelty.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper deals with self-supervised learning and proposes a method named SFRIK that complements a Siamese architecture with a regularization term that consists in the MMD discrepancy between the embedding distribution and the uniform distribution on the hypersphere. In essence, the idea of this term is to prevent representation collapse by making sure that the learned embeddings are uniformly spread out across the hypersphere.
The methods interestingly unifies and subsumes several previously proposed uniformity-based self-supervised learning procedures, including recent ones such as VICReg, and connects to information-maximization methods.
The paper exhaustively examines the effect of the various hyperparameters (such as expander dimensionality, kernel, and kernel truncation order) of SFRIK and compares its performance in terms of quality of the learned representations to those learned by the main competing algorithms like SimCLR, SwAV and VICReg on multiple downstream on ImageNet1k and Places205. The conclusions are that SFRIK is competitive with these algorithms both in terms of quality of the learned embeddings for downstream classification tasks, as well as in training speed and memory efficiency.
strength_and_weaknesses: This paper has no obvious major weaknesses and several strengths:
- The construction of the method is based on solid theoretical foundation connecting to kernel methods, integral probability metrics (MMD) and spherical harmonics.
- The paper established interesting and wide-reaching connections to other methods, unifying uniformity-based self-supervised learning and information-maximization methods.
- SFRIK enjoys different complexity scalings compared to for instance VICReg thanks to its use of the kernel trick, making it clearly advantageous in specific regimes like large expander dimension and small batch size.
- The paper delves into rigorous empirical comparisons with the major competing methods (SimCLR, SwAV, SimSiam, VICReg), fixing architecture and probing different embedding dimensions on multiple downstream tasks of interest (linear probing, kNN classification, semi-supervised learning), and even comparisons with teacher-student methods like BYOL.
- The paper includes a rigorous study on the effect of the main hyperparameters, such as embedding dimension, kernel type and kernel truncation order.
clarity,_quality,_novelty_and_reproducibility: The paper is very clearly motivated and presented.
The method is theoretically supported and the thoroughly empirically benchmarked.
The method is novel and solidifies connections between methods putting them on solid theoretical footing.
The authors will provide code to reimplement their results, which meets the gold standard of Reproducibility in the community.
summary_of_the_review: Very interesting and relevant work proposing a new class of self-supervised learning regularizations, based on solid theoretical foundation with possibly wide-reaching implications, even beyond the convincing empirical results already presented in the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper tackles the problem of self-supervised learning without negative pairs (such as that used in InfoNCE) and defines a so-called “loss family” which expresses existing methods such as VICReg as a specific case of the family. The paper also proposes the use of a truncated kernel which results in a linear scaling of the loss complexity in relation to the SSL embedding dimension (as opposed to quadratic for VICReg). The proposed method shows respectable performances in comparison to the state-of-the-art, while requiring less memory and training time compared to VICReg in particular.
strength_and_weaknesses: The proposed method is quite elegant, and can be understood reasonably well in an intuitive manner. The related works are positioned and explained well. 

The method-wise contribution of this paper can be summarized as the use of a truncated kernel for the uniformity loss (called SFRIK). Though the mathematical formalization of the loss formulation and the rigorous discussion of related work is very much appreciated, one must wonder about the simple question: are the proposed changes demonstrated in a convincing manner. While the improved RAM and training time (shown in Tab. 5) are great, the main results shown in Tab. 4 does not paint a convincing picture as the proposed method mostly reaches performance parity in linear probing and does not perform well in the semi-supervised 1% case.

The authors perform their main experiments on a benchmark they dub “IN20%”. This is not a standard benchmark, so the reviewer would like to ask about the rationale for selecting this setting. Furthermore, despite the heavy comparisons to VICReg, many experiments are missing that were conducted in the VICReg paper (such as VOC07, iNat18, COCO det+seg, MS-COCO retrieval). While not all experiments should be required, one wonders if there are sufficient tasks being considered to argue for the general applicability of the proposed method.
clarity,_quality,_novelty_and_reproducibility: This paper is written well and is well motivated. It is relatively easy to follow the authors’ reasoning, though background knowledge of SSL methods and in particular VICReg is necessary for a smooth reading.

The provided mathematics should allow for a high-level of reproducibility, though no code release is promised in the submitted manuscript.

The work is quite original and seems to be mathematically sound.
summary_of_the_review: The paper was a pleasure to read through, and the formalization of uniformity in instance-wise SSL was done well. The experimental results are reasonably convincing, but fall short when compared to similar works in literature. I would like to read the comments from the other reviewers, as well as the rebuttal from the authors before making a more certain recommendation.


**Edit after rebuttal period**

I have carefully gone through the other reviews as well as the responses provided by the authors. I believe that the authors' responses are rigorous, earnest and sufficient. I agree in spirit with the authors that pre-training on IN100% with ResNet-50 is certainly excessively expensive when considering a fair and sufficiently hyperparameter-tuned experimental setting. Thus, I raise my rating to 8.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces a regularization loss based on kernel mean embeddings with rotation-invariant kernels on the hypersphere for self-supervised learning of image representations.  The regularization enforces the uniformity of transformed representations on hypersphere to prevent feature collapse.  As part of the strengths, the proposed regularization with a truncated kernel can also save memory and runtime.

Reviews of this paper are mostly positive. The use of a positive definite kernel for self-supervised learning provides a new perspective (3r4M). The proposal unifies existing approaches (3r4M, uDSt, w91x). The paper is well written (3r4M), and offers a thorough empirical study (7yz3). Concerns raised were on poor performance of the RBF kernel, and lesser performance gains in the semi-supervised setting. These were resolved with rebuttals which provided more experimental results, convincing the reviewers.

Recommendation: accept.

*Suggestion from the AC*:
The use of positive definite kernels in self-supervised learning makes this work closely related to “Self-supervised learning with kernel dependence maximization”, NeurIPS 2021. I strongly suggest that the authors make a qualitative comparison to it. The aspects of encouraging uniformity on a hypersphere, unifying some existing methods, and the use of MMD (as opposed to HSIC) provide some distinction. However, considering that HSIC can be interpreted as MMD, this work must be better contrasted.

justification_for_why_not_higher_score: Though the results are interesting and novel, the core proposals (regularization loss based on kernel mean embeddings with rotation-invariant kernels on the hypersphere, truncated kernel) build on earlier works. These are thus not at the level of groundbreaking results. 


justification_for_why_not_lower_score: This paper was very well received by the reviewers. The paper has nicely put together existing results and proposes an interesting approach.
Decision: Accept: notable-top-25%