Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a new Self-Supervised Learning framework with the Hilbert Schmidt Independence Criterion(SSL-HSIC). SSL-HSIC yields a new understanding of InfoNCE, while can optimize statistical dependence in time linear in the batch size. 
main_review: This paper is well written with detailed proofs and theoretical results.
This paper gives a unified view of contrastive learning through dependence maximization, by establishing relationships between SSL-HSIC, InfoNCE, and metric learning with detailed analysis. 

**Post Rebuttal Response**
I have read the response and would remain the current score.
limitations_and_societal_impact: Although the paper gives detailed theoretical proof, the experiments are somewhat weak. I still have some concerns:
1）The most related works SwaV and Barlow Twins outperform the proposed method in some experimental results, as shown in Table 1,2,5. What are the main advantages of this method compared with SwaV and Barlow Twins?
2) HSIC(Z, Y) can be seen as a distance metric in the kernel space, where the cluster structure is defined by the identity. Although this paper maps identity labels into the kernel space, the information of one-hot label is somewhat limited compared with views embeddings in Barlow Twins. 
3)Since the cluster structure is defined by the identity. How does the number of images impact the model performance? Do more training images make the performance worse or better ?

4) BYOL in the abstract should be explained for its first appearance.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes to use the HSIC metric in order to impose view feature dependency on the image identity during self-supervised learning (SSL). The authors made non-trivial efforts in both deriving the SSL-HSIC approach and theoretically linking SSL-HSIC with the existing mainstream SSL approaches. Empirical results show that the method offers competitive results. 
main_review: Strength: The paper is well written, and the paper is fairly easy to follow. The intuition behind the method is clear and convincing: HSIC regularizer explicitly penalizes the dependency between different images while it encourages views from the same images to be close. The theoretical results are strong. 

Weakness:
1. The eventual loss boils down to Eq. (11). Particualrly, if one employs Gaussian Kernel on the eventual loss on Eq. (11), it is immediate that the first term in Eq.(5)/Eq. (11) is proportional to BYOL up to a scaler, whereas the second term further plays additional contrastive role in the form of inter-image $\ell_2$ norm on top of the first term; In this sense, this extra contrastive term perhaps explains why SSL-HSIC always remains comparable to BYOL and behaves potentially superior over BYOL. However, my main concern is that, given that SSL-HSIC is closely related to BYOL, it seems the proposed approach SSL-HSIC is still inferior to the BYOL in most of the experiments shown in both Table 2 and 3. Is there any insight behind this phenomenon?  Intuitivelly, SSL_HSIC poses tighter constraints based on this extra contrastive term. Also, for the same reason, the empirical results failed to advacate on the SSL-HSIC side in comparison to BYOL, making the empirical justification much weaker for SSL-HSIC.  Would you like any discussion on this? 

2. The notation of $\Delta l$ term is confused, why this term is out of the sumation over index i and j, since it is an entrywise calculation given specific i and j?  
limitations_and_societal_impact: Yes, the authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: Authors proposes a novel HSIC based self-supervised learning loss for image representation learning. It relates to InfoNCE and Metric Learning and provides representations that perform well on multiple tasks. 
main_review: Paper introduces HSIC based SSL loss that learns generic image representations that perform well across multiple image related tasks. Strong empirical results on ImageNet classification and other tasks such as semantic segmentation, depth estimation and object detection.  It makes a good case where generic learned representations valuables as task specific representations will do better in terms of performance.

Framework uses unique identity of the image as part of loss function (y) which is a strong assumption but authors do describe it as an assumption which can be addressed.

Paper is well written with strong theoretical support of the intuition behind SSL-HSIC loss with ablation studies and choice of parameters.

A t-SNE visualizations of SSL-HSIC with class label would provide a qualitative analysis of the learn representations. It would be interesting to know if it helps improve  Top-1% accuracy of imageNet where SwaV does better (Table 1).


limitations_and_societal_impact: Limitations of the technique are addressed but bias and explainability aspects of learned representations are not, even qualitatively.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary:  
This paper proposed a new loss function for self-supervised learning using the Hilbert-Schmidt Independence Criterion (HSIC) to measure the kernel space dependence between the learned latent representation of images and their augmented versions. The main contribution is that authors demonstrated the connections between the proposed framework and InfoNCE, metric learning and Maximum Mean Discrepancy, respectively. An algorithm with random fourier feature kernel estimation is proposed and the method is shown to have comparable performance with state-of-the-art self-supervised learning (SSL) paradigms. The sample complexity and bias analysis for kernel estimation are provided.

main_review: Strengths:
- This paper offers an interesting viewpoint and an alternative formulation for maximal mutual information-based self-supervised learning (SSL). It makes much intuitive sense that one could maximize the minus HSIC to maximize the dependence between two random variables. The proposed method has some advantages in terms of measuring the gap between the population formulation and its sample based ones, reminiscent of HSIC and kernel estimation results. The combination of HSIC and random feature-based kernel computation is natural. The sample complexity analysis supporting this part makes sense. The regularization also makes sense, which is essentially a norm regularization in the kernel space.
- The paper is relatively well written, although the introduction of Y (image identify) may be confusing.


Weaknesses and questions:
- The novelty seems to be limited and impact in terms of changing the SSL paradigms could be marginal.  Essentially, the proposed method follows [35] but changes to another mutual information (MI) approximator. The effectiveness of this approximator is not rigorously shown but argued from a bit of a loose angle. In particular, the authors argued that exact MI maximization may not be desired due to invertible ambiguities. But it is hard to see why using a looser lower bound is a better choice. In other words, the motivation of using HSIC to replace MI is not crystal clear.  It indeed makes sense that maximizing dependence of different views may not give the best result of SSL. But it feels not easy to grasp the idea of intentionally using a loose lower bound for MI. Why is a loose lower bound better? How loose should one aim for? This is quite unclear to the reviewer. One suspicion is that maybe a good regularization (like the HSIC(Z,Z) regularization) can fix the ambiguity brought by the dependence criterion. How much performance gain is brought upon by the –HSIC part? Ablation study on $\gamma$ could perhaps provide some insights. 
- The connections between the proposed regularized HSIC and InfoNCE criteria are somewhat interesting, but not identical or equivalent. It does not feel that such a loose connection merits a new publication.
- Using the kernel method may make the computation a bit more challenging. Indeed, random features could alleviate the memory burden, but it introduces another level of approximation. This also goes back to the previous comment - there seems to be no strong motivation for using HSIC.

- Following the above comment, what is the unique or salient feature of this entire SSL framework is unclear. As a benefit of the proposed method mentioned by the authors, large negative sampling is not needed. However, to the reviewer’s understanding, [23] also has this advantage, which interprets contrastive loss as alignment and uniformity. How is the performance of the proposed method compared with [23]?
- The proposed formulation seems confusing. In Eq. (4) Y is used, but the actual estimation for HSIC(Z,Y) does not require the information of Y (otherwise it is supervised rather than self-supervised). This was quite confusing when first reading the paper, since SSL does not need the label information Y. By looking into the supplementary material, it was then clever that essentially the kernel dependence maximization is computed between the sample and its augmented counterpart. The Y information was never used in computation. Perhaps it would be better and clearer to elaborate on this gap between the conceptual formulation and the practical implementation. Is it needed to introduce the “label” or image identity term y when formulating the problem? Removing it may make the story smoother, in the reviewer’s opinion.
-  The analyses mostly are concerned with sample complexity and kernel estimation accuracy. These are of course not uninteresting, but are not exactly relevant to the main question, i.e., how the HSIC formulation affects SSL?
- The experiment results do not really show improvement compared with state-of-the-art in most cases. For example, in transfer learning tasks (Table 3), most results are not improved compared to other approaches, even the better ones are only with very small margins. Similar results are also observed in segmentation (Table 4) and object detection (Table 5) tasks. Besides, as mentioned above, study on gamma could be interesting and helpful to evaluate the contribution of the regularization term, i.e., HSIC(Z,Z). Overall, it seems that the proposed method does not make a big difference in terms of empirical performances. 
- On the computational side, it seems to require more computations when using kernel functions even though random Fourier features are employed. Is there a time consumption comparison with existing methods to illustrate the computation cost?   

- There are some discussions on scaling the HSIC estimator to reduce bias, and some analyses to support the design. How much does the scaling impact the final result? Could this be validated or supported by some simulations? This part also reads a bit confusing. From the paragraph, it seems that the goal is to handle non-iid samples. But the proofs in Supp Sec. A seem to still use iid assumptions. Can the authors clarify a bit?
- Overall, the HSIC based dependence measuring seems to be reasonable, but its motivation and impacts in practice are not crystal clear.
 
 
 
 


limitations_and_societal_impact: none noted
review_title: null

Meta-review: This paper studies the HSIC loss for self-supervised learning. While there was significant enthusiasm about this paper from the reviewers, there were also major concerns surrounding the conclusions of the experiments and questions about what to take away from the results. The authors do provide some good clarifications in their general response surrounding the fact that they don’t expect that HSIC should beat other methods and that this is not the point of the paper. However, the exact conclusions are still a bit muddy in the text. 

An important point: there is a high degree of similarity between the objective underlying HSIC and BarlowTwins. This must be clarified in the paper. Now, the description is rather vague and there are no explicit ablations to compare or understand the differences between the two. 

Decision: Accept (Poster)