Kati8/ICLR_2023_2025 huggingface dataset:
authors: Zhiquan Tan
authors: Yifan Zhang
authors: Jingqin Yang
authors: Yang Yuan
reviews: 
Comments: This paper proves that contrastive learning with the standard InfoNCE loss is equivalent to spectral clustering on the similarity graph. Using this equivalence as the building block, the authors extend our analysis to the CLIP model and rigorously characterize how similar multi-modal objects are embedded together. Based on the maximum entropy principle, the authors demonstrate that the exponential kernels are the natural choices for capturing the local similarity structure for contrastive learning.
Strengths: 1. The  originality, quality, and significance are well supported by the proof of the equivalence of SimCLR and spectral clustering on the similarity graph and the extension to the multi-modal setting.

2. The clarity is satisfied based on the clear illustration of the analysis in Figure 1 and the clear motivations and contributions in the Introduction part of this paper.
Weaknesses: 1. The rationality of treating K_Z and \pi as MRFs and comparing the induced probability distributions on subgraphs should be better explaned, which is not well persuasive as shown in this version of the submitted paper.

2. For the definition of W, the authors are expected to further explain the mentioned unitary out-degree filter, which may be confused for the readers in understanding this definition.

3. The reason that cross-entropy loss can be converted to the combination of repulsion and attraction terms is expected to be further given after Lemma 2.4. Is it closely related to Lemma 2.5 and what is the specific relation?

4. In the experiment, the improvements of the proposed method is not obvious compared with SimCLR on the given datasets. The authors should further analyze the reason for 200 epochs and 400 epochs, respectively.

5. The authors should repeat each experiment for many times and list the mean and deviation to avoid the possible randomness, i.e.,  Table 1 and Table 4.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: - The paper proves that contrastive learning with the standard InfoNCE loss is equivalent to spectral clustering on the similarity graph, which is defined by the data augmentation process. 
- The paper extends this result to the multi-modal setting and shows that CLIP is equivalent to spectral clustering on the pair graph.
Strengths: - It provides a novel theoretical analysis of contrastive learning and its connection to spectral clustering, which can help understand the underlying mechanisms and principles of this popular self-supervised learning method.
- It proposes a new Kernel-InfoNCE loss with mixture of kernel functions that is inspired by theory and achieves better performance than the standard Gaussian kernel on several benchmark vision datasets
Weaknesses: - I think the motivation is not good enough, such a conclusion is easy to obtain, i.e.,  InfoNCE loss is equivalent to spectral clustering. Since graph is pariwise relationship and constrastive is also pairwise relationship, both have the similar objective.
- as the first point, I think the kernel infoNCE is also not well motivated. 

#### It's important to address these concerns regarding motivation in your paper. To improve the motivation for both the InfoNCE loss and the kernel InfoNCE, you might consider the following:

> InfoNCE Loss Motivation:

- Emphasize the practical significance and real-world applications of the InfoNCE loss. How does it relate to real-world problems or datasets in a way that goes beyond spectral clustering?
- Highlight specific challenges or limitations in existing methods that the InfoNCE loss aims to address.

> Kernel InfoNCE Motivation:

- Explain how the kernel InfoNCE extends the motivation from the InfoNCE loss. What specific problems or scenarios does the kernel-InfoNCE address that are not covered by the standard InfoNCE?
- Provide examples or use cases where kernel InfoNCE can be especially valuable.
> By offering a more compelling rationale and demonstrating the practical relevance of these concepts, you can strengthen the motivation for these components in your paper.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: The paper applies a probabilistic graph coupling perspective to view two typical constrastive learning methods, including SimCLR and CLIP, and interpretes them as spectral clustering or generalized spectral clustering. Moreover, it also attempts to propose to use exponential kernels to replace the Gaussian kernel. Preliminary experiments show that using a mixtures of exponential kernels to replace the Gaussian kernel in the SimCLR loss yields improved classification accuracy.
Strengths: + It is interesting to interprete the InfoNCE loss in SimCLR and CLIP into the perspective of probablistic graph coupling and thus find the connection to spectral clustering or generalized spectral clustering.
Weaknesses: - The reviewer was confused by the discussion before introducing problem (P1). Since that it is required that the $\mathbf \alpha$ has fewer nonzero entries, some objective of sparsity-promoting property is necessary. However, in (P1) an entropy regularization term is imposed. It is well known that the optimal solution for the maximal entropy problem in the discrete random variable is a uniform distribution. Here, the optimal solution for $\alpha_i$ should be $1/n$. It is weired to have a problem in (P2) and the solution in Theorem 5.1. Note that $\tau$ is the Lagrangian multiplier, i.e., dual variable, it is incomplete to have the dual variable inside. 

- Moreover, there are mistakes in the formulation of (P2). It is neither the Lagrangian nor the Lagrangian dual problem. It is misleading to claim minimizing (P2) producing an upper bound of (P1). 

- In Section 5.2, it is stated that Theorem 5.1 suggests that the loss function of that form is a natural choice for characterizing the neighborhood similarity structure. The reviewer cannot see this point. Such a form is nothing but a choice on purpose to use the maximal entropy (or due ot mistakes?). 

- In Eq. (6), it is a RBF kernel, cannot be directly yielded from the form in Theorem 5.1. Because having an exponential form does not imply to have the property of a RBF kernel. In this way, the so-called kernel-InforNCE is nothing but a heuristic form to define the similarity in the InfoNCE loss function. 

- The related work is not good. Some remarks on the previous work are either improper or even misleading. 

- The experimenal evaluation is limited.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 2 fair
reviews: 
Comments: The authors present a theoretical result concerning contrastive learning. Contrastive learning is a semi-supervised task that aims to map objects into an embedding space such that similar objects are close and dissimilar objects are far apart. Their results concern the widely-used SimCLR loss, an example of an InfoNCE loss. The authors show that optimizing InfoNCE is equivalent to solving a spectral clustering problem. Based on this theoretical insight, they give an argument that exponential kernels are natural and propose a variant of InfoNCE, Kernel-InfoNCE, where they use an alternative exponential kernel in place of the usual Gaussian kernel. Doing so led them to using a Simple Sum kernel, which achieves slightly improved empirical performance on CIFAR image-text data sets. 

This paper is closely related to HaoChen et al 2021; that paper proposed a type of contrastive loss that constitutes performing spectral clustering. The authors of this paper extend this by proving that SimCLR itself constitutes performing spectral clustering. This paper is also related to Van Assel et al., 2022, which analyzes dimensionality reduction methods such as t-SNE using a Markov random field framework adopted by the authors of this paper.
Strengths: This is an important theoretical result concerning a widely-used method. This work helps bridge the gap between theory and practice in contrastive learning. I have several comments, none of which are major.
Weaknesses: Minor comments: 

I found parts of the text difficult to follow because it lacks guideposts explaining the purpose of each section at a high level. 

It would help to have a definition of spectral clustering for the purposes of the paper. 

Eq1: I think this is meant to be a sum over different q's; the text says as much, but that's not how it's defined in Eq1. 

"we will flip $X_i$ to get $X_j$ with probability 1/9; it's not clear what "flip" means or why the probability is 1/9.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 4 excellent
Clarity: 3 good