$\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs

Authors: Vlad Sobal, Mark Ibrahim, Randall Balestriero, Vivien Cabannes, Diane Bouchacourt, Pietro Astolfi, Kyunghyun Cho, Yann LeCun
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: contrastive learning, vision language model, simclr, clip

Abstract:
Learning good representations involves capturing the diverse ways in which data samples relate. Contrastive loss—an objective matching related samples—underlies methods from self-supervised to multimodal learning. Contrastive losses, however, can be viewed more broadly as modifying a similarity graph to indicate how samples should relate in the embedding space.  This view reveals a shortcoming in contrastive learning: the similarity graph is binary, as only one sample is the related positive sample. Crucially, similarities \textit{across} samples are ignored. 
Based on this observation, we revise the standard contrastive loss to explicitly encode how a sample relates to others. We experiment with this new objective, called $\mathbb{X}$-Sample Contrastive, to train vision models based on similarities in class or text caption descriptions.
Our study spans three scales: ImageNet-1k with 1 million, CC3M with 3 million, and CC12M with 12 million samples. The representations learned via our objective outperform both contrastive self-supervised and vision-language models trained on the same data across a range of tasks. When training on CC12M, we outperform CLIP by $0.6\%$ on both ImageNet and ImageNet Real. Our objective appears to work particularly well in lower-data regimes, with gains over CLIP of $17.2\%$ on ImageNet and $18.0\%$ on ImageNet Real when training with CC3M. Finally, our objective encourages the model to learn representations that separate objects from their attributes and backgrounds, with gains of $3.3$-$5.6$\% over CLIP on ImageNet9. The proposed method takes a step towards developing richer learning objectives for understanding sample relations in foundation models.

Decision: accept (poster)
Reviewer scores: 5, 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1730106455949

thread_type:
official_review

<thread_object>
creator: reviewer_mxji

<content>
summary:
The paper argue for revising the standard contrastive cross-entropy loss from binary targets of a sparse augmentation graph to a soft target graph where the target values can be between [0, 1]. The new method is called X-CLR. The authors suggest to use extra metadata associated with the dataset to find the soft target distribution, and in this work mostly use the pairwise similarities from a pretrained text encoder on text captions/descriptions to encode this distribution. Although the authors note that the framework is general, "the similarities can come from any source including any metadata". Conduct experiments to train vision models with text captions/descriptions. The study spans different datasets and the results show improvements over contrastive baselines.

strengths:
1. The paper is well-written and easy to read.
2. Experiment study spans different datasets and comparisons to contrastive baselines.
3. The authors test with different similarity graphs (random graph, graph inferred from metadata, augmentation graph, true class graph), and find that a soft graph can improve over the binary graph of existing contrastive approaches.

weaknesses:
1. Relying on extra metadata and a pretrained model limits the applicability compared to the contrastive baselines. I also think this is a simple solution.
2. The idea of soft-targets for cross-entropy based contrastive loss is very general. It also lack some novelty to the paper's related work on soft-targets.

questions:
Suggestion: Finding a soft target graph without extra metadata or pretrained model with self-supervised techniques would give better applicability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730294867367

thread_type:
official_review

<thread_object>
creator: reviewer_r31v

<content>
summary:
This paper introduces a new objective, called $\mathbb{X}$-Sample Contrastive Learning, where the binary designations in standard contrastive learning are replaced by a soft target based on similarities between samples. These similarities can be obtained through any preferred method, with this paper mainly using frozen text encoders for the experiments. Moreover, the method can be applied fairly easily to any contrastive objective, with this paper focusing on the InfoNCE objective.

The proposed method is trained on ImageNet, CC3M and CC12M and compared against one supervised learning method and various SSL methodologies. It is shown that $\mathbb{X}$-CLR leads to consistent improvements for classification and background decomposition compared to the baselines, with minimal increase in computation.

strengths:
This paper recognizes an important problem, which is that commonly used contrastive objectives assume binary similarity scores, which in many settings do not make much sense. The proposed method is an intuitive way of generalizing contrastive learning to non-binary similarity scores, effectively dealing with this problem. $\mathbb{X}$-CLR is tested through a comprehensive suite of experiments, showcasing its effectiveness and highlighting some of the important design choices that should be taken into account when using this objective.

The paper is well written with a clear explanation of the problem of existing contrastive learning methods, followed by a great presentation of the proposed method. Moreover, the experiment descriptions are both clear and concise, effectively highlighting the strengths and weaknesses of the method. Finally, the paper is well structured overall, making it enjoyable to read.

weaknesses:
In my opinion, there are no significant weaknesses in the paper. The only thing I was wondering regarding the experiments is how fair the comparison between $\mathbb{X}$-CLR and the SSL methods in Table 1 is. It seems to me that $\mathbb{X}$-CLR is a supervised method in this setting. If the authors agree with me, then it might be nice to explicitly mention this in the text to avoid any possible confusion. That being said, I do think including the comparison is important to showcase the effect of proper similarity scoring, so this is a minor remark.

Aside from that, I have some small notes regarding the presentation, which I will list below:
  - It may be useful to shortly mention what $f_\theta$ and $K$ are after equation (1) in line 202 if space is not a concern. It is reasonably clear from the context, but being explicit rarely hurts.
  - In the data  scarcity setting (figure 3a), SupCon appears to be better than $\mathbb{X}$-CLR, while the text mentions that $\mathbb{X}$-CLR improves over SupCon.
  - The coloring of SupCon and SimCLR in figures 3a and 3b is flipped, which is a bit confusing at first glance.
  - The paragraph related to table 4 does not mention training on CC3M, which might be useful to mention for clarity.

questions:
As mentioned before, I think the given experiments paint a clear picture of the method. So the questions listed here are mostly out of interest and not things that I necessarily think make sense to include in the paper. That being said, I was wondering a few things about the method:
  - What happens in imbalanced settings? I imagine that the similarities for an instance with rare caption/label/meta data would be much lower than for instances with more common caption/label/meta data. Do you think this would have a strong effect on the method? Have you tried experimenting with for example ImageNet-LT?
  - An annoying problem of contrastive learning is its sensitivity to the batch size ($B$), which stems from its effect on the ratio of positives to negatives across the training set ($B^{-1}$). I imagine that $\mathbb{X}$-CLR partially solves this problem, especially for larger batch sizes, given that the similarities $\mathbf{G}_{ij}^{(\text{soft})}$ approximately follow $\mathbb{P}\_X(\text{sim}(X\_i, X))$, where $\mathbb{P}\_X$ denotes the population distribution on inputs $X$. The only bias being the small effect of the true positive $\mathbf{G}\_{ii}^{(\text{soft})} = 1$. Is this something you looked into? If yes, how sensitive is $\mathbb{X}$-CLR to changes in the batch size? Furthermore, does the application of the softmax to obtain the distribution $s$ have any negative effects for larger batch sizes (since $s_i \rightarrow 0$ when $B \rightarrow \infty$)?
  - Do the similarities for ImageNet (shown in figure 5a) come from the way the text encoder input is constructed (so from the fixed template part)? If this is the case, do the similarities not give a false view of how strongly related images are? I expected there to be at least some very low similarities for each image, although I might be misunderstanding something here.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730503393620

thread_type:
official_review

<thread_object>
creator: reviewer_qwur

<content>
summary:
This paper proposes a new contrastive objective that uses a target similarity graph based on additional information or models. The modification is applied to contrastive losses in both multi-modal learning (image-text CLIP training) as well as self-supervised learning (images). The proposed modification is called “X-Sample Contrastive Loss” or X-CLR, where the model assumes access to an external source of metadata and possibly pretrained models that can define a graph of similarity between inputs. The goal is to replace the 0-1 hard similarity graph used in contrastive methods with soft similarity graphs based on some ground-truth information.

strengths:
- Unifying SSL, multi-modal contrastive learning, and distillation methods in one formulation is helpful in defining generalized training methods.
- The ablation on the similarity sources in Table 2 is interesting as it shows strong text encoders are not helpful for X-CLR loss than more simple sentence transformer.

weaknesses:
- The paper should clearly separate the SSL methods on images from image-text methods such as CLIP and clearly define X-CLR modification for both setups. There is a clear difference between contrastive losses for the two setups, one relies on image augmentations to define contrastive pairs while the other relies on paired image-text datasets. Figure 1.b presents a pseudocode for the X-CLR loss that is valid only when paired image-text data is available. In fact, the only reason this loss does not collapse to a trivial and all equal solution is that the `text_emb` is detached from the computation graph.
- The tables and comparisons should specify which models have access to additional data and/or additional pretrained models. For example, Table 1 compares X-CLR to SimCLR while X-CLR not only has access to the labels, but also access to pretrained text models. The comparison to methods that do not have the advantages of X-CLR is not fair. Specifically, when the method using the labels in ImageNet to define the loss, it can no longer be fairly compared with SSL methods that do not have access to labels. For example, the analysis in section 4.5 that compares the overhead of X-CLR to SimCLR needs to account for the training time of the text encoder as well.
- On line 354, the paper claims that “X-CLR is more general than model distillation methods as the similarities can come from any source” and cites the TinyCLIP [1]. Are the authors claiming X-CLR can be better than CLIP distillation methods? If so, the paper should compare to state-of-the-art CLIP distillation methods such TinyCLIP [1] and MobileCLIP [2]. That being said, I did not find a clear definition of X-CLR for image-text loss that allows for training both image/text towers even if it uses pretrained image and/or text pretrained models. Specifically, the loss should have image2text and text2image terms or show that it can learn a comparable representation without these terms.

1. Wu et al, Tinyclip: Clip distillation via affinity mimicking and weight inheritance, ICCV 2023.
2. Vasu et al., MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training, CVPR 2024.

questions:
- Are both Tables 1,3,4 using pretrained frozen sentence transformers as the text encoder?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730591042879

thread_type:
official_review

<thread_object>
creator: reviewer_hzrt

<content>
summary:
This work presents X-CLR, a contrastive loss objective that replaces the usual binary positive/negative samples in contrastive learning with soft labels that can vary inside [0,1]. The resulting loss function requires pairwise similarity labels, which the authors suggest can come in the form of metadata or via text embeddings of captions generated from class labels. A comprehensive set of experiments on ImageNet and CC3M/CC12M show improvement in performance of X-CLR relative to existing contrastive learning baselines on classification tasks.

strengths:
1) The proposed method and its advantages are intuitive and easy to understand, and there is a comparison with existing vision-based contrastive loss functions.

2) The suite of experimental results is comprehensive, and considers many useful ablations. I particularly appreciated the practical experiments about finetuning and refining existing vision backbones, and the explorations of high-quality vs noisy similarity data.

weaknesses:
1) There is limited novelty to the proposed contrastive learning loss function, which is a relatively simple swapping of binary positive/negative sample labels in existing contrastive learning losses with a soft label.

2) Pursuant to the first point, there not much detailed analysis of how this method related to existing ones beyond a comparison of the binary vs. soft label graph.

3) The presentation of the paper is inconsistent - there are some very concise and informative figures, like Figure 1, but for example, there are many wording issues throughout, and there are other places where more detail is needed (this is particularly true for table captions).

4) Some weaknesses in experimental results - there is only one network architecture tested, though I accept the given justification to some extent.

questions:
1) Some of the results you get for competing methods are far below what those methods report in their papers. Just as an example, in Table 1, ImageNet accuracy from your experiment with SimCLR is 63.4% vs. the 69.3% reported in their paper, and SupCon is 74.3% in your table vs. 78.7% reported in their paper. This is a very significant difference, since in the case of SupCon, their reported result outperforms X-CLR. Did you look into the large differences here? Can you explain them?

2) Please add some more detail to the table captions. The captions right now generally tell me what the takeaway should be, but they don't tell me the specifics of what I'm looking at: things like what the numbers actually represent (accuracy), what the columns mean (i.e., for Table 1, "Accuracy of networks pretrained on ImageNet with various contrastive learning algorithms, and evaluated on a set of..."), etc. Also, I think it is a stretch to call Theorem 1 a theorem - this is an re-expression of these losses in matrix form, and the cited paper doesn't even offer a proof. I think these can simply be stated.

3) I found the CLIP experiments section confusing. In your CLIP baseline, are you actually training a vision encoder and language encoder jointly with a CLIP loss, or are you taking a pretrained CLIP model and performing training with X-CLR loss? Also, in general I thought a deeper analysis with CLIP was missing from Section 3, particularly in the case of generating similarities with captions + a CLIP text encoder. Is X-CLR equivalent to CLIP training with a frozen / pre-trained text encoder in this case? If not, how is it different? I think this comparison is important to contextualize your method.

4) In the case of ImageNet, you use class labels to generate text captions, and then embed these to get text embeddings, which ultimately determine the connection strengths in the graph. This creates a similarity matrix that has some particular properties, i.e., images within the same class have a similarity of 1, as in the Supervised graph in Figure 1, along with other nonzero entries based on the similarities of two different images' classes. Does this end up being the same as SupCon? There are some similarities with supervised learning here, but there is no analysis of this similarity.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731987480797

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We thank all the reviewers for taking the time to read our submission and providing the feedback!

We are glad that the reviewers found:
- that the problem we investigate is **very important in contrastive learning literature** (reviewer R31V)
- our method **intuitive and easy to understand** (reviewers HZRT, R31V)
- the **experiments comprehensive** and interesting (reviewers HZRT, QWUr, R31V)
- the overall paper **well-written** (reviewers R31V, MXJi).

# Revision updates
We have and updated the paper and uploaded the revision with the following changes based on the provided feedback:
- Revised table captions to make results more clear (reviewer HZRT)
- Added columns to distinguish methods that use labels to table 1 to clearly show what data the learning methods have access to (reviewers QWUr and R31V).
- Fixed figure colors in figure 3a and 3b (reviewer R31V).
- Added a section in the appendix comparing our method to CLIP to address any confusion regarding the relationship between X-CLR and CLIP objectives (Appendix 10) (reviewers HZRT and QWUr) 
- Added pseudocode to show X-CLR loss in the general case, when the similarities don't come form the text encoder (Appendix 11) (reviewer QWUr)
- Mentioned that we train all methods for 100 epochs and do not explore training longer as was done in SimCLR and SupCon papers in the main text (reviewer HZRT). As mentioned in the response, we believe CC12M experiments are more valuable than training for more epochs on ImageNet as they test X-CLR in a more real-world setting with noisy labels.

# Main concerns
- Lack of novelty in the objective (reviewers HZRT and QWUr) - While we agree that the extension of this loss function to soft similarities is simple and is akin to the distillation objectives, to the best of our knowledge, **using such a loss function to inject external similarity information into the objective is novel. Moreover, we believe simplicity does not detract from the novelty.** We also believe this is an important contribution because, as reviewer R31V pointed out, **we consider an important problem in contrastive learning objectives.**
- Experiments not supporting the method enough (reviewers HZRT and QWUr) - Our experiments explore different choices of the similarity graph (table 2), comparison to a wide range of baslines on ImageNet (table 1), fine-tuning performance (table 4), extension to the large conceptual captions dataset with 12M samples (table 3), KNN evaluations, temperature ablations and low-data regime performance (Figure 3). **We believe our experiments thoroughly test this method in a variety of settings, with different datasets, benchmarks, and probing methods. Reviewers HZRT QWUr and R31V agree that the experiments are comprehensive.**

We hope the improved revision addresses the concerns you have. We remain available to answer any additional questions you may have. Thank you for helping us make this paper better!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission proposes X-CLR, a contrastive learning objective that incorporates soft similarity labels derived from external metadata or pretrained models, as opposed to traditional binary similarity scores. While the method shows some performance improvements over baseline methods in classification tasks, there are critical concerns raised by reviewers regarding the novelty, fairness of comparisons, and the robustness of the experiments.

The proposed method offers a practical solution to a well-recognized problem in contrastive learning, with demonstrated performance improvements. While some reviewers remained skeptical of novelty, they acknowledged the potential value of the approach for future research.