$\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs
Authors: Vlad Sobal
Mark Ibrahim
Randall Balestriero
Vivien Cabannes
Diane Bouchacourt
Pietro Astolfi
Kyunghyun Cho
Yann LeCun
Venue: ICLR 2025 Poster (2025)
Keywords: contrastive learning
vision language model
simclr
clip
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: we improve contrastive loss by learning sample similarity graphs

Abstract: Learning good representations involves capturing the diverse ways in which data samples relate. Contrastive loss—an objective matching related samples—underlies methods from self-supervised to multimodal learning. Contrastive losses, however, can be viewed more broadly as modifying a similarity graph to indicate how samples should relate in the embedding space.  This view reveals a shortcoming in contrastive learning: the similarity graph is binary, as only one sample is the related positive sample. Crucially, similarities \textit{across} samples are ignored. 
Based on this observation, we revise the standard contrastive loss to explicitly encode how a sample relates to others. We experiment with this new objective, called $\mathbb{X}$-Sample Contrastive, to train vision models based on similarities in class or text caption descriptions.
Our study spans three scales: ImageNet-1k with 1 million, CC3M with 3 million, and CC12M with 12 million samples. The representations learned via our objective outperform both contrastive self-supervised and vision-language models trained on the same data across a range of tasks. When training on CC12M, we outperform CLIP by $0.6\%$ on both ImageNet and ImageNet Real. Our objective appears to work particularly well in lower-data regimes, with gains over CLIP of $17.2\%$ on ImageNet and $18.0\%$ on ImageNet Real when training with CC3M. Finally, our objective encourages the model to learn representations that separate objects from their attributes and backgrounds, with gains of $3.3$-$5.6$\% over CLIP on ImageNet9. The proposed method takes a step towards developing richer learning objectives for understanding sample relations in foundation models.

Published as a conference paper at ICLR 2025
X-SAMPLE CONTRASTIVE LOSS: IMPROVING CON- TRASTIVE LEARNING WITH SAMPLE SIMILARITY GRAPHS
Vlad Sobal
Mark Ibrahim'
Randall Balestriero 3 Vivien Cabannes'
Diane Bouchacourt Pietro Astolfi
Kyunghyun 2
Yann LeCun 'Genentech &CIFAR
Meta FAIR
NNew York Univeristy
Brown University
ABSTRACT
Learning good representations involves capturing the diverse ways in which data samples relate. Contrastive loss-an objective matching related undeless methods from self-supervised to multimodal learning. Contrastive losses, however, can be viewed more broadly as modifying similarity graph to indicate how samples should relate in the embedding space. This view reveals a shortcoming in contrastive learning: the similarity graph binary, as only one sample is the related positive sample. Crucially, similarities across samples are ignored. Based on this observation, we revise the standard contrastive loss to explicitly encode how a sample relates to others. We experiment with this new objective, called X- Sample Contrastive, to train vision models based on similarities in class or text caption descriptions. Our study spans three scales: ImageNet- Ik with 1 million, CC3M with 3 million, and CC12M with 12 million samples. The representations learned via our objective outperform both contrastive self-supervised and vision- language models trained on the same data across a range of tasks. When training on CC12M, we outperform CLIP by 0.6% on both ImageNet and ImageNet Real. Our objective appears to work particularly well in lower-data regimes, with gains over CLIP of 17.2% on ImageNet and 18.0% on ImageNet Real when training with CC3M. Finally, our objective encourages the model to learn representations that separate objects from their attributes and backgrounds, with gains of 3.3-5.6 6% over CLIP on ImageNer9. The proposed method takes a step towards developing richer learning objectives for understanding sample relations in foundation models.
INTRODUCTION
Contrastive loss underlies methods from self-supervised learning (SSL) to multimodal learning (Radford et al., 2021; Chen et al., 2020; Oord et al., 2018). In SSL, contrastive learning encourages the model to associate sample with another view of the sample created using hand-crafted data augmentation-tthe this related view is the positive sample. Other samples are then pushed away as negative, unrelated samples in the models' representation space. Contrastive losses also play a crucial role in multimodal models such CLIP (Radford et al., 2021), where the model associates an image with its text caption in representation space. Here contrastive learning designates the caption and image representations as positives while all other text-image pairs are designated negatives.
More broadly, contrastive losses can be seen as constructing a similarity graph to indicate how samples should relate in the model's representation space (Cabannes et al., 2023; Zhang et al., 2023; HaoChen et al., 2021; Wang et al., 2023; 2022). This view reveals shortcoming in contrastive learning: the similarity graph is binary, as only one sample is the related positive sample. Crucially, similarities across samples which contain precious signals about how aspects of one sample may relate to another, are ignored, thereby limiting the quality of the learned representations. For the example shown in fig. la, contrastive learning treats each image independently, without explicitly encoding similarities between the images depicting different pets. While prior work showed that data samples be implicitly linked by sutti-step connections in the augmentations graph when using contrastive learning (Wang et al., 2022; 2023), we explore how to capture such similarities explicitly by modifying the standard contrastive objective. We show that, in contrast to modeling

---

Published as a conference paper at ICLR 2025
Source of graph
dags
def training step imsges]: encode copt lons Frazen text encodar
enc(captions) detach() training model exb ing enc enclirages)
D_ing
X-CLR loss
pets
Similarity
el calculate simitarities
tgt. sims
0.8
taxtta figgsias
txt, orb.T tao s, dinel)
Images
finsas:
Zranny
0.4
4 the loss is The cress-antrop antropy between the Two return CE(tgt sims, ing sims, dim=l) reant)
emb.T tao, dim=1)
(a)
Figure 1: a) The diagram of X-CLR. X-CLR objective learns representations of images with the help of soft relationship graph. The graph can be built based on accompanying data, e.g. taxonomy for biological data. In our experiments, we use captioned images, and build similarities based on caption similarities. b) Python-style pseudo-code of X-CLR with similarity based on text captions.
those connections implicitly, explicitly incorporating similarities between visually different but semantically related samples makes representations more robust.
To account for similarities across samples, we first remove the binary negative vs. positive designa- tions in standard contrastive loss. We introduce instead a similarity graph with continuous scalars capturing the extent to which two samples are related. Consider the example in fig. 1, where the two dog images have a high similarity while the dog and cat images have a more moderate similarity, We build our objective by incorporating the soft similarity targets into the InfoNCE objective (Oord et al., 2018) in a manner similar to objectives used for distillation (Hinton et al., 2015; Wu et al., 2023). In contrast to distillation, we do not focus training model using outputs of another model, but instead focus on building a meaningful similarities graph and incorporating it into the objective. The proposed target similarity graph does have to come from another model, and can be inferred from any additional metadata. We experiment with this new objective, called X-Sample Contrastive Learning (X-CLR), by training vision models using graph similarities inferred from class or text caption descriptions found in common datasets. Our study spans three training dataset scales from 1 million samples with high-quality labels from ImageNet (Deng et al., 2009) to 3 and 12 million noisy image-text caption pairs from CC3M and CC12M (Sharma et al., 2018).
We find that compared to contrastive baseline methods trained on the same data, representations trained using X-CLR outperform contrastive training on range of tasks from standard classification to tasks involving the decomposition of objects from their attributes and backgrounds. When training on CC12M, we outperform CLIP by 0.6% on both ImageNet and ImageNet Real (Beyer et al., 2020). Furthermore, X-LLR yields representations that separate objects from their attributes and backgrounds well, with gains of 3.4-4.9% over CLIP ImageNet9 (Xiao et al., 2020). We also find for fine- grained disambiguation of object attributes, the quality of labels used to infer the similarity graph much more important than the data quantity. Compared to noisier web caption data, we find X- CLR trained on million higher quality class labels outperforms representations learned via standard contrastive CLIP trained 12x more data. Finally, we find X-CLR appears work particularly well in lower-data regimes, with gains over CLIP of 16.8% on ImageNet and 18. on ImageNet Real when training with CC3M. In short, we find representations learned using X-CLR generalize better, decompose objects from their attributes and backgrounds, and are more data-efficient.
Our contributions are:
1. We revisit the graph similarity perspective of contrastive losses, revealing that standard losses encode sparse similarity matrix that treats other possibly related samples as negatives;
2. We propose a new X-CLR loss that explicitly accounts for soft similarities across samples;
3. We experiment with this objective across three levels of data scale from 1-12 million samples, and find that the representations learned via X-CLR:
(a) Generalize better standard classification tasks with consistent gains over contrastive baselines trained on the same data. For example, when training on CC12M we outper- form CLIP by 0.6% both ImageNet and ImageNet Real.
2

---

Published as a conference paper at ICLR 2025
(b) Disambiguate aspects of images such attributes and backgrounds more reliably, with gains of 3.3-5.6% over CLIP on background robustness benchmarks for ImageNet.
(c) Finally, we find X-CLR learns more efficiently when data is scarce, with gains of 17.2% on ImageNet and 18.0% on ImageNet Real when pretraining on the smaller million sample CC3M dataset.
The proposed solution takes a step towards developing richer learning objectives for understanding sample relations in foundation models to encode richer, more generalizable representations.
RELATED WORK
Contrastive learning Various contrastive objectives have been proposed over the years (Chopra al., 2005; Schroff et al., 2015). More recently, the InfoNCE objective (Oord et al., 2018) has been the most popular choice for self-supervised methods, e.g. SimCLR (Chen et al., 2020) and MoCo (He et al., 2020). InfoNCE objective has also been successfully used to learn vision-language models using CLIP (Radford et al., 2021). The basis of those objectives is to make positive pairs have similar representations, while the negatives, which typically are just all other elements in a batch, should have a different representation. In its original form, InfoNCE is binary, meaning it only works with positive and negative pairs, and does not support degrees of similarity. The positive pairs are usually two augmentations of the same sample, which makes well-tuned augmentations crucial for good performance (Ryali et al., 2021). Dwibedi et al. (2021) estimate positives using nearest neighbors in the latent space instead and therefore can use weaker augmentations, while Caron et al. (2020) use cluster assignment. A few methods have proposed modifications wherein multiple positive pairs are supported, e.g., Khosla et al. (2020) groups positive by class labels, Hoffmann et al. (2022) propose using WordNet (Fellbaum, 1998) hierarchy to define ranked positive samples, and Tian et al. (2024) uses generative model to obtain multiple positives for the same concept. HaoChen et al. (2021) also look at contrastive learning through the lens of graphs, and propose a novel spectral objective. Wang al. (2023) draw connections between contrastive learning and message passing on the augmentation graph, while Wang et al. (2022) show that aggressive data-augmentations like cropping can connect samples of the same class. Zhang et al. (2023) show that contrastive learning objective implicitly learns the graph in which the samples are connected via augmentations in the case of SimCLR or via captions the case of CLIP. However, in that paradigm only visually similar samples or samples with a common caption get connected in the graph, while in our method the samples are connected based on provided graph, which can connect visually different yet semantically related samples.
Soft targets Using soft targets provides more learning signal to the model, possibly making it learn better and faster. This has been explored with distillation by Hinton et al. (2015). Soft targets have also been used with InfoNCE in the context of distillation in ReSSI. (Zheng et al., 2021) and SCE (Denize et al., 2023), where the target cross- tross-sample similarity comes from the teacher model. (Feng and Patras, 2023) use soft targets from deff-distillation to train an image encoder with coarse labels. Similarly, Fini et al. (2023a) compute soft targets via latent clustering and apply it to semi-superrvsse learning. Shen et al. (2023) use patch-mixing to train VIT image encoders to model inter-sample relationships. Andonian et al. (2022) proposes to use soft targets for CLIP (Radford et al., 2021) training, and calculates the targets via self-distillation. Wu et al. (2023) use similar objective to ours to distill the CLIP model into a smaller one. Further soft CLIP objectives are explored by Fini al. (2023b), who apply label smoothing to obtain soft targets, and Gao al. (2024), who estimate soft targets by comparing fine-grained image information. Finally, Huang et al. (2024) train CLIP with non-zero cross- sample similarities computed based on pre-trained uni-modal models for text and vision. In this study, we build on the work of Cabannes et al. (2023) who propose unifying framework to view SSL and supervised learning objectives as learning with different underlying similarity graphs. We take inspiration from the soft targets literature and propose using soft graph. Unlike distillation methods, where the soft targets typically come from a teacher model, the targets in X-CLR originate from any source, not necessarily another model. We try different similarity graph sources, including ones not based on the outputs of another model, and show that we can build better graph than ones commonly used in contrastive learning (see table 2).
3

---

Published as a conference paper at ICLR 2025
sample idx
sample
sample idx
Self-supervised
Supervised
X-CLR
Dog
0.0
0.2
0.4
0.6
0.8
1.0
Figure 2: Sample similarity adjaceney matrices of existing methods vs. our X-Sample Con- trastive similarity loss (right). We show pairwise similarities of 20 samples belonging to 4 classes. they are completely unrelated. In case of self-
Similarity of means the samples are identical, supervised learning, none of the inter-sample relationships are modelled (left). Supervised learning relies on the labels to group samples of the same class together (center). X-CLR models inter-class relationships by associating cats with dogs and pianos with guitars.
3 UNDERSTANDING CONTRASTIVE LOSSES VIA SIMILARITY GRAPHS
3.1 X-SAMPLE GRAPHS
Throughout this study, a similarity graph denotes graph in which the nodes represent data samples, and edges similarity relationships. Given the number of data samples in the dataset N, graph is expressed through its symmetric adjacency matrix G E IV- X the semantic relation between inputs i and being encoded in the real entry Gt.- fig. 2, we show graphs different learning paradigms. SSL does not rely on labels, but positive pairs/tuples/views generated at each epoch. Let us denote by the number of positive views generated, commonly V for positive pairs, and denote by E the training epochs. In that case, the original N input samples are transformed into N x E "augmented" samples
X (A)
T ...
repeated E times
where each T is random input transformation with its own random parameters. The corresponding graph given by:
G(sal) 1 ((WVE)=L/VE))
(1)
where the associated similarity graph captures two samples were generated as augmentations same original input. Such graphs G, defined by eq. (1), are the ones used as targets in common SSL methods, formalized below denoting Z & fo(X) NXD, Here, fo is the encoder, and D the encoding dimension.
Theorem ((Cabannes et al., 2023)). VICReg (Bardes et al., 2021), SimCLR (Chen et al., 2020), and BarlowTwins (Zhontar al., 2021) losses be expressed in terms of the graph G (1)
LVICE GZ; = Tz -G1
exp( [T. ž,)
LSimCIR (Z; - Σ
Gij log
ELE(N] exp(2, xx)
i.j([N)
LBT(Z;G) G2 -1
where & A x/ |||| and i the column normalized Z so that each column has unit norm.
In our study, we will focus on contrastive learning, i... SimCLR family losses. We will demonstrate how to away from the ad-hoe graph G from eq. (1).
4

---

Published as a conference paper at ICLR 2025
3.2 REVISITING CONTRASTIVE LOSSES WITH SIMILARITY GRAPHS: X-CLR
We introduce the soft cross-sample similarity to the widely used InfoNCE objective (Oord et al., 2018). We note that the proposed framework isn't necessarily limited to InfoNCE-based methods and can potentially be integrated into non-contrastive objectives such as BYOL, SimSiam, or VICReg (Grill et al., 2020; Chen and He, 2020; Bardes et al., 2021). although we leave the extensions to future work. In SimCLR (Chen et al., 2020), given a batch of No images, each image is augmented twice, so each sample has true positive. The 2N, images are then encoded representations 2. Then:
exp(sim((z,, 2,)/T)
PLJ Ek=1 1x exp(sim(22 2x)/T)
2N,
LSmBLL 2Nb
Σ H(L..p)
where H is the cross-entropy, and 1 is the one-hot distribution where all the probability mass is assigned to the index of the positive sample corresponding to i, and sim is the cosine similarity. Intuitively, we are training the model to classify positive examples in a batch, so the similarity p should be high only for the true positive. We introduce the soft objective by replacing the hard positive distribution 1y with a distribution 8;. Or, in terms of graphs, we replace the graph from the eq. (1) with a soft graph where connection strengths can be any number in [0, IJ. and, similarly, the distribution 8; and does not have to one-hot. Considering the example of fig. 1, we want photo of dog to have representation similar to that of another photo of dog, somewhat similar to the representation of a cat photo, and different from the representation of a photo of a mug. Given that distribution 8, we can plug in directly:
2Nd
LX 2N
Σ H(s.p:)
There are many possible ways to obtain this distribution 8. We could use the meta-data associated with the dataset. In our particular case, we utilize trained text encoder frext, and encode the text provided with each image to obtain a representation, which then used to calculate similarity between samples i and using the cosine similarity. Those pairwise similarities describe the soft graph:
((soft) (ime (c.), Frex((;;)) i.j
Where a is the caption associated with the i-th sample. Note that the similarities (saoft) do not have come from another model, and can be inferred based any additional information available, making this type of objective applicable to other datasets and modalities. The last step before plugging the similarities into the objective is converting them valid probability distribution using softmax:
exp(G (soft) /TO)) 1.j
81J
Ek=l exp( (
(loott) /T.)
Ty is a separate hyperparameter from 7 in the softmax to calculate the learned similarities. Higher values of Ts put more weight the 'soft' positives, while lower values in the limit recover the original SimCLR objective. For more details about this, see appendix A.9.
4 EXPERIMENTS
4.1
EXPERIMENTAL SETUP
We test X-CLR on three datasets of varying scale: ImageNet (Deng et al., 2009) (IM), and Conceptual Captions 3M and 12M (Sharma al., 2018). We blur faces in all datasets before training our models. We use the Sentence Transformer (Reimers and Gurevych, 2019) as the text encoder to construct similarities unless stated otherwise. For ImageNet experiments,
generate captions by using the template "a photo of a " to generate captions out of class names. our experiments with the conceptual captions dataset (Sharma et al., 2018),
the captions is. For experiments
5

---

Published as a conference paper at ICLR 2025
Table Accuracy of X-CLR and range of baselines on a of classification benchmarks. We freeze the encoder, and train linear layer on top to probe the quality of representations. We test on ImageNet classification, as well on set of robustness benchmarks.
Background Decomp.
MIT States
Soft loss
Uses labels
Method
ImageNet
ImageNet Real
Same Class
Mixed
ObjectNet
Objects
Attributes
SCE
X
71.3
78.7
61.7
58.4
20.2
44.5
31.0
ReSSL. (1 crop)
X
69.4
76.9
56.3
53.2
18.3
44.5
31.2
VICReg
72.4
79.0
60.8
56.8
20.5
43.5
26.9
Barlow Twins
X
X
72.8
80.0
62.7
59.4
21.6
45.9
31.7
SimCLR
X
X
63.4
67.8
44.7
38.9
12,1
40.9
29.1
SupCon
74.3
79.7
64.0
59.1
24.4
45.6
30.8
X-CLR
75.6
81.5
66.6
62.7
27.5
45.9
31.1
ImageNet, follow SupCon (Khosla et al., 2020) and use AutoAugment (Cubuk et al., 2018). All experiments on the ImageNet dataset were run for 100 epochs with 1024 batch size. The learning rate was set to 0,075 for ImageNet models. For experiments on CC3M and CC12M, we used the standard SimCLR augmentations, and a learning rate of 0.1. The rest of the settings were kept the same. Although SimCLR and SupCon both benefit from longer training, we haven't 't experimented with training for more epochs due to computational constraints. For more details, see appendix A.7.
In all our experiments, to isolate the effect of our learning objective, we the backbone architecture to be ResNet-50 (He et al., 2015) model as this is the most widely studied, with optimized hyperparameters, for standard contrastive self-supervised learning (Chen et al., 2020). We use the same architecture for CLIP's vision encoder and take advantage of already optimized publicly available checkpoints provided by OpenCLIP (Ilharco et al., 202 1) for CC12M. Since no comparable public checkpoint is available for CC3M, train our model, appendix A.6.
We evaluate all models across a suite of benchmarks to gauge how well representations generalize in terms of classification performance. We test ImageNet classification with standard as well as with ImageNet Real labels, on ImageNet-9 to test robustness to background change (we refer to this 'Background Decomposition" in results), on ObjectNet test robustness context and view change, and on MIT-States objects and attributes classification to test how well model captures object states. We use linear probing on top of frozen representations for all evaluations, including CLIP. For more details and dataset examples see appendix A.8.
4.2 X-SAMPLE CONTRASTIVE WITH WELL-LABELER SAMPLES
We first experiment with X X-Sample Contrastive using the well-labeled ImageNet dataset to understand the effect of incorporating similarities across samples in the training objective. We compare X- Sample Contrastive (X-CLR) to a range of baselines: SupCon (Khosla et al., 2020) which uses labels; SCE (Denize et al., 2023) and ReSSL (Zheng al., 2021) which use self-distillation with soft targets; and to SimCLR (Chen et al., 2020), VICReg (Bardes et al., 2021) and Barlow Twins (Zbontar et al., 2021), which are SSL methods for learning from images.
We find in table representations learned via X-CLR with Sentence Transformer similarities (Reimers and Gurevych, 2019) improve on standard classification performance, with gains of 12.2% relative to SimCLR and 1.3% relative to Supervised Contrastive on ImageNet. We find similar gains when evaluated on revised labels from ImageNet Real of 13.7% and 1.8%, respectively. Finally, we find by capturing similarities across samples, representations learned via X-CLR are more capable of disambiguating objects from backgrounds and attributes with gains on ImageNet-9 (Xiao al., 2020) and ObjectNet (Barbu et al., 2019). To confirm statistical significance of the results, we report standard deviations over seeds for the models we trained in table (SimCLR, SupCon, and X-CLR)). For the remaining ImageNet models, took published pre-trained encoders, and are limited to one seed.
Effect of the similarity graph. We investigate how the choice of similarity graph affects the performance. To calculate the similarity between samples, use similarity of the captions embedding when using various text encoders: CLIP (Radford et al., 2021), LLama2 (Touvron et al., 2023), Sentence Transformer (Reimers and Gurevych, 2019). We also experiment with the similarity defined by the distance in WordNet hierarchy (Fellbaum, 1998). Baselines include SimCLR Chen et al. (2020), where the similarity graph is the binary augmentation graph, and Supervised Contrastive (Khosla

---

Published as a conference paper at ICLR 2025
Table 2: The effect of the similarity source on the model performance. We train X-CLR with similarities coming from a variety of sources, and evaluate the representations with linear probing.
Background Decomposition
Similarity source
ImageNet
ImageNet Real
Same Class
Mixed
ObjectNet
Augmentation graph (SimCLR)
63.4
67.8
44.7
38.9
12.1
True class graph (SupCon) Sentence Transformer (X-CLR)
74.3
79.7
64.0
59.1
24.4
75.6
81.5
66.6
62.7
27.5
CLIP text encoder
74.4
80.6
67.5
64.2
24.5
L.amma2 text encoder Distance in WordNet hierarchy
40.9
45.8
38.3
36.0
4.3
68.3
74.9
55.7
52.1
21.2
Imagenst Accuracy
60
Imagehet Accuracy
70-
60-
X-CLR SimCLR SupCon
Accurary
40-
60
40
20-
X-CLR
55
20
SimCLR
SupCon
0
200
400
600
800
1000
20
40
60
80
10-3
10a
104
Samples per class
Neerestt Neighbars value of
Temperature
(a)
(b)
(c)
Figure 3: (a) X-Sample Contrastive learning data efficient with ImageNet pretraining, We outperform SimCLR in low data regimes and match Supervised Contrastive trained on ground truth labels at varying levels of data searcity.
(b) KNN performance ImageNet. X-CLR outperforms other methods with KNN probing for range of values of K. (c) Sensitivity of X-Sample Contrastive to temperature. We benchmark our method trained with different values of temperature т, on ImageNet.
et al. 2020), where the similarity graph has connections only between samples of the same class. We emphasize that X-CLR is not model distillation method (Wu et al., 2023) as the similarities can come from any source, including any metadata, c.g., WordNet hierarchy in our case or taxonomy of organisms for biological data. For more details regarding this experiment, appendix A.I. We show results table 2. Overall, we find that using Sentence Transformer the best option for classification performance, and use this similarity in all the following experiments.
Can we improve contrastive learning under data scarcity? To answer this question, we train all three models SimCLR, SupCon, and X-CLR by varying the number of samples seen for each class ImageNet. We find X-CLR, by incorporating class labels and how they relate, is able to learn representations with comparable performance SupCon trained with ground truth class labels and outperform SimCLR even when few training samples are available per class as shown in fig. 3a.
4.3 X-SAMPLE CONTRASTIVE LEARNING WITH NOISY MULTIMODAL SAMPLES
Contrastive loss also plays pivotal role in multimodal vasion-language models such as CLIP. The contrastive training objective matches noisy paption-image pairs. Here we experiment with X-Sample Contrastive loss by using the noisy captions to learn similarities across samples. We compare both SimCLR as a standard contrastive model and CLIP trained on the same caption-image data across two levels of scale: 3 and 12 million samples from CC3M and CC12M.
We find incorporating X-CLR loss leads to representations with higher classification accuracy and disambiguation of objects from their attributes and backgrounds. With CC12M training shown in table 3, X--ample Contrastive learning outperforms SimCLR by 0.5% and CLIP by 0.6% with CC12M with similar gains for ImageNet Real. We also find X-CLR training better disambiguate object foreground from backgrounds, with gains of 0.6-1.5% over SimCLR and 3.3-5.6% over CLIP.
We find learning similarites across samples with X-CLR leads to more considerable gains when less data available. When trained on CC3M, X-CLR outperforms SimCLR by 1.2% and CLIP by 17.2% IetgeNet, with similar gains ImageNet Real shown in table 3. We find X-CLR training can
7

---

Published as a conference paper at ICLR 2025
Table 3: X-Sample Contrastive learning performance with CC3M and CC12M training. We train X-CLR on conceptual caption images with text caption similarities, and compare to CLIP and SimCLR. SimCLR does use text data and is only trained on images.
Background Decomposition
Data
Method
ImageNct
ImageNet Real
Same Class
Mixed
ObjectNet
SimCLR
57.0
64.0
24.4
18.9
10.8
CC3M
CLIP
41.0
47.6
12.5
10.6
7.8
X-CLR
58.2
65.6
26.7
20.3
11.5
SimCLR
58.9
66
24.6
19.8
12.7
CC12M
CLIP
58.8
66.1
20.5
17.1
11.9
X-CLR
59.4
66.7
26.1
20.4
13.4
Table 4: X-CLR performance when used to finetune pretrained models. We take a SimCLR checkpoint and fine-tune it using X-CLR objective for 10 epochs. We then evaluate the original checkpoint and the fine-tuned one using linear probing.
Background Decomposition
ImageNet
ImageNet Real
Same Class
Mixed
ObjectNet
SimCLR
63.4
67.8
44.7
38.9 50.0
12.1
X-CLR finetuning
66.5
74.4
53.9
17.4
more considerably disambiguate object foregrounds from backgrounds compared to CLIP when less training data is available, with gains of 10.3-14.2% over CLIP.
4.4 X-SAMPLE CONTRASTIVE CAN BE USED TO FINETUNE PRETRAINED BACKBONES
We validate whether X-CLR can be used as a finetuning objective for pretrained backbones, given the growing abundance of publicly available model checkpoints. Here, we evaluate pretrained SimCLR model by finetuning for 10 epochs on ImageNet with X-CLR instead of the original SimCLR contrastive objective. We see in table finetuning with X-CLR improves classification performance on ImageNet by 3.1% % and on ImageNet Real by 6.6%. Furthermore, we see by relating samples during the finetuning stage, X-CLR can disambiguate object foregrounds from backgrounds with grains of 9.2-11 1.1 % on ImageNet-9 as well as improvements natural object transformations from ObjectNet with a gain of 5.3% after finetuning.
4.5 X-SAMPLE CONTRASTIVE OBJECTIVE INTRODUCES ONLY MINIMAL COMPUTATIONAL OVERHEAD
Both for ImageNet and conceptual captions datasets, we don't 't run the text encoder for each sample we see, and instead precompute the similarity values. For more details, see appendix A.7. Avoiding running the text encoder during model training avoids the extra overhead at the price of some pre- processing. Pre- Pre-processing takes less than 2 hours for CC12M when using one GPU, about 30 minutes for CC3M, and less than 5 minutes for ImageNet. To further analyze how much overhead there is, we compare the average time it takes to process one batch for SimCLR and X-CLR. The results shown in table 5. Overall, time it takes to train models with the X-CLR objective compared to the regular contrastive objective. To train on ImageNet, we used Nvidia V V100s, and each run took about 30 hours. With the same setup, CC3M runs took about 50 hours, and CC12M runs took roughly 9 days.
didn't notice any significant difference in the amount of
5 ANALYZING REPRESENTATIONS LEARNED WITH X-CLR
KNN Clustering. To confirm the representations leamned via X-CLR also work well for downstream tasks with non-linear decision boundaries, we perform evaluation using the common K-nearest neighbor (KNN) protocol. The results shown in fig. 3b demonstrate X-CLR outperforms both
8

---

Published as a conference paper at ICLR 2025
Table 5: Analyzing the computation overhead of the X-Sample Contrastive objective during training. We measure the mean and standard deviation of the time needed to process one batch update for X-CLR and SimCLR. X-CLR introduces nearly no computational overhead.
Method
Seconds per batch ImageNet
Seconds per batch CC
SimCLR
0.866 = 0.008
0.874 0.034
X-CLR
0.866 + 0.010
0.877 0.032
SupCon target simliarities
X-CLR target similarities
XCRLR Leamed simliarities
vateo da
Mullese dop
Maline dop
Maline dog
Daserron-
Basermoe
Booketb
come bull
acouric
scousic
goodd
good
grand
Sampfere 10 sic
secpphore
Secaprone
Felines
Dogs
Types of balls
Musical instruments
Figure 4: Visualizing pairwise similarities SupCon (Khosla al., 2020) objective does not encour- age non-zero similarity between samples of different classes (left), while X-CLR target similarities take into account semantic closeness within categories such as dogs or types of balls (center). On the right, we see that the trained model successfully learns the soft similarity. For more graphs, see fig. 6.
SimCLR and SupCon baselines across range of choices for K. We also show KNN results for models trained on conceptual captions in appendix A.4.
Analyzing the learned graph from X-Sample Contrastive representations. Here we examine whether the learned representations from X-Sample Contrastive capture semantically meaningful similarities. To do so, we select four groups of three ImageNet classes: felines. dogs, types of balls, and musical instruments. For each pair of classes, we then compare the representation similarities using cosine similarity. A higher average pairwise similarity indicates the model's latent representations encode the classes similarly. In fig. 4 we show the graph of similarities learned after training with X-CLR on ImageNet. We find that the image encoder successfully captures the similarity within the class groups Additionally, we repeat the analysis of the learned similarity graph in (Zhang et al., 2023). In that work, the authors also view contrastive learning through the lens of graphs, and introduce metries to analyze the quality of the learned graph. In appendix A.3, table 9, we find that X- CLR learns a better structured graph compared to CLIP, SimCLR, and SupCon, with samples better connected within the class, and less connected for unrelated classes. We also plot T-SNE ( (Van der Maaten and Hinton, 2008) projections of X-CLR, SimCLR, and SupCon representations trained on ImageNet in fig. 13 and find that X-CLR learns well-structured embeddings.
The effect of softmax temperature, and inferred similarity graph. We show the sensitivity of X-CLR to temperature т, in fig. 3c on ImageNet. In the limit, when temperature goes 0, we recover Supervised Contrastive method for ImageNet, or SimCLR in case of conceptual captions, see appendix A.9 for a more detailed explanation. With low temperature, the similarity is only if the captions are exactly the same. As the temperature increases, more weight is put on the soft positives compared to the true positives (i.e. augmentations of the same sample). With high temperature, our method is unstable as too much emphasis is put on the soft positive examples compared to the true positives. We find that the value of 0 strikes the optimal balance and provides an improvement over pure Supervised Contrastive objective, while still emphasizing true positives enough. We show how Tr changes the objective in fig. 5b. We see that more probability mass is put on the true positvie for low temperatures. with ImageNet In fig. 5a, we see that on average, similarity between samples in ImageNet is much higher than in CC3M, which may be caused by inadequate caption quality in CC3M.
average having lower true positive weight than CC3M samples.

---

Published as a conference paper at ICLR 2025
Positive Probabiiity Batch
147
CC3M similarities
1e6 ImageNet similarities
3.0
1.0
2.0
0.8
2.6
0.6
positiv
0.4
1.0
CT38 sar1
0.2
14
125
256
512
1024 2003 4095 0192
0.0
0.2
0.4
0.6
0.8
1,0
0.0
0.2
0.4
0.6
0.8
1.0
Batch 5i20
(a)
(b)
Figure 5: (a) Histograms of the similarities calculated using Sentence Transformer on ImageNet and CC3M. While for ImageNet the average similarity is around 0.35, it is much lower on CC3M, signifying that the graph contains less information for CC3M. (b) Effect of the temperature and batch size on the weight assigned to the true positvie.
Table 6: The effect of label quality matters for fine-grained attribute disambiguation. We evaluate X-CLR trained on high-quality ImageNet labels and on noisier CC labels, and find that smaller but better quality ImageNet dataset yields better representations for attribute disambiguation on MIT states. For more details on the benchmark, see A.8.3.
Pretraining
Data Size
Quality
MIT States Attributes
MIT States Objects
CLIP CC3M
3M
Noisy
27.0
40.1
CLIP CC12M
12M
Noisy
23.3
36.9
X-CLR CC3M
3M
Noisy
29.5
40.7
X-CLR CC12M
12M
Noisy
30.1
42.1
X- CLR ImageNet
1M
High
30.9
45,8
The impact of label quality for fine-grained attribute disambiguation We show in table 6 how label quality can impact downstream performance on finer-grained attribute disambiguation, We find labels from noisy captions degrade performance for fine-gres fine-griined object attributes in MIT States (Isola et al., 2015) for both X-CLR and CLIP. We find X X-CLR with high quality labels from ImageNet, can outperform models trained much larger noisier data. Compared to CLIP trained on 12x more data, X-CLR achieves 30.9% vs. 23.3% for CLIP on attribute classification and 45.8% vs. 36.9% for CLIP on object classification MIT States benchmark (see appendix A.7 for more details).
DISCUSSION
We revisited the graph view the commonly used contrastive learning methods and developed better learning objective, X-CLR, by integrating a soft similarity graph. The adjacency matrix of the proposed graph contains not just and 1, but also any values between, capturing the degree of similarity across samples. We experiment with different ways of constructing the graph, and find that indeed we can build soft graph that improves over the existing binary graph of contrastive methods. However, we believe that there are better ways of constructing the graph than what we found, particularly for the conceptual captions dataset where the captions are quite noisy. A better graph can possibly be built using other metadata, such as location or time. We also believe that ideas from X-CLR can used with other modalities where extra metadata available, e.g. biological data with the associated taxonomy. The soft graph can also be used to enhance non-contrastive objectives such as BYOL (Grill et al., 2020) or VICReg (Bardes et al., 2021).
The main limitation of the present work is that constructing the cross-sample similarity graph requires extra data, as well as some extra memory to store it. When the extra data is not available, the only options remaining to build the graph using the augmentations, lelf-distillation, or other pre-trained models. The resulting method is also highly dependent on the quality of the graph, we have seen with conceptual captions datasets.
Limitations.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENTS
This material is based upon work supported by the National Science Foundation under NSF Award 1922658.

---

Published as a conference paper at ICLR 2025
REFERENCES
Alex Andonian, Shixing Chen, and Raffay Hamid. Robust cross-modal representation learning with progressive self-distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16430-16441, 2022.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreend, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Adrien Bardes, Jean Ponce, and Yann LeCun. Viereg: Varianc-imarrannc-cvvaannee regularization for self-super learning. arXiv preprint 105iv:2105.04906 2021.
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07159,
2020. 2020.
Steven Bird, Edward Loper, and Ewan Klein. Natural Language Processing with Python. O'Reilly Media Inc., 2009.
Vivien Cabannes, Leon Bottou, Yann Lecun, and Randall Balestriero. Active self-supervised learning: A few low-cost relationships are all you need. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16274-16283, 2023.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912 12-9944, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning. pages 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. in 2021 ieee. In CVF conference computer vision and pattern recognition (CVPR), pages 15745-15753, 2020.
Chopra, R. Hadsell, and Y. LeCun. Learnnng similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR volume 1, pages 339-546 vol. 1, 2005. doi: 10.1 10.1109/CVPP 200.0/0.....0..2222
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment:
Learning augmentation policies from data. arXiv preprint 00555 2018
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009. doi: 10. 20WCCV....0002000600
Julien Denize, Jaonary Rabarisoa, Astrid Orcesi, Romain Hérault, and Stéphane Canu. Similarity contrastive estimation for self-supervised soft contrastive learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2706-2716. 2023.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighhor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9588-9597, 2021.
Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019. URL https::/tttttt thub. com /ooc/
Christiane Fellbaum. WordNet: An Electronic Lexical Database. Bradford Books, 1998. URL
https: /tmm/tttpsssssssemmmmmmmemssesss mit edd.edduu//// add./9700222661655
Chen Feng and Ioannis Patras. Maskcon: Masked contrastive learning for labrlel dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19913-19922, 2023.
12

---

Published as a conference paper at ICLR 2025
Enrico Fini, Pietro Astolfi, Karteek Alahari, Xavier Hameda-Pinedaa Julien Mairal, Moin Nabi, and Elisa Ricci. Semi-supervised learning made simple with self-supervised clustering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3187-3197, 2023a.
Enrico Fini, Pietro Astolfi, Adriana
Jakob Verbeek, and Michal Drozdzal. Improved baselines for vision- saiiunlanguage pre pre-training. Transactions on Machine Learning Research, 2023b. ISSN 2835-8856. URL https: / openreview net/ forum? ? d= NmvVx Featured Certification.
Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu, Enwei Zhang, Ke Li, Jie Yang, Wei Liu, and Xing Sun. Softelip: Softer cross-modal alignment makes clip stronger. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1860-1868, 2024.
Jean-Bastinn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21283 2020.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34:5000-5011, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image (2015). 2015.
recognition. corr
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv: wrXiv:1503.02511, 2015.
David T Hoffmann, Nadine Behrmann, Juergen Gall, Thomas Brox, and Mehdi Noroozi. Ranking info noise contrastive estimation: Boosting contrastive learning via ranked positives. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 897-905, 2022.
Hailang Huang, Zhijie Nie, Ziqiao Wang, and Ziyu Shang. Cross-modal and uni-modal soft-label alignment for image-text retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18298-18306, 2024.
Gabriel Ilharco. Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL https: http://oo.ogggg/// 5281/ zenodo. 5143773. If you use this software, please cite it as below.
Phillip Isola, Joseph Lim, and Edward H Adelson. Discovering states and transformations image collections. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1383-1391, 2015.
Prannay Khosla. Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances neural information processing systems, 33:18666 8661-18673, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint /80.00377 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 1908. 10084, 2019.
13

---

Published as a conference paper at ICLR 2025
Chaitanya K Ryali, David Schwab, and Ari S Morcos. Characterizing and improving the robustness of supe-vuue learning through background augmentations. arXiv preprint wrXh:22033.1211, 2021.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2015.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556 2566,2565, 2018
Chengchao Shen, Dawei Liu, Hao Tang, Zhe Qu, and Jianxin Wang. Inter-instance similarity modeling for contrastive learning, arXiv preprint arXiv:2306. 12243, 2023.
Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. Advances Neural Information Processing Systems, 36, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. arXiv preprint arXiv:2203. 13457, 2022.
Yifei Wang, Qi Zhang, Tianqi Du, Jiansheng Yang. Zhouchen Lin, and Yisen Wang. A message passing perspective on learning dynamics of contrastive learning. arXiv preprint 0434:3230.00435. 2023.
Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengehen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi Stephen Chen, Xinggang Wang. et al. Tinyclip: Clip distillation via affinity mimicking and weight inheritance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21970-21980, 2023.
Zhibiao Wu and Martha Palmer. Verb semantics and lexical selection. In Proceedings of the 32nd annual meeting of the Associations for Computational Linguistics, pages 133-138, 1994.
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.0m 09994, 2020.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 7008.00000000 03888, 2017.
Jure Zhontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. International Conference on Machine Learning, 2021.
Qi Zhang, Yifei Wang, and Yisen Wang. On the generalization of multi-modal contrastive learning. In International Conference on Machine Learning, pages 41677-41693. PMLR, 2023.
Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Ressl: Relational self- superviee learning with weak augmentation. Advances in Neural Information Processing Systems, 34:2543-22 2555, 2021.
14

---

Published as a conference paper at ICLR 2025
Table 7: Analyzing statistical significance of ImageNet results. Each experiment is ran with seeds, we report the mean and standard deviation.
Background Decomposition
MIT States
Method
TmtgeNet
ImageNet Real
Same Class
Mixed
ObjectNet
Objects
Attributes
SimCLR
63.43 = 0.12
67.75 # 0.27
38.88 0.43
44.67 = 0.60
12.07 0.33
40.92 0.26
29.08 = 0.17
SupCon
74.30 0.16
79.66 # 0.12
59.08 0.44
64.00 =
24,42
45.56
30.83
X-CLR
7 0.09
0.133
62.74 0.27
66.59 0.25
2.15
0.155
0.110
A APPENDIX / SUPPLEMENTAL MATERIAL
A.1 MORE LEARNED SIMILARITIES COMPARISONS
We experiment with building the graph in the following ways:
Graph with connections only between samples of the same class (SupCon);
Graph with connections only between augmentations the same image (SimCLR);
Graph where soft similarity is inferred by comparing representations of the sample captions. The representations are computed using the sentence transformer (Reimers and Gurevych, 2019), CLIP text encoder (Radford et al., 2021), LLama2 encoder (Touvron al., 2023). For LLama2, we averaged the output tokens;
Graph where the connection strength is defined by the distance in WordNet (Fellbaum, 1998) hierarchy. We used the NTLK library Bird et al. (2009), and used the Wu-Palmer similarity (Wu and Palmer, 1994) between the class synsets;
We also experiment with random graphs where the cross-sample connections' strengths are fully random. The connections are either random per sample pair, or random per class pair (akin to using random text encoder for captions).
The results are shown in table 8. We find that overall, the Sentence Transformer graph performs the best, although the CLIP text encoder achieves good performance as well. Interestingly, we find that using WordNet hierarchy distance did not work well. Additionally, random per class pair performs quite well, particularly on background decomposition. We hypothesize that it's due to regularizing effect of the random similarities. Nevertheless, the performance of ImageNet, ImageNet Real and Object Net is lower than when using Sentence Transformer. We visualize learned and target similarities for SupCon graph and for the graph built using CLIP text encoder in fig. 6.
Visualising similarities In fig. 4, to visualize learned similarities, for each class we pick 100 examples from the dataset and encode them. Then, to calculate the average learned similarity between two classes, we take the 100 examples for each of the two classes, and calculate the Cartesian product, yielding 10,000 similarities. We take the mean over those 10,000 similarities to represent the average learned similarity for a class pair.
Similarities when training on CC datasets In appendix A.2, we show the similarities learned by X-CLR on CC3M and CC12M datasets.
A.2 ANALYZING STATISTICAL SIGNIFICANCE OF THE RESULTS
To make sure the difference in performance observe is statistically significant, run X-CLR. SimCLR, and SupCon pretraining with different seeds. We report the results of the evaluations in table 7.
A.3 ANALYZING THE LEARNED GRAPH
We follow the analysis of Zhang al. (2023) and show results in table 9. The analysis studies two values: label error which measures how similar samples different classes are on average, and intra- class connectivity, which measures the similarity of the samples within the class relative to those
15

---

Published as a conference paper at ICLR 2025
Target similarities
Learned similarities
Grer an shappeen
couutii qutar
Doberman
Eorni bal
Mateese dag
Daberman
baskeball
ball
gaid plano
cougar
tyxx
Ma
Gbbby
cougar
cougar
tabby
800
Labby
Maltez dog
Maltese dog
German shephard
800
German shephard
Dobermnn
Dobermon
basicttal
taskettalll
Velleybal
volleyenl
ternis ball
tennis tall
acoustcc guilerr 1.8 660 230
accustic galler
022
grand plano
600 sal.
grand piano + *.
013
saxophone 160 am
6:e ca
saxophone
Felines
Dogs
Types of balls
Musical instruments
(a) SupCon target and learned similarities
Target similarities
Learned similarities
Gar an shsphand
acoustic gutar
Garman Moibere
Datbeman
basketal
volkyybal
tonels ball
Maitete dog
Dobherme
baokatal
sellyyball
tenets ball
grand plano
Gouger
Dibby
trix
cougare
cougan
Tynx
Jynte
tabby
Maltese
Maltese
German shepherd 433
German shephers
Doberman
Deberma
hacostbal 1.2 230
bescetbal 0.0
velleytal
volleybal
ternis ball
lameia ball
piano
saxophone 433
saxophone 0-30
Felines
Dogs
Types of balls
Musical instruments
(b) CLIP target and learned similarities
Figure 6: Target and learned similarities for different graphs.
German shepherd
German shepherd
acousdle qutaar
acousd quitar
Mathese dog
mon
boskettalll
vollepball
sonvia ball
grand plano saasppome
Mairese dog
Oobeman
coall
plano
Synx
labbtr
tabby
sunvia
cougar
lyиe
174
331
575 8:0 020 435 037 631
lynz
8MM 219 039
Iabby
100
0.4 340
800 8.0 824 4.5 029 6A4
tabby
00
8.00
8.00 K25 435
038
Nakese dog
034004000 040 034
0.0 940
8000 8:00 9.2 6.5 635
Naltose dog
0.2
8.09
Gurman shepherd
6:45 640 044 940 060
8:01
9.20 6.6
Gemman shapherd
044 045 047
Dobweman
642 032 948 943
Doberman
5.4) 036 043 900
basketbal
639 685 652 056 908 807
balk
5.04 022
068
007 A12
volleytall
(38 682 621 022 001 808
volleytall 0.06 024 023 008 012
tennis ball
(228 029 024 022 935 618 920 114
tennis 0:20 039 029 023 838
53 659 638 638 001 011 516
gultar c.6 029 028 033 004 808
grand
133 627 628 636 106
91A
grand plano- 0.0 619 638
saxophone- 13 611 n54 o56 006 9m0
saxophone cm 027 023 018
(a) CC3M similarities
(b) CC12M similarities
Figure 7: X-CLR Learned similarities when trained on a) CC3M and b) CC12M.
16

---

Published as a conference paper at ICLR 2025
Table 8: The effect of the similarity source on the model performance.
Background Decomposition
Similarity source
ImageNet
ImageNet Real
Same Class
Mixed
ObjectNet
Augmentation graph (SimCLR)
63.4
67.8
44.7
38.9
12.1
True class graph (SupCon)
74.3
79.7
64.0
59.1
24.4
Sentence Transformer (X-CLR)
75.6
81.5
66.6
62.7
27.5
CLIPP text encoder
74.4
80.6
67.5
64.2
24.5
I.I amma2 text encoder
40.9
45.8
38.3
36.0
4.3
Random per class pair
74.5
80.8
71.0
68.0
26.6
Random per sample pair
0.1
0.1
0
Distance in WordNet hierarchy
68.3
74.9
55.7
52.1
21.2
Table Analyzing the learned representations' connectivity
Metric
CLIP
SimCLR
SupCon
X-CLR
Label error (L)
0.550
0.250
0.250
0.223
Intra-class connectivity (1)
1.233
1.700
2.005
2.193
from different classes. This allows us to determine how well the learned graph captures the class relationships in the data. Since the open-source repository of that paper did not contain the code for analysis, we re-implemented it to the best of our ability.
Here, X-CLR, SimCLR, and SupCon are trained on ImageNet, while CLIP is trained on CC12M. According to these metries, X-CLR representation is the best among the baselines, meaning that learn a better structured graph. We note that our SimCLR numbers original paper. We suspect that it's due to the fact that the authors train SimCLR with the batch size of 512, while at 63.4, compared to 61.2.
much better than in the
2048. ImageNet classification performance of our SimCLR model is also higher,
We also note that label error, which is the measure of average similarity between instances of different classes, is lower for our method, although the loss itself encourages it to be higher for related samples. This is due the fact that in this analysis, original procedure), and those classes are not related to each other.
the first 10 classes from ImageNet (replicating the
A.4 KNN EVALUATION
Apart from testing the models trained on ImageNet using KNN, we also evaluate the models trained on CC3M and CC12M. The results are shown in fig. 8. We see that X-CLR performs better on CC3M, and comparatively with SimCLR when trained on CC12M.
A.5 IMAGENET-9 DETAILS
A.6 CLIP DETAILS
In CC3M experiments, we train the model from scratch, as OpenCLIP didn't have checkpoint trained on that dataset. We trained both for 32 and 100 epochs, and found that the model trained for 100 epochs performs better. Since 32 epochs is the default CLIP number of epochs, we also report results for 32 epochs. The results are shown in table 10.
A.7 MORE TRAINING DETAILS
We train SimCLR, SupCon and X-CLR using the LARS optimizer (You et al., 2017). In all cases, we the same R5sNet-50, with two layer projector on top. The output dimension of the projector 128. We note that SupCon benefits from bigger batch sizes and longer training as was shown in (Khosla al., 2020), but in experiments we did not explore training for more than 100 epochs
17

---

Published as a conference paper at ICLR 2025
Trained on ImageNet
Trained on CC3M
Trained on CC12M
ImageNet Accuracy
70
40
41
65-
X-CLR SimCLR
39
40
60
SupCon
38
55
X-CLR SimCLR
X-CLR SimCLR
50
38
20
40
60
80 20
40
60
80
20
40
60
80
Nearest Neighbors value K
Nearest Neighbors value K
Nearest Neighbors value K
Figure 8: Results of models trained on ImageNet, CC3M, CC12M on ImageNet validation when using KNN classifier.
Table 10: CLIP on CC3M We train our own models on CC3M and find that training longer improves the performance. Nevertheless, CLIP struggles with small datasets.
Background Decomposition
Method
ImageNet
ImageNet Real
Same Class
Mixed
ObjectNet
CLIP 100 epochs
41.0
47.6
12.5
10.6
7.8
CLIP 32 epochs
36.8
42.0
11.5
9.8
6.0
with bigger batch sizes. For fair comparison, keep the number of epochs to 100 and batch size to 1024 across all methods we train on ImageNet.
Fetching similarities For ImageNet, since the number of classes is known, we pre-compute the similarity matrix of dimension 1000 1000, and retrieve elements from depending on the associated class labels for given sample pair to obtain the similarity value. For conceptual captions, we run the text encoder on the full dataset and save the encodings to disk. Then, when loading an image from disk, we also load the associated encoding of the corresponding caption. The similarity matrix for given batch is then obtained by calculating the Cartesian product of those encodings.
In order to evaluate on this dataset using linear probing, we split the dataset randomly into even parts, one used for training the linear layer, the other for evaluation. We train separately to classify objects and attributes.
MIT States
A.8 EVALUATION DETAILS
All evaluations done with linear probing. We use the learning rate of and plain SGD optimizer for all evaluations. We do not use zero-shot classification for CLIP, or any other models. Since X- CLR doesn't 't align the representations evaluate. For ImageNet models we train ourselves, we do not train the linear prober separately, we train it along with the rest of the model with detached gradients.
text and images, we cannot utilize zero-shot classification
A.8.1 IMAGENET REAL
ImageNet Real evaluation uses improved ImageNet labels from (Beyer et al., 2020). The improved evaluation allows for multiple labels for scenes where labels are ambiguous, and corrects some mistakes in annotations.
A.8.2 BACKGROUND DECOMPOSITION WITH IMAGENET-9
ImageNet-9 (Xiao et al., 2020) proposes multiple benchmarks to test model robustness to the background perturbation. The benchmark is created by taking samples from ImageNet, segmenting
18

---

Published as a conference paper at ICLR 2025
(a) Mixed
(b) Same class
Figure 9: The two settings for ImageNet-9 (Xiao al., 2020) benchmark. In mixed setting in a). the background is swapped with that of sample belonging to another class. In a), this results in an image of butterfly on underwater background. Performing well in this setting is hard the model learns spurious correlations with the background. In same class setting in b), the background swapped with that of sample belonging to the same class. In b), we see butterfly pasted onto the background of another butterfly sample, in this case tree branch green background. This lets us see how much swapping the background affects the performance and serves as a baseline for 'mixed'.
pureed tomato
fresh tomato
peeled tomato
diced tomato
pureed tomato
typical flower
fresh flower
fresh flower
crumpled flower
crinkled flower
Figure 10: MIT States (Isola et al., 2015) samples from two classes: tomato and flower. Each sample has associated class name and attribute. We show tomatoes and flowers classes with different attributes. For example, the attributes we show for tomato are "pureed", "fresh", "peeled", and "diced". We train models classify class names and attributes. When predicting attributes, do not condition on class names and vice versa.
the object in the scene, and swapping out the background. Since the benchmark uses the same classes as ImageNet, we not retrain the ImageNet classifier. We consider two setups for evaluation:
where the object is seen with backgrounds from the same class, which we label "same class";
where the objects are seen with backgrounds from other classes, which we label "mixed",
These evaluations isolate how well the model can classify an object without relying on spurious associations from the background. We show example in fig. 9.
A.8.3 MIT STATES
Here we measure object classification accuracy as the object attributes vary as well as the models' ability to identify the object attributes such color, texture, shape. We show few examples from the dataset in fig. 10. Because this benchmark uses different set of classes from ImageNet, we need train classifier separately.
19

---

Published as a conference paper at ICLR 2025
ImageNet
ObjectNet
Chairs by rotation
Chairs by background
Chairs by viewpoint
Chairs
Teapots
T-shirts
Examples from ObjectNet dataset. We take the figure from Barbu et al. (2019). As opposed ImageNet samples, ObjectNet shows objects in a unexpected contexts and positions. For example, a chair lying on the bathroom floor or a teapot on its side near bathroom sink. This benchmark tests robustness of the models to these unexpected context changes.
Figure 11:
A.8.4 OBJECTNET
ObjectNeet contains real-world objects with varying poses, backgrounds, and viewpoints. The benchmark uses the same classes as ImageNet, and measures how robust the model is to seeing the objects unexpected contexts. Like for ImageNet-9, since the benchmark uses the same classes as ImageNet, we do not retrain the ImageNet classifier. We show examples in fig. 11.
A.9 CONNECTION BETWEEN SUPERVISED CONTRASTIVE LEARNING AND X-CLR
Here, we will outline how as the temperature тa approaches 0, X-CLR becomes SupCon. Supervised Contrastive Learning (Khosla et al., 2020) also uses image augmentations, and augments each image twice, to obtain what they call "a multiviewed batch". Then, in equation 2, they propose the loss:
ssup out
out
log Pt. Ptp
EET
ie!
PEP())
where Pi.j defined as follows:
exp(sim(z;, 2j)/T)
Pt.
2N 2k=1
[ exp(sim(z), /T)
However, P(() is exactly the number of positive samples, and Pr.p is the probability of and being a positive pair according the model. We set s for positive pairs and define it as follows:
supeon be a distribution over 2N, 1 candidates
j P()
supcon
IP()
0,
otherwise
Then, write down the original loss
Coup
(s
supcon
pp)
where H is the cross-cntropy. This looks exactly like the X-CLR objective. We can recover SupCon objective if we increase the temperature 7,: the resulting distribution 8; will be equal 8 supcon
20

---

Published as a conference paper at ICLR 2025
training step(inages, sims):
# encode images with the model we are training img_ ing (nn(age))))
img
# convert targets distributions
ggt_sims softmax tao_s, dim=1) * calculate similarities ing_sims
sontmax
img_emb,
img emb.T
din-1)
the loss the cross-entron
return (E(tgt_ sins, ing sims, dim=1). mean()
Figure 12: Pseudocode for X-CLR loss for general similarities, necessarily coming from text encoding similarities.
A.10 CONNECTION BETWEEN CLIP AND X-CLR
X-CLR and CLIP are similar because they both use images and text data. That said, X-CLR trains only the image representation, and the text is used only to calculate target similarities. The similarities do not have to come from text, as we showed table 2. CLIP, on the other hand, trains both the image and text encoders. CLIP loss can be expressed as follows:
LCLIP = Lrext + Limage
text, exp(sim (2text, 22 e/magee/T)
)/T)
text image
Pijj
Σxe exp(sim(2 .image //7)
2N H(1,,P:)
Ltext
No. 1=1
exp(sim
image ,2,
2/ext) \/T)
imaaee
Pi.j
Eken exp(simn( image text )/T)
2No
Linnage
H(1,.P image-text,
No
Here, N₂ is the batch size, is the dimension of the encodings, Z image E Nox is the batch of image encodings coming from the visual encoder, Ztext N is the batch of text encodings coming from the text encoder model. 1; is a one-hot distribution on N₂ elements, with all the probability on i-th element.
We can see that as opposed to X-CLR loss, CLIP loss has two major differences:
CLIP loss is pushing similarities of images and text encodings together for positive pairs, while X-CLR loss only pushes the images representations to each other. X-CLR only works with image encodings, and beyond the target similarities, doesn't deal with text encodings at all;
X-CLR loss uses soft target distribution cross entropy, while CLIP uses one-hot distribution target.
Due to these fundamental differences, X-CLR is different from CLIP even when the target similarities coming from frozen pre-trained CLIP text encoder. We hypothesize that CLIP could also benefit from soft target similarities, but leave that extension for future work.
A.11 PSEUDOCODE FOR THE GENERAL SIMILARITIES
As showed in table 2, the similarities do not have to come from text encoder. In fig. 12, show pseudocode for how the loss function would look like general target similairites provided in variable sis..
21

---

Published as a conference paper at ICLR 2025
-20
0004
(a) X-CLR
(b) SimCLR
(c) SupCon
Figure 13: X- X-CLR and SupCon representations fall into a well-defined clusters, whereas SimCLR representations are less structured.
A.12 T-SNE OF THE LEARNED REPRESENTATIONS
In fig. 13, show T-SNE plots of representations of few superclasses from ImageNet. We used the 'living 9' set of classes from (Engstrom et al., 2019).
22


Reviews:
[{"review_id": "XbMik6lcyM", "reviewer": "Reviewer_HZRT", "summary": "This work presents X-CLR, a contrastive loss objective that replaces the usual binary positive/negative samples in contrastive learning with soft labels that can vary inside [0,1]. The resulting loss function requires pairwise similarity labels, which the authors suggest can come in the form of metadata or via text embeddings of captions generated from class labels. A comprehensive set of experiments on ImageNet and CC3M/CC12M show improvement in performance of X-CLR relative to existing contrastive learning baselines on classification tasks.", "questions": "1) Some of the results you get for competing methods are far below what those methods report in their papers. Just as an example, in Table 1, ImageNet accuracy from your experiment with SimCLR is 63.4% vs. the 69.3% reported in their paper, and SupCon is 74.3% in your table vs. 78.7% reported in their paper. This is a very significant difference, since in the case of SupCon, their reported result outperforms X-CLR. Did you look into the large differences here? Can you explain them?\n\n2) Please add some more detail to the table captions. The captions right now generally tell me what the takeaway should be, but they don't tell me the specifics of what I'm looking at: things like what the numbers actually represent (accuracy), what the columns mean (i.e., for Table 1, \"Accuracy of networks pretrained on ImageNet with various contrastive learning algorithms, and evaluated on a set of...\"), etc. Also, I think it is a stretch to call Theorem 1 a theorem - this is an re-expression of these losses in matrix form, and the cited paper doesn't even offer a proof. I think these can simply be stated.\n\n3) I found the CLIP experiments section confusing. In your CLIP baseline, are you actually training a vision encoder and language encoder jointly with a CLIP loss, or are you taking a pretrained CLIP model and performing training with X-CLR loss? Also, in general I thought a deeper analysis with CLIP was missing from Section 3, particularly in the case of generating similarities with captions + a CLIP text encoder. Is X-CLR equivalent to CLIP training with a frozen / pre-trained text encoder in this case? If not, how is it different? I think this comparison is important to contextualize your method.\n\n4) In the case of ImageNet, you use class labels to generate text captions, and then embed these to get text embeddings, which ultimately determine the connection strengths in the graph. This creates a similarity matrix that has some particular properties, i.e., images within the same class have a similarity of 1, as in the Supervised graph in Figure 1, along with other nonzero entries based on the similarities of two different images' classes. Does this end up being the same as SupCon? There are some similarities with supervised learning here, but there is no analysis of this similarity.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1) The proposed method and its advantages are intuitive and easy to understand, and there is a comparison with existing vision-based contrastive loss functions.\n\n2) The suite of experimental results is comprehensive, and considers many useful ablations. I particularly appreciated the practical experiments about finetuning and refining existing vision backbones, and the explorations of high-quality vs noisy similarity data.", "weaknesses": "1) There is limited novelty to the proposed contrastive learning loss function, which is a relatively simple swapping of binary positive/negative sample labels in existing contrastive learning losses with a soft label.\n\n2) Pursuant to the first point, there not much detailed analysis of how this method related to existing ones beyond a comparison of the binary vs. soft label graph.\n\n3) The presentation of the paper is inconsistent - there are some very concise and informative figures, like Figure 1, but for example, there are many wording issues throughout, and there are other places where more detail is needed (this is particularly true for table captions).\n\n4) Some weaknesses in experimental results - there is only one network architecture tested, though I accept the given justification to some extent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sYVHh9bwXf", "reviewer": "Reviewer_QWUr", "summary": "This paper proposes a new contrastive objective that uses a target similarity graph based on additional information or models. The modification is applied to contrastive losses in both multi-modal learning (image-text CLIP training) as well as self-supervised learning (images). The proposed modification is called “X-Sample Contrastive Loss” or X-CLR, where the model assumes access to an external source of metadata and possibly pretrained models that can define a graph of similarity between inputs. The goal is to replace the 0-1 hard similarity graph used in contrastive methods with soft similarity graphs based on some ground-truth information.", "questions": "- Are both Tables 1,3,4 using pretrained frozen sentence transformers as the text encoder?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Unifying SSL, multi-modal contrastive learning, and distillation methods in one formulation is helpful in defining generalized training methods.\n- The ablation on the similarity sources in Table 2 is interesting as it shows strong text encoders are not helpful for X-CLR loss than more simple sentence transformer.", "weaknesses": "- The paper should clearly separate the SSL methods on images from image-text methods such as CLIP and clearly define X-CLR modification for both setups. There is a clear difference between contrastive losses for the two setups, one relies on image augmentations to define contrastive pairs while the other relies on paired image-text datasets. Figure 1.b presents a pseudocode for the X-CLR loss that is valid only when paired image-text data is available. In fact, the only reason this loss does not collapse to a trivial and all equal solution is that the `text_emb` is detached from the computation graph.\n- The tables and comparisons should specify which models have access to additional data and/or additional pretrained models. For example, Table 1 compares X-CLR to SimCLR while X-CLR not only has access to the labels, but also access to pretrained text models. The comparison to methods that do not have the advantages of X-CLR is not fair. Specifically, when the method using the labels in ImageNet to define the loss, it can no longer be fairly compared with SSL methods that do not have access to labels. For example, the analysis in section 4.5 that compares the overhead of X-CLR to SimCLR needs to account for the training time of the text encoder as well.\n- On line 354, the paper claims that “X-CLR is more general than model distillation methods as the similarities can come from any source” and cites the TinyCLIP [1]. Are the authors claiming X-CLR can be better than CLIP distillation methods? If so, the paper should compare to state-of-the-art CLIP distillation methods such TinyCLIP [1] and MobileCLIP [2]. That being said, I did not find a clear definition of X-CLR for image-text loss that allows for training both image/text towers even if it uses pretrained image and/or text pretrained models. Specifically, the loss should have image2text and text2image terms or show that it can learn a comparable representation without these terms.\n\n1. Wu et al, Tinyclip: Clip distillation via affinity mimicking and weight inheritance, ICCV 2023.\n2. Vasu et al., MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training, CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wrbxdQUIQS", "reviewer": "Reviewer_R31V", "summary": "This paper introduces a new objective, called $\\mathbb{X}$-Sample Contrastive Learning, where the binary designations in standard contrastive learning are replaced by a soft target based on similarities between samples. These similarities can be obtained through any preferred method, with this paper mainly using frozen text encoders for the experiments. Moreover, the method can be applied fairly easily to any contrastive objective, with this paper focusing on the InfoNCE objective.\n\nThe proposed method is trained on ImageNet, CC3M and CC12M and compared against one supervised learning method and various SSL methodologies. It is shown that $\\mathbb{X}$-CLR leads to consistent improvements for classification and background decomposition compared to the baselines, with minimal increase in computation.", "questions": "As mentioned before, I think the given experiments paint a clear picture of the method. So the questions listed here are mostly out of interest and not things that I necessarily think make sense to include in the paper. That being said, I was wondering a few things about the method:\n  - What happens in imbalanced settings? I imagine that the similarities for an instance with rare caption/label/meta data would be much lower than for instances with more common caption/label/meta data. Do you think this would have a strong effect on the method? Have you tried experimenting with for example ImageNet-LT?\n  - An annoying problem of contrastive learning is its sensitivity to the batch size ($B$), which stems from its effect on the ratio of positives to negatives across the training set ($B^{-1}$). I imagine that $\\mathbb{X}$-CLR partially solves this problem, especially for larger batch sizes, given that the similarities $\\mathbf{G}_{ij}^{(\\text{soft})}$ approximately follow $\\mathbb{P}\\_X(\\text{sim}(X\\_i, X))$, where $\\mathbb{P}\\_X$ denotes the population distribution on inputs $X$. The only bias being the small effect of the true positive $\\mathbf{G}\\_{ii}^{(\\text{soft})} = 1$. Is this something you looked into? If yes, how sensitive is $\\mathbb{X}$-CLR to changes in the batch size? Furthermore, does the application of the softmax to obtain the distribution $s$ have any negative effects for larger batch sizes (since $s_i \\rightarrow 0$ when $B \\rightarrow \\infty$)?\n  - Do the similarities for ImageNet (shown in figure 5a) come from the way the text encoder input is constructed (so from the fixed template part)? If this is the case, do the similarities not give a false view of how strongly related images are? I expected there to be at least some very low similarities for each image, although I might be misunderstanding something here.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This paper recognizes an important problem, which is that commonly used contrastive objectives assume binary similarity scores, which in many settings do not make much sense. The proposed method is an intuitive way of generalizing contrastive learning to non-binary similarity scores, effectively dealing with this problem. $\\mathbb{X}$-CLR is tested through a comprehensive suite of experiments, showcasing its effectiveness and highlighting some of the important design choices that should be taken into account when using this objective.\n\nThe paper is well written with a clear explanation of the problem of existing contrastive learning methods, followed by a great presentation of the proposed method. Moreover, the experiment descriptions are both clear and concise, effectively highlighting the strengths and weaknesses of the method. Finally, the paper is well structured overall, making it enjoyable to read.", "weaknesses": "In my opinion, there are no significant weaknesses in the paper. The only thing I was wondering regarding the experiments is how fair the comparison between $\\mathbb{X}$-CLR and the SSL methods in Table 1 is. It seems to me that $\\mathbb{X}$-CLR is a supervised method in this setting. If the authors agree with me, then it might be nice to explicitly mention this in the text to avoid any possible confusion. That being said, I do think including the comparison is important to showcase the effect of proper similarity scoring, so this is a minor remark.\n\nAside from that, I have some small notes regarding the presentation, which I will list below:\n  - It may be useful to shortly mention what $f_\\theta$ and $K$ are after equation (1) in line 202 if space is not a concern. It is reasonably clear from the context, but being explicit rarely hurts.\n  - In the data  scarcity setting (figure 3a), SupCon appears to be better than $\\mathbb{X}$-CLR, while the text mentions that $\\mathbb{X}$-CLR improves over SupCon.\n  - The coloring of SupCon and SimCLR in figures 3a and 3b is flipped, which is a bit confusing at first glance.\n  - The paragraph related to table 4 does not mention training on CC3M, which might be useful to mention for clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "443fBBI5cf", "reviewer": "Reviewer_MXJi", "summary": "The paper argue for revising the standard contrastive cross-entropy loss from binary targets of a sparse augmentation graph to a soft target graph where the target values can be between [0, 1]. The new method is called X-CLR. The authors suggest to use extra metadata associated with the dataset to find the soft target distribution, and in this work mostly use the pairwise similarities from a pretrained text encoder on text captions/descriptions to encode this distribution. Although the authors note that the framework is general, \"the similarities can come from any source including any metadata\". Conduct experiments to train vision models with text captions/descriptions. The study spans different datasets and the results show improvements over contrastive baselines.", "questions": "Suggestion: Finding a soft target graph without extra metadata or pretrained model with self-supervised techniques would give better applicability.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to read.\n2. Experiment study spans different datasets and comparisons to contrastive baselines.\n3. The authors test with different similarity graphs (random graph, graph inferred from metadata, augmentation graph, true class graph), and find that a soft graph can improve over the binary graph of existing contrastive approaches.", "weaknesses": "1. Relying on extra metadata and a pretrained model limits the applicability compared to the contrastive baselines. I also think this is a simple solution.\n2. The idea of soft-targets for cross-entropy based contrastive loss is very general. It also lack some novelty to the paper's related work on soft-targets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)