$\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs

## Abstract

Abstract Learning good representations involves capturing the diverse ways in which data samples relate. Contrastive loss—an objective matching related samples—underlies methods from self-supervised to multimodal learning. Contrastive losses, however, can be viewed more broadly as modifying a similarity graph to indicate how samples should relate in the embedding space. This view reveals a shortcoming in contrastive learning: the similarity graph is binary, as only one sample is the related positive sample. Crucially, similarities across samples are ignored.
Based on this observation, we revise the standard contrastive loss to explicitly encode how a sample relates to others. We experiment with this new objective, called 𝕏 𝕏 \mathbb{X} blackboard_X -Sample Contrastive, to train vision models based on similarities in class or text caption descriptions.
Our study spans three scales: ImageNet-1k with 1 million, CC3M with 3 million, and CC12M with 12 million samples. The representations learned via our objective outperform both contrastive self-supervised and vision-language models trained on the same data across a range of tasks. When training on CC12M, we outperform CLIP by 0.6 % percent 0.6 0.6\% 0.6 % on both ImageNet and ImageNet Real. Our objective appears to work particularly well in lower-data regimes, with gains over CLIP of 17.2 % percent 17.2 17.2\% 17.2 % on ImageNet and 18.0 % percent 18.0 18.0\% 18.0 % on ImageNet Real when training with CC3M. Finally, our objective seems to encourage the model to learn representations that separate objects from their attributes and backgrounds, with gains of 3.3 3.3 3.3 3.3 - 5.6 5.6 5.6 5.6 % over CLIP on ImageNet9. We hope the proposed solution takes a small step towards developing richer learning objectives for understanding sample relations in foundation models.

## 1 Introduction

Figure: (a)
Refer to caption: x1.png

Contrastive loss underlies methods from self-supervised learning (SSL) to multimodal learning (Radford et al., ; Chen et al., ; Oord et al., ). In SSL, contrastive learning encourages the model to associate a sample with another view of the sample created using hand-crafted data augmentation—this related view is the positive sample. Other samples are then pushed away as negative, unrelated samples in the models’ representation space. Contrastive losses also play a crucial role in multimodal models such as CLIP (Radford et al., ), where the model associates an image with its text caption in representation space. Here contrastive learning designates the caption and image representations as positives while all other text-image pairs are designated as unrelated negatives.

More broadly, contrastive losses can be seen as modifying a similarity graph to indicate how samples should relate in the model’s representation space (Cabannes et al., ). This view reveals a shortcoming in contrastive learning: the similarity graph is binary, as only one sample is the related positive sample. Crucially, similarities across samples, containing precious signals about how aspects of one sample may relate to another, are ignored. For example, as shown in fig. 1(b), contrastive learning treats each text-image pair independently, without explicitly encoding similarities in the images depicting dogs and the others sharing a grassy background. Standard contrastive objectives do not explicitly account for similarities across samples, thereby limiting the quality of the learned representations. Here, we explore here how to capture such similarities by modifying the standard contrastive objective.

To account for similarities across samples, we first remove the binary negative vs. positive designations in standard contrastive loss. We introduce instead a similarity graph with continuous scalars capturing the extent to which two samples are related. Consider the example in fig. 1(b), where the two dog images have a high similarity while the dog and cat images have a more moderate similarity. We experiment with this new objective, called $\mathbb{X}$-Sample Contrastive ($\mathbb{X}$-CLR), by training vision models using a graph of similarities inferred from class or text caption descriptions found in common datasets. Our study spans three training dataset scales from 1 million samples with high-quality labels from ImageNet (Deng et al., ) to 3 and 12 million noisy image-text caption pairs from CC3M and CC12M (Sharma et al., ).

We find that compared to contrastive baseline methods trained on the same data, representation trained using $\mathbb{X}$-CLR outperform contrastive training on a range of tasks from standard classification to tasks involving the decomposition of objects from their attributes and backgrounds. When training on CC12M, we outperform CLIP by 0.6% on both ImageNet and ImageNet Real (Beyer et al., ). Furthermore, $\mathbb{X}$-CLR seems to encourage the model to learn representations that separate objects from their attributes and backgrounds, with gains of 3.4-4.9% over CLIP on ImageNet9 (Xiao et al., ).
We also find for fine-grained disambiguation of object attributes, the quality of labels used to infer the similarity graph is much more important than the data quantity. Compared to noisier web caption data, we find $\mathbb{X}$-CLR trained on 1 million higher quality class labels outperforms representations learned via standard contrastive CLIP trained $12\times$ more data. Finally, we find $\mathbb{X}$-CLR appears to work particularly well in lower-data regimes, with gains over CLIP of 16.8% on ImageNet and 18.1% on ImageNet Real when training with CC3M. In short, we find representations learned using $\mathbb{X}$-CLR generalize better, decompose objects from their attributes and backgrounds, and are more data-efficient.

Our contributions are:

- 1.
We present a graph similarity perspective of contrastive losses, revealing standard losses encode a sparse similarity matrix that treats other, related, samples as negatives.
- 2.
Consequently, we propose a new $\mathbb{X}$-CLR loss that explicitly accounts for similarities across samples
- 3.
We experiment with this objective across three levels of data scale from 1-12 million samples.
- 4.
We find representations learned via $\mathbb{X}$-CLR
(a)
Generalize better on standard classification tasks with consistent gains over contrastive baselines trained on the same data. For example, when training on CC12M we outperform CLIP by 0.6% on both ImageNet and ImageNet Real.
(b)
Disambiguate aspects of images such as attributes and backgrounds more reliably, with gains of 3.3-5.6% over CLIP on background robustness benchmarks for ImageNet.
(c)
Finally, we find $\mathbb{X}$-CLR learns more efficiently when data is scarce, with gains of 17.2% on ImageNet and 18.0% on ImageNet Real when pretraining on the smaller 3 million sample CC3M dataset.

We hope the proposed solution takes a small step towards developing richer learning objectives for understanding sample relations in foundation models to encode richer, more generalizable representations.

## 2 Related Work

#### Contrastive learning

Various contrastive objectives have been proposed over the years (Chopra et al., ; Schroff et al., ). More recently, the InfoNCE objective (Oord et al., ) has been the most popular choice for self-supervised methods, e.g. SimCLR (Chen et al., ) and MoCo (He et al., ). InfoNCE objective has also been successfully used to learn vision-language models using CLIP (Radford et al., ).
The basis of those objectives is to make positive pairs have similar representations,
while the negatives, which typically are just all other elements in a batch, should have a different representation. In its original form, InfoNCE is binary, meaning it only works with positive and negative pairs, and does not support degrees of similarity. The positive pairs are usually two augmentations of the same sample, which makes well-tuned augmentations crucial for good performance (Ryali et al., ). Dwibedi et al. () estimate positives using nearest neighbors in the latent space instead and therefore can use weaker augmentations, while Caron et al. () use cluster assignment.
A few methods have proposed modifications wherein multiple positive pairs are supported,
e.g., Khosla et al. () groups positive by class labels, Hoffmann et al. () propose
using WordNet (Fellbaum, ) hierarchy to define ranked positive samples, and Tian et al. () uses a generative model to obtain multiple positives for the same concept.
HaoChen et al. () also look at contrastive learning through
the lens of graphs, and propose a novel spectral objective.
Wang et al. () draw connections between contrastive learning and message passing on the augmentation graph, while Wang et al. () show that aggressive data-augmentations like cropping can connect samples of the same class.
Zhang et al. () show that contrastive learning
objective implicitly learns the graph in which the samples are connected via augmentations in the case of SimCLR or via captions in the case of CLIP.
However, in that paradigm only visually similar samples or samples with a common caption get connected in the graph, while in our proposed method the samples are connected based on the semantics, and therefore visually dissimilar samples can be connected.

#### Soft targets

Using soft targets provides more learning signal to the model,
possibly making it learn better and faster. This has been explored with distillation by Hinton et al. ().
Soft targets have also been used with InfoNCE in the context of distillation in ReSSL (Zheng et al., ) and SCE (Denize et al., ), where the target cross-sample similarity comes from the teacher model. (Feng and Patras, ) use soft targets from self-distillation to train an image encoder with coarse labels. Similarly, Fini et al. () compute soft targets via latent clustering and apply it to semi-supervised learning. Shen et al. () use patch-mixing to train ViT image encoders to model inter-sample relationships.
Andonian et al. () proposes to use soft targets for CLIP (Radford et al., ) training,
and calculates the targets via self-distillation. Wu et al. () use a similar objective to ours to distill the CLIP model into a smaller one. Further soft CLIP objectives are explored by Fini et al. (), who apply label smoothing to obtain soft targets, and Gao et al. (), who estimate soft targets by comparing fine-grained image information. Finally, Huang et al. () train CLIP with
non-zero cross-sample similarities computed based on pre-trained uni-modal models for
text and vision. In this study, we build on the work of Cabannes et al. () who propose a unifying framework to view SSL and supervised learning objectives as learning with different underlying similarity graphs. We take inspiration from the soft targets literature and propose using a soft graph. As opposed to distillation, we focus more on the
graph design, and try different graph sources, including ones not based on distillation, see table 8.

## 3 Understanding contrastive losses via similarity graphs

### 3.1 X-Sample Graphs

Throughout this study, a similarity graph denotes a graph in which the nodes represent data samples, and edges similarity – relationships. Given the number of data samples in the dataset $N$, a graph is expressed through its symmetric adjacency matrix ${\bm{G}}\in\mathbb{R}^{N\times N}$, the semantic relation between inputs $i$ and $j$ being encoded in the real entry ${\bm{G}}_{i,j}$. In fig. 2, we show graphs of different learning paradigms.
SSL does not rely on labels, but on positive pairs/tuples/views generated at each epoch.
Let us denote by $V$ the number of positive views generated, commonly $V=2$ for positive pairs, and denote by $E$ the training epochs. In that case, the original $N$ input samples are transformed into $N\times V\times E$ “augmented” samples

$$ $\displaystyle{\bm{X}}^{(A)}$ $\displaystyle\triangleq[\underbrace{{\mathcal{T}}({\bm{x}}_{1}),\dots,{ \mathcal{T}}({\bm{x}}_{1})}_{\text{repeated $V\times E$ times}},\dots,{ \mathcal{T}}({\bm{x}}_{N}),\dots,{\mathcal{T}}({\bm{x}}_{N})]^{\top},$ $$

where each ${\mathcal{T}}$ is a random input transformation with its own random parameters. The corresponding graph is given by:

$$ $\displaystyle{\bm{G}}^{(\operatorname{ssl})}_{i,j}={\bm{1}}_{\{\left\lfloor i/ VE\right\rfloor=\left\lfloor j/VE\right\rfloor\}},$ (1) $$

where the associated similarity graph captures if two samples were generated as augmentations of the same original input. Such graphs ${\bm{G}}$, as defined by eq. 1, are the ones used as targets in common SSL methods, as formalized below denoting ${\bm{Z}}\triangleq f_{\theta}({\bm{X}})\in\mathbb{R}^{N\times K}$.

###### Theorem 1 ( (Cabannes et al., 2023 ) ) .

VICReg (Bardes et al., ), SimCLR (Chen et al., ), and BarlowTwins (Zbontar et al., ) losses can be expressed in terms of the graph ${\bm{G}}$ (1)

$$ $\displaystyle{\mathcal{L}}_{\rm VIC^{2}}({\bm{Z}};{\bm{G}})=$ $\displaystyle\|{\bm{Z}}{\bm{Z}}^{T}-{\bm{G}}\|_{F}^{2},$ $\displaystyle{\mathcal{L}}_{\rm SimCLR}({\bm{Z}};{\bm{G}})=$ $\displaystyle-\sum_{i,j\in[N]}{\bm{G}}_{i,j}\log\left(\frac{\exp(\tilde{\bm{z} }_{i}^{\top}\tilde{\bm{z}}_{j})}{\sum_{k\in[N]}\exp(\tilde{\bm{z}}_{i}^{\top} \tilde{\bm{z}}_{k})}\right),$ $\displaystyle{\mathcal{L}}_{\rm BT}({\bm{Z}};{\bm{G}})=$ $\displaystyle\left\|\tilde{\bm{Z}}^{\top}{\bm{G}}\tilde{\bm{Z}}-I\right\|^{2},$ $$

where $\tilde{\bm{z}}\triangleq{\bm{z}}/\left\|{\bm{z}}\right\|$ and $\tilde{\bm{Z}}$ the column normalized ${\bm{Z}}$ so that each column has unit norm.

In our study, we will focus on contrastive learning, i.e., SimCLR family of losses. We will demonstrate how to move away from the ad-hoc graph ${\bm{G}}$ from eq. 1.

Figure: Figure 2: Sample similarity adjacency matrices of existing methods vs. our $\mathbb{X}$-Sample Contrastive similarity loss (right). We show pairwise similarities of 20 samples belonging to 4 classes. Similarity of 1 means the samples are identical, 0 – they are completely unrelated. In case of self-supervised learning, none of the inter-sample relationships are modelled (left). Supervised learning relies on the labels to group samples of the same class together (center). $\mathbb{X}$-CLR models inter-class relationships by associating cats with dogs and pianos with guitars.
Refer to caption: x3.png

###### Theorem 1 ( (Cabannes et al., 2023 ) ) .

### 3.2 Revisiting contrastive losses with similarity graphs: 𝕏 𝕏 \mathbb{X} blackboard_X -CLR

We introduce the soft cross-sample similarity to the widely used InfoNCE objective (Oord et al., ). We note that the proposed framework isn’t necessarily
only limited to InfoNCE-based methods and can potentially be integrated into non-contrastive objectives such as BYOL (Grill et al., ), SimSiam (Chen and He, ), or VICReg (Bardes et al., ), although we leave the extension to other objectives for future work. In SimCLR (Chen et al., ), given a batch of $N_{b}$ images, each image is augmented twice, so each sample has a true positive. The $2N_{b}$ images are then encoded to get representation vectors $z$.
Then:

$$ $\displaystyle p_{i,j}=\frac{\exp(\mathrm{sim}(z_{i},z_{j})/\tau)}{\sum_{i=1}^{ 2N}\mathbbm{1}_{[k\neq i]}\exp(\mathrm{sim}(z_{i},z_{k})/\tau)}$ $\displaystyle\mathcal{L}_{\mathrm{SimCLR}}=\frac{1}{2N_{b}}\sum_{i=1}^{2N_{b}} H(\mathbbm{1}_{i^{\prime}},p_{i})$ $$

where $H$ is the cross-entropy, and $\mathbbm{1}_{i^{\prime}}$ is the one-hot distribution where all the probability mass is assigned to the index of the positive sample corresponding to $i$, and $\mathrm{sim}$ is the cosine similarity. Intuitively, we are training the model to classify positive
examples in a batch, so the similarity $p$ should be high only
for the true positive.
We introduce the soft objective by replacing the hard positive
distribution $\mathbbm{1}_{i^{\prime}}$ with a distribution $s_{i}$. Or, in terms of graphs, we replace the graph from the
eq. 1 with a soft graph where connection strengths can be any number in $[0,1]$, and, similarly, the distribution $s_{i}$ and does not have to be one-hot. Considering the example of fig. 1(b), we want the
a photo of a dog to have a representation similar to that of another photo of a dog, somewhat similar to the representation of a cat photo, and different from the representation of a photo of a mug. Given that distribution $s$, we can plug it in directly:

$$ $\displaystyle\mathcal{L}_{\mathbb{X}\textnormal{-CLR}}=\frac{1}{2N_{b}}\sum_{i =1}^{2N_{b}}H(s_{i},p_{i})$ $$

There are many possible ways to obtain this distribution $s$.
We could use the meta-data associated with the dataset; in our
case, we utilize a trained text encoder $f_{\mathrm{text}}$, and encode the text
provided with each image to obtain a representation, which is then
used to calculate similarity between samples $i$ and $j$ using the
cosine similarity. Those pairwise similarities describe the soft graph:

$$ $\displaystyle{\bm{G}}^{(\mathrm{soft})}_{i,j}=\mathrm{sim}(f_{\mathrm{text}}(c _{i}),f_{\mathrm{text}}(c_{j}))$ $$

Where $c_{i}$ is the caption associated with the $i$-th sample.
The last step before plugging the similarities into the loss function is converting
them to a valid probability distribution using a softmax function:

$$ $s_{i,j}=\frac{\exp({\bm{G}}^{(\mathrm{soft})}_{i,j}/\tau_{s})}{\sum_{k=1}^{2N_ {b}}\exp({\bm{G}}^{(\mathrm{soft})}_{i,k}/\tau_{s})}$ $$

Note that $\tau_{s}$ is a separate hyperparameter from $\tau$ in the softmax to calculate the learned
similarities. Higher values of $\tau_{s}$ put more weight on the ’soft’ positives, while lower values in the limit recover the original SimCLR objective.

## 4 Experiments

### 4.1 Experimental setup

We test $\mathbb{X}$-CLR on three datasets of varying scale: ImageNet (Deng et al., ) (1M), and conceptual captions 3M and 12M (Sharma et al., ). We blur faces in all datasets before training our models. We compare to SimCLR (Chen et al., ), to CLIP (Radford et al., ) when captions are available. On ImageNet, we compare to SupCon (Khosla et al., ) which uses lables; SCE (Denize et al., ) and ReSSL (Zheng et al., ) which use self-distillation with soft targets, and to SimCLR (Chen et al., ), VICReg (Bardes et al., ) and BarlowTwins (Zbontar et al., ) which are purely self-supervised algorithms for learning from images. On ImageNet, we also report standard deviations over 5 seeds for the models which we trained in table table 7 (SimCLR, SupCon, and $\mathbb{X}$-CLR). For the remaining ImageNet models, we took pre-trained encoders.
We use the Sentence Transformer (Reimers and Gurevych, ) as the text encoder to construct similarities.
For ImageNet experiments, we generate captions by using the template "a photo of a _" to generate captions out of class names. In our experiments with the conceptual captions dataset (Sharma et al., ), we use the captions as is.
For experiments on ImageNet, we follow SupCon and use
AutoAugment (Cubuk et al., ).
All experiments on the ImageNet dataset were run for 100 epochs with
1024 batch size. The learning rate was set to 0.075 for ImageNet models.
For experiments on CC3M and CC12M, we used the standard SimCLR augmentations, and a learning rate of 0.1.
The rest of the settings were kept the same. For more details, see section A.7.

In all our experiments, to isolate the effect of our learning objective, we fix the backbone architecture to be a ResNet-50 (He et al., ) model as this is the most widely studied, with optimized hyperparameters, for standard contrastive self-supervised learning (Chen et al., ). We use the same architecture for CLIP’s vision encoder and take advantage of already optimized publicly available checkpoints provided by OpenCLIP (Ilharco et al., ) for CC12M. Since no comparable public checkpoint is available for CC3M, we train our own model, see section A.6.

### 4.2 𝕏 𝕏 \mathbb{X} blackboard_X -Sample Contrastive with Well-Labeled Samples

We first experiment with $\mathbb{X}$-Sample Contrastive using well-labeled samples to understand the effect of incorporating similarities across samples in the training objective.
To do so, we use class labels from ImageNet.
We compare $\mathbb{X}$-Sample Contrastive ($\mathbb{X}$-CLR) to SimCLR as well as Supervised Contrastive (SupCon), a model whose objective is to explicitly match samples based on their class labels.
We evaluate all models across a suite of benchmarks to gauge how well representations generalize in terms of classification performance.

We find in table 1 representations learned via $\mathbb{X}$-CLR improve on standard classification performance, with gains of 12.2% relative to SimCLR and 1.3% relative to Supervised Contrastive on ImageNet. We find similar gains when evaluated on revised labels from ImageNet Real of 13.7% and 1.8%, respectively. Finally, we find by capturing similarities across samples, representations learned via $\mathbb{X}$-CLR are more capable of disambiguating objects from backgrounds and attributes with gains on ImageNet-9 (for details see section A.5) (Xiao et al., ) and ObjectNet (Barbu et al., ).

**Table 1: $\mathbb{X}$-Sample Contrastive loss outperforms contrastive (SimCLR) and even Supervised Contrastive with ImageNet pretraining.**
|  |  |  | Background Decomposition |  | MIT States |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Method | ImageNet | ImageNet Real | Same Class | Mixed | ObjectNet | Objects | Attributes |
| SCE | 71.3 | 78.7 | 61.7 | 58.4 | 20.2 | 44.5 | 31.0 |
| ReSSL (1 crop) | 69.4 | 76.9 | 56.3 | 53.2 | 18.3 | 44.5 | 31.2 |
| VICReg | 72.4 | 79.0 | 60.8 | 56.8 | 20.5 | 43.5 | 26.9 |
| Barlow Twins | 72.8 | 80.0 | 62.7 | 59.4 | 21.6 | 45.9 | 31.7 |
| SimCLR | 63.4 | 67.8 | 44.7 | 38.9 | 12.1 | 40.9 | 29.1 |
| SupCon | 74.3 | 79.7 | 64.0 | 59.9 | 24.4 | 45.6 | 30.8 |
| $\mathbb{X}$-CLR | 75.6 | 81.5 | 66.6 | 62.7 | 27.5 | 45.9 | 31.1 |

Figure: (a)
Refer to caption: x4.png

#### Can we improve contrastive learning under data scarcity?

To answer this question, we train all three models SimCLR, SupCon, and $\mathbb{X}$-CLR by varying the number of samples seen for each class in ImageNet. We find $\mathbb{X}$-CLR, by incorporating information about class labels and how they relate, is able to learn representations that improve over the performance of SupCon trained with ground truth class labels and outperform SimCLR even when few training samples are available per class as shown in fig. 3(a).

### 4.3 𝕏 𝕏 \mathbb{X} blackboard_X -Sample Contrastive with Noisy Multimodal Samples

Contrastive loss also plays a pivotal role in multimodal vision-language models such as CLIP. The contrastive training objective matches noisy caption-image pairs. Here we experiment with $\mathbb{X}$-Sample Contrastive by using the noisy captions to learn similarities across samples. We compare both SimCLR as a standard contrastive model and CLIP trained on the same caption-image data across two levels of scale: 3 and 12 million samples from CC3M and CC12M.

We find incorporating $\mathbb{X}$-Contrastive leads to representations with higher classification accuracy and disambiguation of objects from their attributes and backgrounds. With CC12M training shown in table 3, $\mathbb{X}$-Contrastive outperforms SimCLR by 0.5% and CLIP by 0.6% with CC12M with similar gains for ImageNet Real. We also find $\mathbb{X}$-CLR training can better disambiguate object foreground from backgrounds, with gains of 0.6-1.5% over SimCLR and 3.3-5.6% over CLIP.

We find learning similarites across samples with $\mathbb{X}$-CLR leads to more considerable gains when less data is available. $\mathbb{X}$-CLR outperforms SimCLR by 1.2% and CLIP by 17.2% on ImageNet, with similar gains on ImageNet Real as shown in table 2. We find $\mathbb{X}$-CLR training can more considerably disambiguate object foregrounds from backgrounds compared to CLIP when less training data is available, with gains of 10.3-14.2% over CLIP.

**Table 2: $\mathbb{X}$-Sample Contrastive with CC3M training outperforms contrastive baselines.**
|  |  |  | Background Decomposition |  |  |
| --- | --- | --- | --- | --- | --- |
| Method | ImageNet | ImageNet Real | Same Class | Mixed | ObjectNet |
| SimCLR | 57.0 | 64.0 | 24.4 | 18.9 | 10.8 |
| CLIP | 41.0 | 47.6 | 12.5 | 10.6 | 7.8 |
| $\mathbb{X}$-CLR | 58.2 | 65.6 | 26.7 | 20.3 | 11.5 |

**Table 3: $\mathbb{X}$-Sample Contrastive with CC12M training outperforms contrastive baselines.**
|  |  |  | Background Decomposition |  |  |
| --- | --- | --- | --- | --- | --- |
| Method | ImageNet | ImageNet Real | Same Class | Mixed | ObjectNet |
| SimCLR | 58.9 | 66 | 24.6 | 19.8 | 12.7 |
| CLIP | 58.8 | 66.1 | 20.5 | 17.1 | 11.9 |
| $\mathbb{X}$-CLR | 59.4 | 66.7 | 26.1 | 20.4 | 13.4 |

### 4.4 𝕏 𝕏 \mathbb{X} blackboard_X -Sample Contrastive can be used to finetune pretrained backbones

**Table 4: $\mathbb{X}$-CLR can be used to finetune pretrained models.**
|  |  |  | Background Decomposition |  |  |
| --- | --- | --- | --- | --- | --- |
|  | ImageNet | ImageNet Real | Same Class | Mixed | ObjectNet |
| SimCLR | 63.4 | 67.8 | 44.7 | 38.9 | 12.1 |
| + $\mathbb{X}$-CLR finetuning | 66.5 | 74.4 | 53.9 | 50.0 | 17.4 |

We validate whether $\mathbb{X}$-CLR can be used as a finetuning objective for pretrained backbones, given the growing abundance of publicly available backbones. Here, we evaluate a pretrained SimCLR model by finetuning for 10 epochs on ImageNet with $\mathbb{X}$-CLR instead of the original SimCLR contrastive objective. We see in table 4 finetuning with $\mathbb{X}$-CLR improves classification performance on ImageNet by 3.1% and on ImageNet Real by 6.6%. Furthermore, we see by relating samples during the finetuning stage, $\mathbb{X}$-CLR can disambiguate object foregrounds from backgrounds with grains of 9.2-11.1% on ImageNet-9 as well as improvements on natural object transformations from ObjectNet with a gain of 5.3% after finetuning.

### 4.5 𝕏 𝕏 \mathbb{X} blackboard_X -Sample Contrastive introduces only minimal computational overhead

Both for ImageNet and conceptual captions datasets, we don’t run the text encoder
for each sample we see, and instead precompute the similarity values. For more details, see section A.7. Avoiding running the text encoder during model training avoids the extra overhead at the price of some pre-processing. Pre-processing takes less than 2 hours for CC12M when using one GPU, about 30 minutes for CC3M, and less than 5 minutes for ImageNet. To further analyze how much overhead there is, we compare the average
time it takes to process one batch for SimCLR and $\mathbb{X}$-CLR. The results are shown in table 5.
Overall, we didn’t notice any significant difference in the amount of time it takes to train models with the $\mathbb{X}$-CLR objective compared to the regular contrastive objective. To train on ImageNet, we used 8 Nvidia V100s, and each run took about 30 hours. With the same setup, CC3M runs took about 50 hours, and CC12M runs took roughly 9 days.

**Table 5: Analyzing the computation overhead of the $\mathbb{X}$-Sample Contrastive objective during training. $\mathbb{X}$-CLR introduces nearly no computational overhead compared to SimCLR.**
| Method | Seconds per batch ImageNet | Seconds per batch CC |  |  |
| --- | --- | --- | --- | --- |
| SimCLR | 0.866 ± 0.008 | 0.874 ± 0.034 |  |  |
| $\mathbb{X}$-CLR | 0.866 ± 0.010 | 0.877 ± 0.032 |  |  |

## 5 Analyzing representations learned via 𝕏 𝕏 \mathbb{X} blackboard_X -Sample Contrastive

### 5.1 KNN Clustering

To confirm the representations learned via $\mathbb{X}$-CLR also work well for downstream tasks with non-linear decision boundaries, we perform evaluation using the common K-nearest neighbor (KNN) protocol.
The results shown in fig. 3(b) demonstrate $\mathbb{X}$-CLR outperforms both SimCLR and SupCon baselines across a range of choices for $K$. We also show KNN results for models trained on conceptual captions in section A.4.

### 5.2 Visualizing the learned graph from 𝕏 𝕏 \mathbb{X} blackboard_X -Sample Contrastive representations

Here we examine whether the learned representations from $\mathbb{X}$-Sample Contrastive capture semantically meaningful similarities.
To do so, we select four groups of three ImageNet classes: felines, dogs, types of balls, and musical instruments.
For each pair of classes, we then compare the representation similarities using cosine similarity. A higher average pairwise similarity indicates the model’s latent representations encode the classes similarly.
In fig. 4 we show the graph of similarities learned after training with $\mathbb{X}$-CLR on ImageNet.
We find that the image encoder successfully captures the similarity within the class groups.

### 5.3 The effect of softmax temperature, and inferred similarity graph

We also examine the effect of hyperparameter choices. We show the sensitivity of $\mathbb{X}$-CLR to temperature $\tau_{s}$ in fig. 3(c) on ImageNet. In the limit, when temperature goes to 0, we recover Supervised Contrastive method for ImageNet, or SimCLR in case of conceptual captions. With low temperature, the similarity is 1 only if the captions are exactly the same. As
the temperature increases, more weight is put on the soft positives compared to the true positives (i.e. augmentations
of the same sample). With high temperature, our method is unstable as too much emphasis is put on the soft positive examples compared to the true positives. We find that the value of 0.1 strikes the optimal balance and provides an improvement over pure Supervised Contrastive objective, while still emphasizing true positives enough. For more details
regarding how $\tau_{s}$ changes the objective, see fig. 8(b).

We also experiment with different ways of inferring the graph, including using different text encoders,
using WordNet (Fellbaum, ) hierarchy distance, and the purely random graph. We find that overall, calculating the similarities using the sentence transformer worked the best (Reimers and Gurevych, ). A more detailed comparison of different graph sources can be found in section A.1.

### 5.4 The impact of label quality for fine-grained attribute disambiguation

we show in table 6 how label quality can impact downstream performance on finer-grained attribute disambiguation. we find larger labels from noisy captions degrades performance for fine-grained object attributes in mit states (Isola et al., ) for both contrastive and clip. we find $\mathbb{X}$-CLR with high quality labels from imagenet, can outperform models trained on much larger noisier data. compared to clip trained on 12$\times$ larger data, $\mathbb{X}$-CLR achieves 30.9% vs. 23.3% for clip on attribute classification and 45.8% vs. 36.9% for clip on object classification under different states. to see more details regarding the mit states evaluation, see section A.7.

**Table 6: Label quality matters for fine-grained attribute disambiguation.**
| Pretraining | Data Size | Quality | MIT States Attributes | MIT States Objects |
| --- | --- | --- | --- | --- |
| CLIP CC3M | 3M | Noisy | 27.0 | 40.1 |
| CLIP CC12M | 12M | Noisy | 23.3 | 36.9 |
| $\mathbb{X}$-CLR CC3M | 3M | Noisy | 29.5 | 40.7 |
| $\mathbb{X}$-CLR CC12M | 12M | Noisy | 30.1 | 42.1 |
| $\mathbb{X}$-CLR ImageNet | 1M | High | 30.9 | 45.8 |

Figure: Figure 4: Visualizing pairwise similarities SupCon (Khosla et al., ) objective does not encourage non-zero similarity between samples of different classes (left), while $\mathbb{X}$-CLR target similarities take into account semantic closeness within categories such as dogs or types of balls (center). On the right, we see that the trained model successfully learns the soft similarity. For more graphs, see fig. 5.
Refer to caption: x7.png

## 6 Discussion

In the present work, we have proposed a new graph perspective on the commonly used contrastive learning methods and used our insights to develop a better learning objective, $\mathbb{X}$-CLR, by using a soft similarity graph.
The adjacency matrix of the proposed graph contains not just 0 and 1, but also any values between,
with the ability to capture the degree of similarity across samples. We experiment with different ways of constructing the
graph, and find that indeed we can build a soft graph that improves over the existing binary graph contrastive methods. However, we believe that there are better ways of constructing the graph than
what we found, particularly for the conceptual captions dataset where the captions are quite noisy.
A better graph can possibly be built using other metadata, such as location or time. We also
believe that ideas from $\mathbb{X}$-CLR can possibly be integrated into non-contrastive objectives such as BYOL (Grill et al., ) or VICReg (Bardes et al., ) to enrich representations with similarities across samples.

Limitations   The main limitation of the present work is that constructing the cross-sample similarity graph requires extra data, as well as some extra memory to store it. When the extra data is not available, the only options remaining are to build the graph using the augmentations,
self-distillation, or other pre-trained models. The resulting method is also highly dependent on the quality of the graph, as we have seen with conceptual captions datasets.

## 7 Acknowledgements

This material is based upon work supported by the National Science Foundation under NSF Award 1922658.

## Appendix A Appendix / supplemental material

**Table 7: Analyzing statistical significance of ImageNet results. Each experiment is ran with 5 seeds, we report the mean and standard deviation.**
|  |  |  | Background Decomposition |  | MIT States |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Method | ImageNet | ImageNet Real | Same Class | Mixed | ObjectNet | Objects | Attributes |
| SimCLR | 63.43 ± 0.12 | 67.75 ± 0.27 | 12.07 ± 0.33 | 38.88 ± 0.43 | 44.67 ± 0.60 | 40.92 ± 0.26 | 29.08 ± 0.17 |
| SupCon | 74.30 ± 0.16 | 79.66 ± 0.12 | 24.42 ± 0.25 | 59.08 ± 0.44 | 64.00 ± 0.62 | 45.56 ± 0.16 | 30.83 ± 0.20 |
| $\mathbb{X}$-CLR | 75.56 ± 0.09 | 81.54 ± 0.13 | 27.53 ± 0.13 | 62.74 ± 0.27 | 66.59 ± 0.25 | 45.86 ± 0.15 | 31.10 ± 0.18 |

### A.1 More learned similarities comparisons

We compare inferring the similarity graph using different text encoders:

- •
Graph with connections only between samples of the same class (SupCon);
- •
Graph with connections only between augmentations of the same image (SimCLR);
- •
Graph where soft similarity is inferred by comparing representations of the sample captions. The representations are computed using the sentence transformer [Reimers and Gurevych, ], CLIP text encoder [Radford et al., ], LLama2 encoder [Touvron et al., ];
- •
Graph where the connection strength is defined by the distance in WordNet [Fellbaum, ] hierarchy;
- •
Random graph where the cross-sample connections’ strengths are fully random;

The results are shown in table 8. We find that overall, the Sentence Transformer graph performs the best,
although the CLIP text encoder achieves good performance as well. Interestingly, we find that using WordNet hierarchy distance did not work well.
We visualize learned and target similarities for SupCon graph and for the graph built using CLIP text encoder in fig. 5.

#### Visualising similarities

In fig. 4, to visualize learned similarities, for
each class we pick 100 examples from the dataset, encode them. Then, to calculate the average learned similarity between
two classes, we take the 100 examples for each of the two classes, and calculate the Cartesian product, yielding 10,000 similarities. We take the mean over those 10,000 similarities to represent the average learn similarity for a class pair.

#### Similarities when training on CC datasets

In fig. 6(b), we show the similarities learned by $\mathbb{X}$-CLR on CC3M and CC12M datasets.

### A.2 Analyzing statistical significance of the results

To make sure the difference in performance we observe is statistically significant, we run $\mathbb{X}$-CLR, SimCLR, and SupCon
pretraining with 5 different seeds. We report the results of the evaluations in table 7.

Figure: (a) SupCon target and learned similarities
Refer to caption: x8.png

Figure: (a) CC3M similarities
Refer to caption: x10.png

**Table 8: The effect of the similarity source on the model performance.**
|  |  |  | Background Decomposition |  |  |
| --- | --- | --- | --- | --- | --- |
| Similarity source | ImageNet | ImageNet Real | Same Class | Mixed | ObjectNet |
| Augmentation graph (SimCLR) | 63.2 | 67.5 | 45.5 | 38.3 | 12.5 |
| Sentence Transformer ($\mathbb{X}$-CLR) | 75.6 | 81.6 | 66.5 | 62.3 | 27.7 |
| CLIP text encoder | 74.4 | 80.6 | 67.5 | 64.2 | 24.5 |
| LLama2 text encoder | 40.9 | 45.8 | 38.3 | 36.0 | 4.3 |
| Random per class pair | 74.5 | 80.8 | 71.0 | 68.0 | 26.6 |
| Random per sample pair | 0.1 | 0.1 | 0 | 0 | 0 |
| True class graph (SupCon) | 74.4 | 79.7 | 63.3 | 58.8 | 24.1 |
| Distance in WordNet hierarchy | 68.3 | 74.9 | 55.7 | 52.1 | 21.2 |

### A.3 Analyzing the learned graph

**Table 9: Analyzing the learned representations’ connectivity**
| Metric | CLIP | SimCLR | SupCon | X-CLR |
| --- | --- | --- | --- | --- |
| Label error (↓) | 0.550 | 0.250 | 0.250 | 0.223 |
| Intra-class connectivity (↑) | 1.233 | 1.700 | 2.005 | 2.193 |

We follow the analysis of Zhang et al. [] and results in table 9. The analysis studies two values: label error which measures how similar samples of different classes are on average, and intra-class connectivity, which measures the similarity of the samples within the class relative to those from different classes. This allows us to determine how well the learned graph captures the class relationships in the data.
Since the open-source repository of that paper did not contain the code for analysis, we re-implemented it to the best of our ability.

According to these metrics, $\mathbb{X}$-CLR representation is the best among the baselines.
We note that our SimCLR numbers are much better than in the original paper. We suspect that it’s due to the fact that the authors train SimCLR with the batch size of 512, while we use 2048. ImageNet classification performance of our SimCLR model is also higher, at 63.4, compared to 61.2.

We also note that label error, which is the measure of average similarity between instances of different classes, is lower for our method, although the loss itself encourages it to be higher for related samples. This is due to the fact that in this analysis, we use the first 10 classes from ImageNet (replicating the original procedure), and those classes are not related to each other.

### A.4 KNN evaluation

Apart from testing the models trained on ImageNet using KNN, we also evaluate the models trained on CC3M and CC12M. The results are shown in fig. 7.
We see that $\mathbb{X}$-CLR performs better on CC3M, and comparatively with SimCLR when trained on CC12M.

### A.5 ImageNet-9 details

ImageNet-9 [Xiao et al., ] proposes multiple benchmarks to test model robustness to the background perturbation.
In our work, we use "Mixed-Same" and "Mixed-Rand" tasks from ImageNet-9, and refer to them together as "Background Decomposition".

Figure: Figure 7: Results of models trained on ImageNet, CC3M, CC12M on ImageNet validation when using KNN classifier.
Refer to caption: x12.png

### A.6 CLIP details

In CC3M experiments, we train the model from scratch, as OpenCLIP didn’t have a checkpoint trained on that dataset.
We trained both for 32 and 100 epochs, and found that the model trained for 100 epochs performs better.
Since 32 epochs is the default CLIP number of epochs, we also report results for 32 epochs. The results are
shown in table 10.

**Table 10: CLIP on CC3M We train our own models on CC3M and find that training longer improves the performance. Nevertheless, CLIP struggles with small datasets.**
|  |  |  | Background Decomposition |  |  |
| --- | --- | --- | --- | --- | --- |
| Method | ImageNet | ImageNet Real | Same Class | Mixed | ObjectNet |
| CLIP 100 epochs | 41.0 | 47.6 | 12.5 | 10.6 | 7.8 |
| CLIP 32 epochs | 36.8 | 42.0 | 11.5 | 9.8 | 6.0 |

### A.7 More training details

We train SimCLR, SupCon and $\mathbb{X}$-CLR using the LARS optimizer [You et al., ].
In all cases, we use the same ResNet-50, with a two layer projector on top. The output dimension of the projector is 128.

#### Fetching similarities

For ImageNet, since the number of classes is known, we pre-compute the similarity matrix of dimension
$1000\times 1000$, and retrieve elements from it depending on the associated class
labels for a given sample pair to obtain the similarity value.
For conceptual captions, we run the text encoder on the full dataset and save the encodings to disk. Then, when loading an image from disk, we also load the associated
encoding of the corresponding caption. The similarity matrix for a given batch is then
obtained by calculating the Cartesian product of those encodings.

#### MIT States

In order to evaluate on this dataset using linear probing, we split the dataset randomly into two
even parts, one used for training the linear layer, the other for evaluation. We train separately to classify objects and
attributes.

### A.8 Understanding similarities

Figure: (a)
Refer to caption: x13.png

To understand the graphs we built using for different datasets, we investigate the average cross-sample
similarity in the dataset. The result is shown in fig. 8(a). We find that CC3M similarities are in
general lower, possibly because of lower quality annotations. We also investigate how much weight is assigned to the
true positive examples. For SimCLR, it’s always 1. For our method, the amount of similarity assigned to other samples in the
batch depends on the temperature $\tau_{s}$, and the batch size. The exact relationship is shown in fig. 8(b).

### A.9 Connection between Supervised Contrastive Learning and 𝕏 𝕏 \mathbb{X} blackboard_X -CLR

Here, we will outline how as the temperature $\tau_{s}$ approaches 0, $\mathbb{X}$-CLR becomes SupCon.
Supervised Contrastive Learning [Khosla et al., ] also uses image augmentations, and augments each image twice, to obtain what they call "a multiviewed batch". Then, in equation 2, they propose the loss:

$$ $\mathcal{L}_{\mathrm{out}}^{\mathrm{sup}}=\sum_{i\in I}\mathcal{L}_{\mathrm{ out},i}^{\mathrm{sup}}=\sum_{i\in I}\frac{-1}{|P(i)|}\sum_{p\in P(i)}\log p_{i ,p}$ $$

where $p_{i,j}$ is defined as follows:

$$ $p_{i,j}=\frac{\exp(\mathrm{sim}(z_{i},z_{j})/\tau)}{\sum_{i=1}^{2N_{b}}\mathbb {1}_{[k\neq i]}\exp(\mathrm{sim}(z_{i},z_{k})/\tau)}$ $$

However, $|P(i)|$ is exactly the number of positive samples, and $p_{i,p}$ is the probability of $i$ and $p$ being a positive pair according to the model. We set $s_{i}^{\mathrm{supcon}}$ to be a distribution over $2N_{b}-1$ candidates for positive pairs and define it as follows:

$$ $s_{i,j}^{\mathrm{supcon}}=\begin{cases}\frac{1}{|P(i)|},&\text{if }j\in P(i)\\ 0,&\text{otherwise}\end{cases}$ $$

Then, we can write down the original loss as:

$$ $\mathcal{L}_{\mathrm{out},i}^{\mathrm{sup}}=H(s_{i}^{\mathrm{supcon}},p_{i})$ $$

where H is the cross-entropy. This looks exactly like the $\mathbb{X}$-CLR objective. We can recover SupCon objective if we increase the temperature $\tau_{s}$: the resulting distribution $s_{i}$ will be equal to $s_{i}^{\mathrm{supcon}}$.

### A.10 T-SNE of the learned representations

In fig. 9(c), we show T-SNE plots of representations of a few superclasses from ImageNet. We used the ’living 9’
set of classes from [Engstrom et al., ].

Figure: (a) $\mathbb{X}$-CLR
Refer to caption: x15.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1) Some of the results you get for competing methods are far below what those methods report in their papers. Just as an example, in Table 1, ImageNet accuracy from your experiment with SimCLR is 63.4% vs. the 69.3% reported in their paper, and SupCon is 74.3% in your table vs. 78.7% reported in their paper. This is a very significant difference, since in the case of SupCon, their reported result outperforms X-CLR. Did you look into the large differences here? Can you explain them?

2) Please add some more detail to the table captions. The captions right now generally tell me what the takeaway should be, but they don't tell me the specifics of what I'm looking at: things like what the numbers actually represent (accuracy), what the columns mean (i.e., for Table 1, "Accuracy of networks pretrained on ImageNet with various contrastive learning algorithms, and evaluated on a set of..."), etc. Also, I think it is a stretch to call Theorem 1 a theorem - this is an re-expression of these losses in matrix form, and the cited paper doesn't even offer a proof. I think these can simply be stated.

3) I found the CLIP experiments section confusing. In your CLIP baseline, are you actually training a vision encoder and language encoder jointly with a CLIP loss, or are you taking a pretrained CLIP model and performing training with X-CLR loss? Also, in general I thought a deeper analysis with CLIP was missing from Section 3, particularly in the case of generating similarities with captions + a CLIP text encoder. Is X-CLR equivalent to CLIP training with a frozen / pre-trained text encoder in this case? If not, how is it different? I think this comparison is important to contextualize your method.

4) In the case of ImageNet, you use class labels to generate text captions, and then embed these to get text embeddings, which ultimately determine the connection strengths in the graph. This creates a similarity matrix that has some particular properties, i.e., images within the same class have a similarity of 1, as in the Supervised graph in Figure 1, along with other nonzero entries based on the similarities of two different images' classes. Does this end up being the same as SupCon? There are some similarities with supervised learning here, but there is no analysis of this similarity.
soundness: 2
strengths: 1) The proposed method and its advantages are intuitive and easy to understand, and there is a comparison with existing vision-based contrastive loss functions.

2) The suite of experimental results is comprehensive, and considers many useful ablations. I particularly appreciated the practical experiments about finetuning and refining existing vision backbones, and the explorations of high-quality vs noisy similarity data.
summary: This work presents X-CLR, a contrastive loss objective that replaces the usual binary positive/negative samples in contrastive learning with soft labels that can vary inside [0,1]. The resulting loss function requires pairwise similarity labels, which the authors suggest can come in the form of metadata or via text embeddings of captions generated from class labels. A comprehensive set of experiments on ImageNet and CC3M/CC12M show improvement in performance of X-CLR relative to existing contrastive learning baselines on classification tasks.
weaknesses: 1) There is limited novelty to the proposed contrastive learning loss function, which is a relatively simple swapping of binary positive/negative sample labels in existing contrastive learning losses with a soft label.

2) Pursuant to the first point, there not much detailed analysis of how this method related to existing ones beyond a comparison of the binary vs. soft label graph.

3) The presentation of the paper is inconsistent - there are some very concise and informative figures, like Figure 1, but for example, there are many wording issues throughout, and there are other places where more detail is needed (this is particularly true for table captions).

4) Some weaknesses in experimental results - there is only one network architecture tested, though I accept the given justification to some extent.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: - Are both Tables 1,3,4 using pretrained frozen sentence transformers as the text encoder?
soundness: 2
strengths: - Unifying SSL, multi-modal contrastive learning, and distillation methods in one formulation is helpful in defining generalized training methods.
- The ablation on the similarity sources in Table 2 is interesting as it shows strong text encoders are not helpful for X-CLR loss than more simple sentence transformer.
summary: This paper proposes a new contrastive objective that uses a target similarity graph based on additional information or models. The modification is applied to contrastive losses in both multi-modal learning (image-text CLIP training) as well as self-supervised learning (images). The proposed modification is called “X-Sample Contrastive Loss” or X-CLR, where the model assumes access to an external source of metadata and possibly pretrained models that can define a graph of similarity between inputs. The goal is to replace the 0-1 hard similarity graph used in contrastive methods with soft similarity graphs based on some ground-truth information.
weaknesses: - The paper should clearly separate the SSL methods on images from image-text methods such as CLIP and clearly define X-CLR modification for both setups. There is a clear difference between contrastive losses for the two setups, one relies on image augmentations to define contrastive pairs while the other relies on paired image-text datasets. Figure 1.b presents a pseudocode for the X-CLR loss that is valid only when paired image-text data is available. In fact, the only reason this loss does not collapse to a trivial and all equal solution is that the `text_emb` is detached from the computation graph.
- The tables and comparisons should specify which models have access to additional data and/or additional pretrained models. For example, Table 1 compares X-CLR to SimCLR while X-CLR not only has access to the labels, but also access to pretrained text models. The comparison to methods that do not have the advantages of X-CLR is not fair. Specifically, when the method using the labels in ImageNet to define the loss, it can no longer be fairly compared with SSL methods that do not have access to labels. For example, the analysis in section 4.5 that compares the overhead of X-CLR to SimCLR needs to account for the training time of the text encoder as well.
- On line 354, the paper claims that “X-CLR is more general than model distillation methods as the similarities can come from any source” and cites the TinyCLIP [1]. Are the authors claiming X-CLR can be better than CLIP distillation methods? If so, the paper should compare to state-of-the-art CLIP distillation methods such TinyCLIP [1] and MobileCLIP [2]. That being said, I did not find a clear definition of X-CLR for image-text loss that allows for training both image/text towers even if it uses pretrained image and/or text pretrained models. Specifically, the loss should have image2text and text2image terms or show that it can learn a comparable representation without these terms.

1. Wu et al, Tinyclip: Clip distillation via affinity mimicking and weight inheritance, ICCV 2023.
2. Vasu et al., MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training, CVPR 2024.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 3
questions: As mentioned before, I think the given experiments paint a clear picture of the method. So the questions listed here are mostly out of interest and not things that I necessarily think make sense to include in the paper. That being said, I was wondering a few things about the method:
  - What happens in imbalanced settings? I imagine that the similarities for an instance with rare caption/label/meta data would be much lower than for instances with more common caption/label/meta data. Do you think this would have a strong effect on the method? Have you tried experimenting with for example ImageNet-LT?
  - An annoying problem of contrastive learning is its sensitivity to the batch size ($B$), which stems from its effect on the ratio of positives to negatives across the training set ($B^{-1}$). I imagine that $\mathbb{X}$-CLR partially solves this problem, especially for larger batch sizes, given that the similarities $\mathbf{G}_{ij}^{(\text{soft})}$ approximately follow $\mathbb{P}\_X(\text{sim}(X\_i, X))$, where $\mathbb{P}\_X$ denotes the population distribution on inputs $X$. The only bias being the small effect of the true positive $\mathbf{G}\_{ii}^{(\text{soft})} = 1$. Is this something you looked into? If yes, how sensitive is $\mathbb{X}$-CLR to changes in the batch size? Furthermore, does the application of the softmax to obtain the distribution $s$ have any negative effects for larger batch sizes (since $s_i \rightarrow 0$ when $B \rightarrow \infty$)?
  - Do the similarities for ImageNet (shown in figure 5a) come from the way the text encoder input is constructed (so from the fixed template part)? If this is the case, do the similarities not give a false view of how strongly related images are? I expected there to be at least some very low similarities for each image, although I might be misunderstanding something here.
soundness: 3
strengths: This paper recognizes an important problem, which is that commonly used contrastive objectives assume binary similarity scores, which in many settings do not make much sense. The proposed method is an intuitive way of generalizing contrastive learning to non-binary similarity scores, effectively dealing with this problem. $\mathbb{X}$-CLR is tested through a comprehensive suite of experiments, showcasing its effectiveness and highlighting some of the important design choices that should be taken into account when using this objective.

The paper is well written with a clear explanation of the problem of existing contrastive learning methods, followed by a great presentation of the proposed method. Moreover, the experiment descriptions are both clear and concise, effectively highlighting the strengths and weaknesses of the method. Finally, the paper is well structured overall, making it enjoyable to read.
summary: This paper introduces a new objective, called $\mathbb{X}$-Sample Contrastive Learning, where the binary designations in standard contrastive learning are replaced by a soft target based on similarities between samples. These similarities can be obtained through any preferred method, with this paper mainly using frozen text encoders for the experiments. Moreover, the method can be applied fairly easily to any contrastive objective, with this paper focusing on the InfoNCE objective.

The proposed method is trained on ImageNet, CC3M and CC12M and compared against one supervised learning method and various SSL methodologies. It is shown that $\mathbb{X}$-CLR leads to consistent improvements for classification and background decomposition compared to the baselines, with minimal increase in computation.
weaknesses: In my opinion, there are no significant weaknesses in the paper. The only thing I was wondering regarding the experiments is how fair the comparison between $\mathbb{X}$-CLR and the SSL methods in Table 1 is. It seems to me that $\mathbb{X}$-CLR is a supervised method in this setting. If the authors agree with me, then it might be nice to explicitly mention this in the text to avoid any possible confusion. That being said, I do think including the comparison is important to showcase the effect of proper similarity scoring, so this is a minor remark.

Aside from that, I have some small notes regarding the presentation, which I will list below:
  - It may be useful to shortly mention what $f_\theta$ and $K$ are after equation (1) in line 202 if space is not a concern. It is reasonably clear from the context, but being explicit rarely hurts.
  - In the data  scarcity setting (figure 3a), SupCon appears to be better than $\mathbb{X}$-CLR, while the text mentions that $\mathbb{X}$-CLR improves over SupCon.
  - The coloring of SupCon and SimCLR in figures 3a and 3b is flipped, which is a bit confusing at first glance.
  - The paragraph related to table 4 does not mention training on CC3M, which might be useful to mention for clarity.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Suggestion: Finding a soft target graph without extra metadata or pretrained model with self-supervised techniques would give better applicability.
soundness: 3
strengths: 1. The paper is well-written and easy to read.
2. Experiment study spans different datasets and comparisons to contrastive baselines.
3. The authors test with different similarity graphs (random graph, graph inferred from metadata, augmentation graph, true class graph), and find that a soft graph can improve over the binary graph of existing contrastive approaches.
summary: The paper argue for revising the standard contrastive cross-entropy loss from binary targets of a sparse augmentation graph to a soft target graph where the target values can be between [0, 1]. The new method is called X-CLR. The authors suggest to use extra metadata associated with the dataset to find the soft target distribution, and in this work mostly use the pairwise similarities from a pretrained text encoder on text captions/descriptions to encode this distribution. Although the authors note that the framework is general, "the similarities can come from any source including any metadata". Conduct experiments to train vision models with text captions/descriptions. The study spans different datasets and the results show improvements over contrastive baselines.
weaknesses: 1. Relying on extra metadata and a pretrained model limits the applicability compared to the contrastive baselines. I also think this is a simple solution.
2. The idea of soft-targets for cross-entropy based contrastive loss is very general. It also lack some novelty to the paper's related work on soft-targets.

Meta-review: This submission proposes X-CLR, a contrastive learning objective that incorporates soft similarity labels derived from external metadata or pretrained models, as opposed to traditional binary similarity scores. While the method shows some performance improvements over baseline methods in classification tasks, there are critical concerns raised by reviewers regarding the novelty, fairness of comparisons, and the robustness of the experiments.

The proposed method offers a practical solution to a well-recognized problem in contrastive learning, with demonstrated performance improvements. While some reviewers remained skeptical of novelty, they acknowledged the potential value of the approach for future research.

Decision: Accept (Poster)