RANKCLIP: Ranking-Consistent Language-Image Pretraining

## Abstract

Abstract Self-supervised contrastive learning models, such as CLIP, have set new benchmarks for
vision-language models in many downstream tasks.
However, their dependency on rigid one-to-one mappings overlooks the complex and often
multifaceted relationships between and within texts and images.
To this end, we introduce RankCLIP , a novel pretraining method that extends
beyond the rigid one-to-one matching framework of CLIP and its variants.
By extending the traditional pair-wise loss to list-wise, and leveraging both in-modal and
cross-modal ranking consistency, RankCLIP improves the alignment process, enabling it to
capture the nuanced many-to-many relationships between and within each modality.
Through comprehensive experiments, we demonstrate the effectiveness of RankCLIP in
various downstream tasks, notably achieving significant gains in zero-shot classifications
over state-of-the-art methods, underscoring the importance of this enhanced learning process.
Code and model checkpoints are available at https://github.com/Jam1ezhang/RankCLIP .

## 1 Introduction

In the realm of computer vision (CV) (Voulodimos et al., ),
natural language processing (NLP) (Chowdhary and Chowdhary, ), and multimodal deep
learning (Jabeen et al., ; Zhao et al., ), the alignment between visual and textual
modalities (Singh et al., ; Chen et al., ) has emerged as a cornerstone for downstream
applications, ranging from image captioning (Ghandi et al., ) to zero-shot
classification (Pourpanah et al., ).
Contrastive Language-Image Pretraining (CLIP) (Radford et al., ) marks a
significant advancement in this field, demonstrating incredible performance from training
on large amounts of text-image pairs to create self-supervised models that
understand (Hendrycks et al., , ; Chen et al., )
and generate (Ramesh et al., ; Crowson et al., ) descriptions of visual contents.

Figure: Figure 1: Comparison of learning outcomes between (a) CLIP and (b) RankCLIP using three text-image pairs: dog ( red), cat ( blue), and airplane ( magenta), where matched pairs share the same color boundaries. CLIP treats all unmatched relationships equally, failing to distinguish similarities better between dog and cat versus airplane. RankCLIP addresses this by training with ranking consistency, enhancing its understanding of complex relationships.
Refer to caption: x1.png

Following the success of this contrastive learning paradigm, many recent
works have been developed upon the original CLIP.
More specifically, these enhancements focus on optimizing data efficiency through
intrinsic supervision (Li et al., ), as well as improving downstream performance
via cross-modal late interaction (Yao et al., ), hierarchical feature
alignment (Gao et al., ), geometric consistency
regularization (Goel et al., ), additional learning (Mu et al., ),
adaptive loss (Yang et al., ), hierarchy-aware attentions (Geng et al., ),
and softer cross-modal alignment (Gao et al., ).

Despite the improvements, these methods often have reliance on strict
pairwise, cross-modal, and one-to-one mappings between images and texts,
overlooking the actual many-to-many relationships that exist both cross-modal
and in-modal in real-world data (Chun, ).
For example, as shown in Fig. 1, while pretrained models like CLIP
can correctly classify dog, cat and airplane, they do not
necessarily learn that dog and cat are more close to each other than
dog and airplane, in terms of both in-modal (dog text
is more similar to cat text than to airplane text) and cross-modal
(dog text is more matched to cat image than to airplane image)
similarities.
Because it is rooted from the current contrastive loss that only the correct pairs are
optimized while the rest of the unmatched pairs are treated the same, resulting in a
large amount of information not used and unknown to the model during and after the
training process.

Recognizing the complex many-to-many relationships as well as the rich information
contained within both in-modal and cross-modal data, we introduce
Ranking-Consistent Language-Image Pretraining,
(RankCLIP), which employs ranking consistency to learn and optimize
similarity levels both between (cross-modal) and within (in-modal) the text-image pairs.
The concept of ranking consistency stems from the simple observations that similar
texts often correlate with similar images, as seen with the dog, cat and
airplane example in Fig. 1.
It effectively captures secondary similarity relationships among unmatched pairs, enabling the
model to learn more efficiently for free compared to relying solely on matched pairs.
Ranking consistency is conveniently modeled as an additional loss term to the traditional
contrastive loss, requiring no extra external modules.
It acts as a plug-and-play improvement for many existing methods, including those focusing
on data-efficiency (Li et al., ), potentially boosting performance in both
efficiency and effectiveness.

The main contributions of this paper are: (1) RankCLIP, a novel contrastive
pretraining method that uses ranking consistency to exploit the
many-to-many relationships within data, thereby enhancing performance in
downstream tasks such as zero-shot classification and retrieval accuracy;
and (2) through comprehensive experiments conducted on multiple datasets, we demonstrate
the superior effectiveness of RankCLIP in improving pretraining model performance without
requiring any additional data or computational resources.

## 2 Related Work

Vision-language pretraining has witnessed significant advancements over the past
years (Chen et al., ; Du et al., ; Long et al., ).
Models such as CLIP (Radford et al., ), ALIGN (Jia et al., ) and
FLAVA (Singh et al., ) have pioneered the contrastive learning paradigm applied with
text-image pairs, showcasing remarkable performance and robustness in downstream tasks.
Many follow-up works, mostly built upon CLIP, have been proposed since then.
Li et al. () introduced DeCLIP, improving zero-shot performance through
intrinsic supervision.
FILIP (Yao et al., ) advances CLIP’s alignment between image patches and text with a
cross-modal interaction mechanism.
Gao et al. () developed PyramidCLIP, using hierarchical feature alignment to
boost model efficiency and performance.
Additionally, SLIP (Mu et al., ) merges self-supervised learning with CLIP pre-training
for improved visual representation and accuracy.
Goel et al. () introduced CyCLIP, augmenting CLIP with geometric consistency
regularizers to enhance robustness and performance under varied conditions.

Figure: Figure 2: Illustrative overview of RankCLIP. Unlike conventional contrastive loss, which includes only the middle term, RankCLIP introduces both cross-modal and in-modal consistency terms by minimizing a self-supervised, list-wise ranking loss. Paired images and texts are indicated by matching contour line colors. $V$, $T$, and $S$ represent image embeddings, text embeddings, and similarity scores, respectively.
Refer to caption: x2.png

Recently, Yang et al. () introduced ALIP, an adaptive pre-training model that enhances
language-image alignment using raw text and synthetic captions with dynamic adjustments.
HiCLIP (Geng et al., ) refines CLIP by adding hierarchy-aware attentions to uncover
semantic hierarchies in images and texts.
EqSim (Wang et al., ) incorporates equivariance loss into vision-language
models, significantly improving sensitivity to semantic changes in image-text pairs.
Additionally, SoftCLIP (Gao et al., ) softens CLIP’s one-to-one constraint, enabling
more flexible cross-modal alignment through fine-grained adjustments.

Compared with existing approaches, RankCLIP sets itself apart by fully leveraging the
many-to-many relationships within each batch of text-image pairs, promoting learning
from both matched and unmatched pairs with varying similarities by integrating in-modal and
cross-modal list-wise ranking consistencies into the contrastive training objective.
Crucially, RankCLIP diverges from existing models’ pair-wise training objective by adopting a
global, list-wise optimization approach.
In other words, it considers the rankings of all images and texts collectively within each
batch, rather than focusing on pairwise similarities as seen in other methods.

## 3 RankCLIP

RankCLIP efficiently leverages the many-to-many relationships in real-world data by focusing on
both matched and unmatched pairs.
As shown in Fig. 2, it not only identifies if an image-text pair matches but
also assesses their relative semantic similarities to other images and texts of both
modalities in the dataset through self-supervised ranking consistency.
Uniquely, RankCLIP employs a list-wise loss for training batches, distinguishing it from other
methods that solely rely on pair-wise relationships, as discussed in §2.

### 3.1 Ranking Model Formulation

RankCLIP leverages the Plackett-Luce (PL) ranking
model Plackett (); Luce (); Guiver and Snelson () to estimate the
probability distribution over rankings for every image-text pair $(I_{i},T_{j})$, so that the
consistency in their relative ordering with respect to a reference ranking can be measured.
Specifically, for a given data pair, whether it is in-modal (image-image, text-text), or
cross-modal (image-text), we calculate its in- or cross-modal cosine similarity $S_{ij}$ to
serve as the score when measuring the alignment of its ranking with respect to another
reference ranking $y_{\text{ref}}$.

Following Plackett (), we first sort the reference ranking in a descending
order to construct the optimal ranking $y^{*}$, and assume that the ego ranking $y$ is sampled
from $y^{*}$.
Thus the probability that item $d$ with score $S_{ij}$ is ranked $k^{\text{th}}$ in the ego
ranking $y$ from a set of items $\mathcal{D}$ is the score of $e^{S_{ij}}$ divided by the sum
of scores for the items that have not been placed yet:

$$ $\pi(d\mid y_{1:k-1},y_{\text{ref}},\mathcal{D})=\frac{e^{S_{ij}}}{\sum_{d^{ \prime}\in\mathcal{D}\setminus y_{1:k-1}}e^{S^{\prime}_{ij}}},$ (1) $$

where $y_{1:k-1}=[y_{1},y_{2},...,y_{k-1}]$ denotes the set of items ranked before $d$.
In addition, we propose a decaying factor $\mu=1/\log(k+1)$ to scale the loss, so that the
top-ranked items can obtain higher weights:
Consequently, the probability of the entire ranking $y$ is the product of all individual
placement probabilities:

$$ $\mathcal{P}(y,y_{\text{ref}})=\prod_{k=1}^{K}\mu\cdot\pi(y_{k}\mid y_{1:k-1}, \mathbf{y_{\text{ref}}},\mathcal{D}).$ (2) $$

RankCLIP’s objective is to maximize the consistency log-likelihood of the list ranking in
one modality towards the reference ranking (from the same/in-modal and different/cross-modal
data), which conveniently aligns with minimizing the negative log-likelihood loss:

$$ $\mathcal{L}_{\text{PL}}=-\log\mathcal{P}(y,y_{\text{ref}})$ (3) $$

where $y$ and $y_{\text{ref}}$ can be in either modality.

### 3.2 Cross-modal Consistency Ranking

As illustrated by the green box in Fig. 2, RankCLIP utilizes secondary
relationships between unmatched visual and textual representations by constructing a
list-wise rank loss.
This approach ensures that the semantic similarity rankings between one image and multiple
texts align with those between one corresponding text and multiple images.
For example, as shown in Fig. 1, from the dog perspective,
the semantic distance between dog image and cat text is closer compared to
the plane text.
This relationship should also apply between the dog text and the cat,
plane images.
Mathematically, Eq. (3) can be specified as:

$$ $\displaystyle\mathcal{L}_{\text{cross-modal}}$ $\displaystyle=-\log\mathcal{P}(\bf{y}_{\text{image-text}},\bf{y}_{\text{text- image}})$ (4) $\displaystyle=-\log\mathcal{P}(\bf{\hat{v}}\cdot\bf{\hat{t}}^{T},\bf{\hat{t}} \cdot\bf{\hat{v}}^{T})$ (5) $$

By optimizing Eq. (4), RankCLIP enhances its ability to bridge the
semantic gap between modalities by leveraging nuanced unmatched correlations.
This can also be viewed as learning a symmetric cosine-similarity matrix, further reinforcing
semantic consistency across modalities.

### 3.3 In-modal Consistency Ranking

The pink box in Fig. 2 highlights the in-modal consistency component of the
proposed rank loss.
RankCLIP ensures semantic consistency within each modality – image to image and text to text
– enhancing the use of secondary unmatched relationships as an optimization objective.
The underlying principle is that similar images should correspond to similar texts.
For example, in Fig. 1, from the dog image perspective, the
cat image is the most similar, followed by the plane image.
This relationship should hold true for their corresponding texts as well, where we utilize this
to construct our $y$ and $y_{\text{ref}}$ from Eq. (3).
Mathematically, Eq. (3) can be specified as:

$$ $\displaystyle\mathcal{L}_{\text{in-modal}}$ $\displaystyle=-\log\mathcal{P}(\bf{y}_{\text{text-text}},\bf{y}_{\text{image- image}})$ (6) $\displaystyle=-\log\mathcal{P}(\bf{\hat{t}}\cdot\bf{\hat{t}}^{T},\bf{\hat{v}} \cdot\bf{\hat{v}}^{T})$ (7) $$

where $\bf{\hat{t}}$ and $\bf{\hat{v}}$ are the text and image batch embedding matrix,
respectively.
Via Eq. (6), the model can efficiently leverage the nuanced in-modal
relationships to learn a richer and more structured semantic representation.

### 3.4 RankCLIP Loss

Combining both cross-modal and in-modal consistency with the traditional contrastive loss (more
details in Appendix A), the complete rank loss is thus formulated as:

$$ $\mathcal{L}_{\text{{RankCLIP}}}=\mathcal{L}_{\text{CLIP}}+\lambda_{1}\mathcal{ L}_{\text{in-modal}}+\lambda_{2}\mathcal{L}_{\text{cross-modal}}$ (8) $$

which is also depicted in Fig. 2.
By supplementing the pairwise contrastive loss with cross-modal and in-modality ranking
consistency loss, RankCLIP systematically organizes embeddings to fully exploit both global
and fine-grained secondary unmatched relationships, which enhances the learning of more
informative and accurate representations, better supporting downstream multi-modal tasks.

**Table 1: Zero-shot top-1, top-3 and top-5 classification accuracy on CIFAR-10, CIFAR-100 and ImageNet1K. Relative to CLIP, RankCLIP achieves higher accuracy with average top-1, top-3, and top-5 improvements of +18.95%, +12.5%, and +9.73%, respectively. RankCLIP also outperforms ALIP consistently across the datasets.**
|  | CIFAR-10 | CIFAR-100 | ImageNet1K |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Top1 | Top3 | Top5 | Top1 | Top3 | Top5 | Top1 | Top3 | Top5 |
| CLIP | 36.35% | 70.28% | 85.02% | 12.22% | 24.93% | 33.56% | 12.08% | 21.86% | 27.48% |
| ALIP | 35.71% | 72.39% | 88.77% | 13.67% | 27.10% | 34.76% | 15.62% | 26.90% | 32.50% |
| RankCLIP | 37.03%<br>(+1.87%) | 67.67%<br>(-3.71%) | 83.09%<br>(-2.27%) | 13.98%<br>(+14.40%) | 27.70%<br>(+11.11%) | 36.17%<br>(+7.78%) | 17.02%<br>(+40.89%) | 28.44%<br>(+30.10%) | 33.99%<br>(+23.69%) |

**Table 2: Zero-shot top-1, top-3 and top-5 classification accuracy on variants of ImageNet1K that have natural distribution shifts. Relative to CLIP, RankCLIP achieves higher accuracy with average top-1, top-3, and top-5 improvements of +45.55%, +30.24%, and +25.83%, respectively. Notice that the average improvements are more significant than when tested on ImageNet1K without distribution shift, indicating better robustness.**
|  | ImageNetV2 | ImageNetSketch | ImageNet-A | ImageNet-R |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Top1 | Top3 | Top5 | Top1 | Top3 | Top5 | Top1 | Top3 | Top5 | Top1 | Top3 | Top5 |
| CLIP | 12.11% | 22.66% | 28.57% | 3.20% | 7.00% | 9.83% | 3.16% | 8.81% | 13.04% | 11.34% | 21.38% | 27.10% |
| ALIP | 15.62% | 27.34% | 32.82% | 5.10% | 10.37% | 14.01% | 3.53% | 9.14% | 13.61% | 14.25% | 25.74% | 32.43% |
| RankCLIP | 17.03%<br>(+40.63%) | 28.60%<br>(+26.21%) | 34.18%<br>(+19.64%) | 5.82%<br>(+81.88%) | 11.35%<br>(+62.14%) | 14.87%<br>(+51.27%) | 3.82%<br>(+20.89%) | 9.16%<br>(+3.97%) | 13.77%<br>(+5.60%) | 15.74%<br>(+38.80%) | 27.51%<br>(+28.67%) | 34.36%<br>(+26.79%) |

## 4 Experiments

### 4.1 Experimental Setup

#### Baselines.

The most direct baseline to RankCLIP is the original CLIP (Radford et al., ).
In addition, to further demonstrate the superior performance of RankCLIP, we include
ALIP (Yang et al., ), which leverages synthetic captions to enhance vision-language
representation learning.
More specifically, it employs a unique architecture that dynamically adjusts sample and
pair weights to mitigate the impact of noisy or irrelevant data, which is quite orthogonal
to our approach.
The training procedures and parameters of all models are detailed in
Appendix B.

#### Pretraining dataset.

Both baseline models, CLIP (Radford et al., ), ALIP (Yang et al., ) and the
proposed RankCLIP are pretrained on the Conceptual Captions 3M (CC3M)
dataset (Sharma et al., ), which contains around 3.3 million text-image pairs.
Despite being much smaller than CLIP’s initial dataset (Ilharco et al., ),
CC3M effectively supports the development of pretrained models with strong zero-shot
capabilities and is widely used in existing language-image pretraining
research (Carlini and Terzis, ; Li et al., ; Tejankar et al., ; Mu et al., ; Goel et al., ).

### 4.2 Zero-shot Classification

Zero-shot capability is one of the most significant improvements that CLIP achieves.
Thus in this section, we first evaluate the zero-shot classification performance of
CLIP, ALIP and the proposed RankCLIP.
Following (Goel et al., ), we conduct our experiments on
CIFAR-10 (Krizhevsky et al., ),
CIFAR-100 (Krizhevsky et al., ), and
ImageNet1K (Deng et al., ; Russakovsky et al., ) dataset.

As shown in Table 1, RankCLIP consistently outperforms CLIP across
CIFAR-10, CIFAR-100, and ImageNet1K datasets.
Relative to CLIP, RankCLIP shows average improvements of +18.95%, +12.5%,
and +9.73% in top-1, top-3 and top-5 metrics, respectively.
Particularly on the more challenging ImageNet1K dataset, RankCLIP improves relative top-1
accuracy by +40.89% over CLIP, highlighting the effectiveness of the proposed
ranking consistency in enhancing language-image alignment and understanding with the same
amount of training data.
The two cases where RankCLIP does not excel are the top-3 and top-5 accuracy on CIFAR-10.
However, this is likely because CIFAR-10 with top-3 and top-5 metrics is much simpler,
reducing the demand for a deeper model understanding.

Additionally, we observe that RankCLIP consistently outperforms ALIP, suggesting that our
ranking consistency more effectively enhances text-image representations and alignments
compared to the synthetic captions proposed in ALIP.
Another trend we observe is that RankCLIP shows the most significant improvement in top-1
accuracy compared to top-3 and top-5.
Considering the real-world emphasis on the topmost model output, RankCLIP is likely to offer
considerable advantages in practical applications.

**Table 3: Linear probing top-1 accuracy on 11 downstream datasets. RankCLIP achieves higher accuracy than CLIP with an average improvement of +2.31%. RankCLIP also outperforms ALIP, although less significantly.**
|  | CIFAR-10 | CIFAR-100 | DTD | FGVGAircraft | Food101 | GTSRB | Imagenet1K | OxfordPets | SST2 | STL10 | SVHN | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP | 72.40% | 48.43% | 49.89% | 26.10% | 48.59% | 65.20% | 77.49% | 49.74% | 53.71% | 83.59% | 44.80% | 56.37% |
| ALIP | 73.87% | 51.00% | 58.09% | 27.72% | 49.74% | 60.34% | 73.14% | 59.36% | 53.98% | 87.94% | 38.07% | 57.56% |
| RankCLIP | 72.54%<br>(+0.20%) | 49.16%<br>(+1.50%) | 53.24%<br>(+6.71%) | 24.99%<br>(-4.25%) | 47.11%<br>(-3.05%) | 63.37%<br>(-2.81%) | 86.40%<br>(+11.50%) | 54.10%<br>(+8.77%) | 54.09%<br>(+3.00%) | 86.10%<br>(+0.71%) | 43.30%<br>(-3.35%) | 57.67%<br>(+2.31%) |

**Table 4: Zero-shot image and text retrievals on Flickr30K and MSCOCO. RankCLIP achieves higher accuracy than both CLIP and ALIP on most cases.**
|  | Flickr30K | MSCOCO |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Text Retrieval | Image Retrieval | Text Retrieval | Image Retrieval |  |  |  |  |  |  |  |  |
|  | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R5 | R@10 |
| CLIP | 84.00% | 88.70% | 91.0% | 8.70% | 16.90% | 21.20% | 82.06% | 85.24% | 87.82% | 5.04% | 12.98% | 18.32% |
| ALIP | 84.40% | 90.00% | 92.50% | 9.40% | 17.60% | 21.30% | 82.56% | 86.04% | 88.26% | 6.08% | 13.96% | 19.38% |
| RankCLIP | 84.10%<br>(+0.12%) | 89.40%<br>(+0.79%) | 91.90%<br>(+0.99%) | 8.10%<br>(-6.90%) | 16.40%<br>(-2.96%) | 21.70%<br>(+2.36%) | 82.90%<br>(+1.02%) | 85.68%<br>(+0.52%) | 88.00%<br>(+0.20%) | 5.60%<br>(+11.11%) | 13.20%<br>(+1.69%) | 18.02%<br>(-1.64%) |

### 4.3 Robustness to Distribution Shifts

To evaluate the robustness of RankCLIP under distribution shifts, we test it alongside
CLIP and ALIP across four ImageNet variants, including ImageNetV2 (Recht et al., ),
ImageNetSketch (Wang et al., ), ImageNet-A (Hendrycks et al., ),
and ImageNet-R (Hendrycks et al., ), which are designed to assess resilience to different
distribution shifts.

As shown in Table 2, RankCLIP outperforms both CLIP and ALIP
consistently.
Notably, relative to CLIP, RankCLIP’s accuracy improvements in shifted conditions
are +45.55% (top-1), +30.24% (top-3), and +25.83%
(top-5), surpassing its performance in standard settings (Table 1) of
+40.89% (top-1), +30.1% (top-3), and +23.69%
(top-5), indicating the even more superior performance in robustness under distribution shifts.

### 4.4 Linear Probing

We also evaluate whether the introduced ranking consistency retains its advantages when
supplemented with additional in-domain supervision.
Specifically, we use linear probing, where the pretrained encoders from CLIP, ALIP, and
RankCLIP remain unchanged while a logistic regression classifier is trained on domain-specific
datasets.
We evaluate on a suite of 11 standard image classification datasets as our in-domain datasets,
which include CIFAR-10, CIFAR-100,
Describable Textures Dataset (DTD) (Cimpoi et al., ),
Fine-Grained Visual Classification of Aircraft
(FGVG-Aircraft) (Maji et al., ), Food101 (Bossard et al., ),
German Traffic Sign Detection Benchmark (GTSDB) (Stallkamp et al., ),
ImageNet1K (Deng et al., ; Russakovsky et al., ),
OxfordPets (Parkhi et al., ),
Stanford Sentiment Treebank v2 (SST2) (Socher et al., ),
STL-10 (Coates et al., ),
and Street View House Numbers (SVHN) (Netzer et al., ) dataset.

Table 3 indicates that RankCLIP consistently outperforms CLIP, with
relative improvements ranging from +0.2% to +11.5% and an average
accuracy increase of +2.31%.
When compared to ALIP, RankCLIP also shows better performance on average, though the gains are
relatively modest.

### 4.5 Zero-shot Image-text Retrieval

In the final part of our experiments, we assess RankCLIP on zero-shot cross-modal retrieval
tasks (image-to-text and text-to-image) using the Flickr30k (Plummer et al., ) and
MSCOCO (Lin et al., ) datasets.
As shown in Table 4, RankCLIP generally outperforms the two
baseline methods, though improvements are less significant compared to earlier results in
Table 1, 2 and
3.
The relatively modest gains in retrieval tasks may stem from the complex requirements of
discerning image-text similarities across varying resolutions and object details,
a significant departure from the simpler demands of image classification tasks.
Despite this, the overall improvement highlights RankCLIP’s advantage, thanks to the
deeper insights provided by ranking consistency in the language-image training process.

## 5 Ablation Studies

**Table 5: Ablation zero-shot classification accuracy of cross-modal-only model $\textsc{RankCLIP}_{C}$ and in-modal-only model $\textsc{RankCLIP}_{I}$ on CIFAR-10, CIFAR-100 and ImageNet1K datasets. Bold indicates the best performance, while blue indicates the second best.**
|  | CIFAR-10 | CIFAR-100 | ImageNet1K |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Top1 | Top3 | Top5 | Top1 | Top3 | Top5 | Top1 | Top3 | Top5 |
| CLIP | 36.35% | 70.28% | 85.02% | 12.22% | 24.93% | 33.56% | 12.08% | 21.86% | 27.48% |
| RankCLIP | 37.03% | 67.67% | 83.09% | 13.98% | 27.70% | 36.17% | 17.02% | 28.44% | 33.99% |
| $\textsc{RankCLIP}_{I}$ | 37.47% | 69.89% | 84.53% | 13.89% | 27.34% | 35.90% | 16.66% | 27.63% | 33.15% |
| $\textsc{RankCLIP}_{C}$ | 28.26% | 59.65% | 75.45% | 13.29% | 26.85% | 34.71% | 16.98% | 28.25% | 33.90% |

### 5.1 Ablation on Loss Components

To further assess the effectiveness of the proposed ranking consistency, we developed two
variants of RankCLIP: $\textsc{RankCLIP}_{C}$, focusing solely on cross-modal consistency
with $\lambda_{i}=0$, and $\textsc{RankCLIP}_{I}$, emphasizing in-modal consistency with
$\lambda_{c}=0$.
Both models underwent the same pretraining as outlined in Appendix B
and were tested in a zero-shot classification experiment on ImageNet1K as in
§4.2.
The results are shown in Table 5, with bold font indicating the best
performance, and blue color representing the second best results.
We can see that, while RankCLIP achieves the best performance, both $\textsc{RankCLIP}_{C}$ and
$\textsc{RankCLIP}_{I}$ demonstrate notable improvements over CLIP.
Interestingly, $\textsc{RankCLIP}_{I}$ matches the performance of $\textsc{RankCLIP}_{C}$,
highlighting the often-underestimated value of in-modal consistency in enhancing model
effectiveness.

Figure: Figure 3: Ablation studies of CLIP and RankCLIP trained with different data sizes. Left: zero-shot top-1 classification accuracy on ImageNet1K with various data sizes randomly sampled from CC3M. RankCLIP consistently outperforms CLIP with significant margins. Right: zero-shot top-1 classification accuracy on ImageNet1K (horizontal axis) and ImageNet1K-R (vertical axis). RankCLIP demonstrates better robustness as well as accuracy.
Refer to caption: x3.png

### 5.2 Ablation on Data Sizes

To evaluate the scalability of RankCLIP, we trained both CLIP and RankCLIP using 500K, 750K,
1M, and 3M text-image pairs from the CC3M dataset following the same procedures in
§B.
Fig. 3 left presents the zero-shot top-1 classification accuracy
on ImageNet1K, where RankCLIP consistently outperforms CLIP.
Notably, it shows a greater performance increase as dataset size grows from 1M to 3M pairs,
suggesting RankCLIP’s superior scalability, a critical attribute for language-image pretraining.
Furthermore, Fig. 3 right illustrates RankCLIP’s robustness across
different dataset sizes.
The horizontal axis shows the top-1 accuracy on standard ImageNet1K, and the vertical axis on
ImageNet1K-R, with a black diagonal line ($y=x$) representing ideal robustness.
Any deviation below this line indicates reduced robustness.
RankCLIP consistently stays well above both the red baseline, which reflects typical
in-distribution to out-of-distribution generalization (Miller et al., ),
and close to the ideal line, demonstrating exceptional robustness to distribution shifts.

## 6 Analysis

### 6.1 Modality Gap

In this section, we analyze the modality gaps of CLIP and our proposed RankCLIP by
visualizing 250 text-image pair embeddings, reduced to two dimensions using
UMAP (McInnes et al., ), and complement this with a histogram of the gaps.
Modality gap (Liang et al., ) refers to a geometric phenomenon observed in the
representation spaces of multimodal models, where different data modalities
(like images and texts) are embedded at a noticeable distance from each other,
rather than being uniformly distributed as ideally expected.
This gap, inherent from initialization and preserved during the contrastive learning process
like in CLIP, poses a challenge in language-image pretraining by impacting joint data modeling
and understanding.
Recent studies (Srivastava and Sharma, ; Kumar and Marttinen, ; Oh et al., ) suggest that
reducing this gap could enhance multimodal representations and downstream task performance.
The results shown in Fig. 4 indicate that RankCLIP exhibits a significantly
smaller modality gap than CLIP, demonstrating that our ranking consistency approach effectively
enhances understanding of text-image semantics.

Figure: Figure 4: Scatter and histograms plots illustrating modality gaps of (a) CLIP and (b) RankCLIP.
Refer to caption: x4.png

### 6.2 Alignment and Uniformity

Besides alleviating modality gap, it is also commonly believed that a successful contrastive
learning method should as well ensure a broad and uniform distribution
covering an hypersphere in space (Wang and Isola, ).
These two goals, characterized as similarity and uniformity, can be assessed with alignment
and uniformity scores, respectively.
More specifically, following Goel et al. () and notations defined in
§3, we calculate the alignment score $S_{\text{A}}$, and uniformity
score $S_{\text{U}}$ to be:

$$ $\displaystyle S_{\text{A}}$ $\displaystyle=\frac{1}{N}\sum_{j=1}^{N}\hat{I}^{T}_{j}\hat{T}_{j},$ (9) $\displaystyle S_{\text{U}}$ $\displaystyle=\log\left(\frac{1}{N(N-1)}\sum_{j-1}^{N}\sum_{k=1,j\neq k}^{N} \exp^{-\hat{I}^{T}_{j}\hat{T}_{k}}\right)$ (10) $$

where $N$ is the total number of text-image pairs.
Essentially, $S_{\text{A}}$ represents the averaged cosine similarity between text and image
embeddings, and $S_{\text{U}}$ averages the dissimilarity measures (exponentiated negative dot
products) between all unique pairs of text-image embeddings in the dataset, quantifying how
evenly these embeddings are distributed.

A high alignment score represents a strong correlation or similarity between pairs of
text-image embeddings, indicating that the images and their textual descriptions are closely
aligned in the embedding space.
Conversely, a high uniformity score suggests that embeddings are not uniformly distributed;
they may be clustering together or not utilizing the embedding space efficiently, which can
indicate redundancy in the representations or a lack of diversity.
A low uniformity score, on the other hand, suggests that the embeddings are well spread out
across the space, indicating a diverse and efficient use of the embedding space, which is
generally desirable for tasks like retrieval, where a wide coverage of possible queries are
preferred.

As shown in Table 6, we observe that, although CLIP
learns representations that are better aligned, as evidenced by its top-ranking alignment
scores, these representations fail to achieve uniform distribution across the hypersphere,
as highlighted by its significantly higher absolute uniformity scores.
On the other hand, RankCLIP, along with two of its ablated version, $\textsc{RankCLIP}_{I}$
and $\textsc{RankCLIP}_{C}$, presents much better balance between alignment and uniformity,
which results in improved downstream task performance as illustrated in previous experiments
as well as in the representative ZS-Top1 results in Table 6.
We also find the results to be informative on a higher level where it indicates that optimizing
contrastive learning towards single objective such as alignment or uniformity would not
intuitively result in higher downstream task performance.

**Table 6: Alignment and uniformity scores of CLIP, RankCLIP, and its two ablated variants.**
|  | CIFAR-10 | CIFAR-100 | ImageNet1K |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | $S_{\text{A}}$ | $S_{\text{U}}$ | ZS-Top1 | $S_{\text{A}}$ | $S_{\text{U}}$ | ZS-Top1 | $S_{\text{A}}$ | $S_{\text{U}}$ | ZS-Top1 |
| CLIP | 0.40 | -0.35 | 36.35% | 0.42 | -0.35 | 12.22% | 0.44 | -0.29 | 12.08% |
| RankCLIP | 0.23 | -0.17 | 37.03% | 0.26 | -0.16 | 13.98% | 0.33 | -0.11 | 17.02% |
| $\textsc{RankCLIP}_{I}$ | 0.24 | -0.16 | 37.47% | 0.26 | -0.15 | 13.89% | 0.32 | -0.10 | 16.66% |
| $\textsc{RankCLIP}_{C}$ | 0.18 | -0.12 | 28.26% | 0.18 | -0.10 | 13.29% | 0.26 | -0.09 | 16.98% |

## 7 Conclusion

In this paper, we introduce RankCLIP, a novel language-image pretraining method that integrates
ranking consistency into the contrastive learning paradigm.
RankCLIP aims to better understand the complex many-to-many relationships in diverse text-image
pairs by optimizing a self-supervised, list-wise rank loss.
Through extensive experiments, including zero-shot classification, robustness to distribution
shifts, linear probing, and zero-shot image-text retrieval, RankCLIP not only enhances
performance but also improves model robustness and semantic comprehension, outperforming the
baseline CLIP and another state-of-the-art model ALIP.
Our ablation studies and analyses further demonstrate and interpret the significance of each
component of RankCLIP in boosting performance and understanding across modalities.
We believe that the methodologies and principles of RankCLIP will inspire further research and
lead to the development of models with a deeper understanding of the intricate interactions
between visual and textual data.

## Limitations

This work has two main limitations.
First, due to computational constraints, we were not able to perform more baseline comparisons
by pretraining additional existing methods.
And second, we were unable to scale the pretraining process to include larger datasets,
such as those used in OpenCLIP (Ilharco et al., ).

## Appendix

## Appendix A CLIP Preliminaries

CLIP (Radford et al., ) has been a prominent method for learning detailed multimodal
representations through the alignment of images and texts.
Given a set $\mathcal{D}=\{(I_{j},T_{j})\}_{j=1}^{N}$ of $N$ image-text pairs, where
$I_{j}$ denotes an image and $T_{j}$ is the corresponding text, the goal is to learn
representations that map semantically similar images and texts closer in the embedding space,
while dissimilar pairs are distanced apart.
More specifically, the foundational CLIP model employs two encoders: an image encoder
$f_{I}:\mathcal{I}\rightarrow\mathbb{R}^{m}$ that processes raw images into visual embeddings
and a text encoder $f_{T}:\mathcal{T}\rightarrow\mathbb{R}^{n}$ which encodes textual data
into text embeddings.
Then both the text and visual features are projected to a latent space with identical dimension.
Formally, the embeddings for a text-image pair $(I_{j},T_{j})$ are denoted as $v_{k}=f_{I}(I_{j})$
and $t_{j}=f_{T}(T_{j})$, respectively.
The embeddings are then normalized to lie on an unit hypersphere by enforcing
$l_{2}$-norm constraint:

$$ $\hat{v}_{j}=\frac{v_{j}}{\|v_{j}\|_{2}},\quad\hat{t}_{j}=\frac{t_{j}}{\|t_{j} \|_{2}}.$ (11) $$

so that the magnitude information is erased and only direction is preserved.

To align the image and text representations, a contrastive loss function, typically a
variant of the InfoNCE loss (Oord et al., ), which optimizes the similarity of the matched pair against unmatched pairs, is utilized, i.e.:

$$ $\displaystyle\mathcal{L}_{\text{CLIP}}=-\frac{1}{2N}\sum_{j=1}^{N}\biggr{[} \underbrace{\log\frac{\exp(\hat{v}_{j}^{\top}\hat{t}_{j}/\tau)}{\sum_{k=1}^{N} \exp(\hat{v}_{j}^{\top}\hat{t}_{k}/\tau)}}_{{\color[rgb]{1,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{1,0,0}\leavevmode\hbox to11.75pt{\vbox to11.75pt{ \pgfpicture\makeatletter\hbox{\hskip 5.87407pt\lower-5.87407pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{ 1,0,0}\pgfsys@color@rgb@stroke{1}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill {1}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ } \nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{{}}}{{}}{}{}{}{}{}{}{ }{}{}{{}\pgfsys@moveto{5.67407pt}{0.0pt}\pgfsys@curveto{5.67407pt}{3.13374pt}{ 3.13374pt}{5.67407pt}{0.0pt}{5.67407pt}\pgfsys@curveto{-3.13374pt}{5.67407pt}{ -5.67407pt}{3.13374pt}{-5.67407pt}{0.0pt}\pgfsys@curveto{-5.67407pt}{-3.13374 pt}{-3.13374pt}{-5.67407pt}{0.0pt}{-5.67407pt}\pgfsys@curveto{3.13374pt}{-5.67 407pt}{5.67407pt}{-3.13374pt}{5.67407pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto {0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{ } }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-1.75pt}{-2.25555pt}\pgfsys@invoke{ }\hbox{{\definecolor[named]{ pgfstrokecolor}{rgb}{1,0,0}\pgfsys@color@rgb@stroke{1}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{1}{0}{0}\pgfsys@invoke{ }\hbox{{1}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{{}}}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}}}+$ (12) $\displaystyle\underbrace{\log\frac{\exp(\hat{t}_{j}^{\top}\hat{v}_{j}/\tau)}{ \sum_{k=1}^{N}\exp(\hat{t}_{j}^{\top}\hat{v}_{k}/\tau)}}_{{\color[rgb]{0,0,1} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\leavevmode\hbox to11.75pt{ \vbox to11.75pt{\pgfpicture\makeatletter\hbox{\hskip 5.87407pt\lower-5.87407pt \hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,1}\pgfsys@color@rgb@stroke{0}{0}{1}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{1}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{ {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{{}}}{{}}{}{}{}{}{}{}{ }{}{}{{}\pgfsys@moveto{5.67407pt}{0.0pt}\pgfsys@curveto{5.67407pt}{3.13374pt}{ 3.13374pt}{5.67407pt}{0.0pt}{5.67407pt}\pgfsys@curveto{-3.13374pt}{5.67407pt}{ -5.67407pt}{3.13374pt}{-5.67407pt}{0.0pt}\pgfsys@curveto{-5.67407pt}{-3.13374 pt}{-3.13374pt}{-5.67407pt}{0.0pt}{-5.67407pt}\pgfsys@curveto{3.13374pt}{-5.67 407pt}{5.67407pt}{-3.13374pt}{5.67407pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto {0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{ } }{{{{}}\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1 .0}{-1.75pt}{-2.25555pt}\pgfsys@invoke{ }\hbox{{\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,1}\pgfsys@color@rgb@stroke{0}{0}{1}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{1}\pgfsys@invoke{ }\hbox{{2}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{{{}}}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}}}\biggr{]}$ (13) $$

where the first term 1 contrasts images with the texts,
the second term 2 contrasts texts with the images,
and $\tau$ denotes a temperature scaling parameter that adjusts the concentration
of the distribution.
The optimization of Eqn. (13) results in embeddings where the cosine similarity
between matched image-text pairs is maximized in comparison to unmatched pairs, thus
achieving the desired alignment in the joint embedding space.

Despite the efficacy of CLIP in learning correlated multimodal embeddings, it inherently
relies on strict pairwise matched comparisons and fails to capture the more complex,
fine-grained nature of semantic similarity within and across modalities that are
generally treated as unmatched.
This observation motivates the development of RankCLIP, which innovates beyond binary pairwise
contrasts to consider holistic listwise consistency within and across modalities.

## Appendix B Training Procedures

### B.1 Implementation Details

For CLIP (Radford et al., ), we use the official implementation released by
OpenAI(^1^11CLIP repository on GitHub: https://github.com/openai/CLIP.).
And for ALIP (Yang et al., ), we also use the official implementation released by
the paper authors(^2^22ALIP repository on GitHub: https://github.com/deepglint/ALIP.).
As the proposed RankCLIP essentially shares the same model architecture (separate vision, text
encoders, projection layer, and a classification head) as CLIP, we build upon the CLIP code
repository for our model construction(^3^33RankCLIP repository will be released upon acceptance.).
We set the scaling parameters for cross-modal ($\lambda_{c}$) and in-modal ($\lambda_{i}$)
ranking consistency to 1/16 and 1/16 respectively throughout all the experiments unless
otherwise noted.
All CLIP, ALIP and RankCLIP models are initialized from scratch without loading any
existing weights.
And the embedding sizes for both modalities all project to 1024 across the three models.

### B.2 Training Parameters

Following CLIP (Radford et al., ), we adopt the
ResNet-50 (He et al., ) and transformer architectures (Devlin et al., ) for image
and text encoding, respectively.
Training is conducted from scratch over 64 epochs using a single NVIDIA A100 GPU, with a batch
size of 512, an initial learning rate of 0.0005 employing cosine scheduling, and 10,000
warm-up steps.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please refer to the weaknesses.
soundness: 2
strengths: - Reasonable Motivation: The paper identifies the limitations of existing models like CLIP, discussing the importance of capturing many-to-many relationships in multimodal data, which provides a reasonable motivation for the development of RANKCLIP.

- Extensive Evaluation: RANKCLIP is evaluated across a variety of downstream tasks, including zero-shot image classification, retrieval, and linear probe classification, showing its applicability in different contexts.
summary: The paper presents RANKCLIP, a vision-language pretraining method that enhances the alignment between visual and textual modalities. It shifts from traditional pair-wise loss to a list-wise approach, allowing the model to capture many-to-many relationships. RANKCLIP introduces a ranking consistency mechanism to optimize similarity levels within and across modalities. This approach helps the model learn nuanced relationships, such as the closeness between similar images and texts. By leveraging secondary similarities among unmatched pairs, RANKCLIP improves learning efficiency without needing extra data or resources. Comprehensive experiments demostrates the effectiveness of the proposed method in downstream tasks, particularly in zero-shot classification and retrieval accuracy.
weaknesses: - Lack of Discussion on Related Works: The paper does not adequately discuss other works that also aim to construct many-to-many relationships in vision-language pretraining. For example, [1] proposed a progressive self-distillation method that uses image-to-text logits (and vice versa) as targets, while [2] introduced in-modal consistency.

- Lack of Novelty: RANKCLIP closely resembles the method described in [1], raising questions about its novelty.

- Misaligned Experiment Settings: The experimental setup is misaligned, making the results less convincing. While many CLIP-related works utilize the ViT-B/32 architecture as the vision backbone, RANKCLIP employs RN50, which could affect the comparability of the results.

- Performance Downgrade in Linear Probe Classification: The proposed method underperforms in linear probe classification on fine-grained datasets, such as GVGAircraft, Food101, and GTSRB. The paper does not address this phenomenon, which limits the interpretation of its effectiveness.

- Unconvincing Results in Zero-Shot Text/Image Retrieval: There is a substantial disparity between the results of image retrieval and text retrieval (84.1% vs. 8.1%), which raises doubts about the reliability of these findings.


[1] Andonian, Alex, Shixing Chen, and Raffay Hamid. "Robust cross-modal representation learning with progressive self-distillation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Goel, Shashank, et al. "Cyclip: Cyclic contrastive language-image pretraining." Advances in Neural Information Processing Systems 35 (2022): 6704-6719.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: see weakness.
soundness: 2
strengths: 1.	RANKCLIP is designed to handle the tricky many-to-many relationships between images and text. Instead of just looking at pairs in isolation, it uses a ranking approach to enhance model performance.
2.	When testing against data that’s a little different than what it was trained on, RANKCLIP still holds up well. It also has a knack for understanding semantic nuances, making it better at tasks like image-text retrieval.
summary: RANKCLIP is a language-image pre-training method that incorporates ranking consistency into contrastive learning to enhance model performance. It seeks to better understand complex many-to-many relationships between diverse text-image pairs by optimizing a self-supervised ranking loss. Extensive experiments show that RANKCLIP improves performance, robustness, and semantic understanding across tasks like zero-shot classification and image-text retrieval, outperforming existing models such as CLIP and ALIP.
weaknesses: 1.	Although it performs well on variants of ImageNet1K with natural distribution shifts, its top-3 and top-5 accuracy on CIFAR-10 is even lower than that of CLIP. 
2.	The comparison includes too few SOTA methods; additional methods such as CyCLIP and SoftCLIP should be included to convincingly demonstrate the superiority of the proposed method.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please see the weaknesses section for questions regarding the complementarity between rank consistency and the CLIP objective.

Importantly, I would like to see the authors provide a comparison to more baseline methods for advancing CLIP-style cross-modal learning. I am eager to raise my rating if this comparison can be presented.
soundness: 3
strengths: - **Novelty of Rank Consistency Objective**: The rank consistency objective appears to be novel, as existing methods like the CLIP objective rely on an instance discrimination task focused on distinguishing positive examples from negatives. While embedding-level correlations between similar samples are known to emerge naturally from this approach, no method to date has directly leveraged this correlation. The proposed rank consistency loss effectively utilizes this inherent similarity, potentially improving the overall quality of the learned embeddings.

- **Improvement and Scalability in Embedding Quality**: Consistent improvements in embedding quality are observed across various benchmarks. Notably, the gains become more pronounced as training dataset sizes increase (e.g., from CIFAR to ImageNet to YFCC), indicating promising scalability with larger datasets.
summary: This paper introduces **RANKCLIP**, a method for training visual-text embedding models that enforces consistent similarity ranking within and between modalities. The objective comprises two components:

1. **Cross-Modal Rank Consistency**: This ensures that the similarity ranking of text instances to an image sample aligns with the ranking of corresponding images to the text sample.
  
2. **In-Modal Rank Consistency**: This component maintains that rankings are consistent within each modality, such that text-to-text and image-to-image similarity searches yield comparable ranks.

These two ranking objectives are combined with the traditional CLIP objective to improve the learning of visual-language embeddings.

Experimental results on zero-shot, linear probing, and text-to-image search tasks demonstrate that the proposed objectives enhance embedding quality, yielding more accurate and consistentrepresentations.
weaknesses: - **Integration with the Original CLIP Objective**: While the method improves experimental results, further analysis could clarify how the proposed rank consistency objective interacts with the original CLIP objective. For instance, it would be helpful to understand the balance between the two objectives, or if the rank consistency objective alone could effectively learn cross-modal alignment embeddings. This discussion is currently lacking.

- **Limited Ablation Study on Loss Components**: The ablation study on the loss components appears insufficient. Table 5 shows that cross-modal consistency alone performs close to the combined objectives, suggesting that in-modal consistency may have limited impact. This raises questions about whether in-modal consistency is essential, or if the CLIP objective could also benefit from in-modality instance discrimination. 

- **Limited comparison with other CLIP modifications**: The paper compares with ALIP and CLIP in experiments. However, there are other recent works on improving the CLIP objective, such as SigLIP,. It is a bit difficult to justify the significance of the quality improvement introduced by the method without comparison with the recent methods and analysis of the comparison results. 

Overall, the complementarity between rank consistency and the CLIP objective could be explored further, as it may offer insights beyond the experimental improvements presented. Stopping at the observed gains may overlook a deeper understanding of the method’s theoretical and practical value.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the weaknesses.
soundness: 2
strengths: - Overall this paper is well-written and is easy to understand.
- RANKCLIP achieves significant gains in zero-shot classifications over state-of-the-art methods in various downstream tasks.
summary: This work introduce RANKCLIP, a novel pretraining method that extends beyond the rigid one-to-one matching framework of CLIP and its variants. By extending the traditional pair-wise loss to list-wise, and leveraging both in-modal and cross-modal ranking consistency, RANKCLIP improves the alignment process, enabling it to capture the nuanced many-to-many relationships between and within each modality.
weaknesses: - There is a lack of discussion and citation of some related works [A][B], which also propose new alignment objectives for efficient vision-language pre-training. The author should discuss them in the main table results or the related work.

     [A] SaCo Loss: Sample-wise Affinity Consistency for Vision-Language  Pre-training
     [B] ProtoCLIP: Prototypical Contrastive Language Image Pretraining


- The authors demonstrated the effectiveness of the framework on limited image encoder (e.g., ResNet50). In order to verify the generalization ability, the authors should conduct sufficient experimental comparisons on more backbone networks.


- This work resorts to the self-supervised ranking consistency for learning relative semantic similarities. However, without manual labeling, the reference ranking may be noisy and cause the construction of the optimal ranking to be unreliable. As a result, the derived objective does not necessarily learn the relative semantic similarity as the authors mention in the introduction.

Meta-review: This paper introduces RANKCLIP, a method designed to capture many-to-many relationships in both in-modal and cross-modal comparisons. It addresses the limitations of existing contrastive learning methods, such as CLIP, which rely solely on pairwise matching. Experiments demonstrate improved performance compared to baseline methods like CLIP, ALIP, and FLIP.

However, as noted by the reviewers, the approach shares similarities with CyCLIP and ProtoCLIP, raising concerns about its novelty. Additionally, the paper lacks direct comparisons to these methods. Furthermore, the reliance on self-supervised ranking consistency without manual labeling may introduce noise, potentially undermining the effectiveness of the semantic similarity learning process.

Four reviewers provided negative feedback on this work, and the AC concurs with their assessment. The method holds potential, and with more comprehensive comparisons, ablations, and clarity on contributions, it could make a stronger impact in future submissions.

Decision: Reject