Ranking-aware adapter for text-driven image ordering with CLIP
Authors: Wei-Hsiang Yu
Yen-Yu Lin
Ming-Hsuan Yang
Yi-Hsuan Tsai
Venue: ICLR 2025 Poster (2025)
Keywords: Vision Language Models
CLIP
Learning-to-Rank
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: A lightweight CLIP-based ranking adapter with novel relational ranking-aware attention to for text-driven image ordering.

Abstract: Recent advances in vision-language models (VLMs) have made significant progress in downstream tasks that require quantitative concepts such as facial age estimation and image quality assessment, enabling VLMs to explore applications like image ranking and retrieval. However, existing studies typically focus on the reasoning based on a single image and heavily depend on text prompting, limiting their ability to learn comprehensive understanding from multiple images. To address this, we propose an effective yet efficient approach that reframes the CLIP model into a learning-to-rank task and introduces a lightweight adapter to augment CLIP for text-guided image ranking. Specifically, our approach incorporates learnable prompts to adapt to new instructions for ranking purposes and an auxiliary branch with ranking-aware attention, leveraging text-conditioned visual differences for additional supervision in image ranking. Our ranking-aware adapter consistently outperforms fine-tuned CLIPs on various tasks and achieves competitive results compared to state-of-the-art models designed for specific tasks like facial age estimation and image quality assessment. Overall, our approach primarily focuses on ranking images with a single instruction, which provides a natural and generalized way of learning from visual differences across images, bypassing the need for extensive text prompts tailored to individual tasks.

Published as a conference paper at ICLR 2025
RANKING-AWARE ADAPTER FOR TEXT-DRIVEN IMAGE ORDERING WITH CLIP
Wei-Hsiang Yu'
Yun-Yu Lin'
Ming-Hsuan Yang²
Yi-Hsuan Tsai
National Yang Ming Chiao Tung University
UCC Merced 3 Atmanity Inc.
ABSTRACT
Recent advances in vision-language models (VLMs) have made significant progress in downstream tasks that require quantitative concepts such as facial age estimation and image quality assessment, enabling VLMs to explore applications like image ranking and retrieval. However, existing studies typically focus on the reasoning based single image and heavily depend on text prompting, limiting their ability to learn comprehensive understanding from multiple images. To address this, we propose effective yet efficient approach that reframes the CLIP model into learning-to-rank task and introduces a lightweight adapter to augment CLIP for text-guided image ranking. Specifically, our approach incorporates learnable prompts to adapt to new instructions for ranking purposes and an auxiliary branch with ranking-aware attention, leveraging text-conditioned visual differences for additional supervision in image ranking. Our ranking anniingawwre aware adapter consistently outperforms fine-tuned CLIPs on various tasks and achieves competitive results compared to state-of- of-the-art models designed for specific tasks like facial age esti- mation and image quality assessment. Overall, our approach primarily focuses on ranking images with single instruction, which provides natural and generalized way of learning from visual differences across images, bypassing the need for extensive text prompts tailored individual tasks. The source code is available at https:/ / thub. com wwwp///uae/////eae Raaeee//aanni Rank reAWa
INTRODUCTION
Human perception can handle multiple images simultaneously, performing tasks like semantic categorization, e.g., counting cats an image, and abstract evaluations, e.g., image quality, aesthetics, and facial age. This ability is critical for comprehending the relationships between objects and concepts across images and is essential to effective reasoning and decision vreision-making. In machine learning, learning-to-rann
(LTR) algorithms are designed to investigate these relationships among items based on their relevance to a given context. LTR has shown remarkable success various domains, including information retrieval (Chen et al., 2021) and document ranking (Cao et al., 2007).
The emergence of Vision-Languaae Models (VLMs) like CLIP (Radford et al., 2021) has sparked growing interest in adapting pre-trained VLMs to understand specific image attributes (Ke et al., 2023; Liang et al., 2023; Wang et al., 2023a;d). However, prior research has primarily focused on single-image reasoning (see Figure la), potentially hindering the model's ability to explore complex relationships across images. In this paper, we aim to equip the pre-trained CLIP to rank multiple images based on various text queries, thereby facilitating wide range of tasks from semantic understanding (e.g., object counting) to abstract attribute estimation (e.g., image quality assessment). To this end, we introduce a ranking-awai -ware adapter that aims to provide text-visual embeddings with sense of visual relevance to a given text query, which is lightweight and effective for learning image ordering tasks (see Figure 1b).
A straightforward approach to adapt VLMs to target task is to conduct instruction tuning with pre-defined text queries followed by post- post-processing for ranking. For instance, Paiss et al. (2023) pre-define the text query to include the number of objects like "three cats" for each image. In this manner, the CLIP model can be fine-tuned using contrastive objectives for counting, and then images sorted based on their predicted counts. Similarly, Zhang et al. (2022) effectively addresses the depth estimation task by defining an object's proximity from "close" to "far." Other approaches augment the CLIP text encoder with additional learnable prompts to capture ordinal relationships

---

Published as a conference paper at ICLR 2025
Multi prompts with enumerated numbers
Latent Space
Text embeddinng
Modalated ordinal miations
Visual enibedking
A of 2anie.. photo of
CLIP IL
Modulation
Irage
CLIPr
(a) Prior Work
Latent Space
Single rank- related prompt
Sort images by the number of cats.
Ranking-awerd aware Adapter
CLIP
Image
Rayk maqginiaed
Image
CLIP tauge
Image
leal-oomiiooned vusual embediing pairs
Emieediing renl-reated promptt
(b) Ours
Teal-cundikkadd visual enbesliingg
Figure 1: Comparisons with prior work. (a) Prior works require generating caption combinations covering numbers of bins from N1, N2...., N; with the task- target, such as "cat," paired with each image (e.g., J, Iz). addition, text modulation is necessary to map these numerical values into ordinal latent space for contrastive learning. (b) Our method streamlines the ranking process through learning-to-rant rank-related prompt. A lightweight, ranking- aware adapter then generates cext-conditioned visual embedding pairs and their relational differences. By optimizing the relational differences among pairs, our approach learns the visual relevance to the given text query.
framework. A pre-trained CLIP model encodes images and a single
and align these prompts with image embeddings (Li et al., 2022; Wang et al., 2023d). However, aligning visual embeddings with text embeddings for ranking may be suboptimal since it difficult to enumerate and learn all prompts to represent the spectrum of ranks, especially for data in loiget distributions. While some methods achieve satisfactory results through task specifi pre-training and custom decoders, their specialized decoders often limit their adaptability to other ranking tasks.
To address the aforementioned challenges, we propose an LTR framework incorporating lightweight ranking adapter for the pre-trained CLIP model. This framework involves: 1) Transforming the paired image-text contrastive objective to an LTR objective by directly pairing a list images with their scores alongside text guidance; 2) Introducing ranking-aware adapter comprising regression branch for ranking score prediction and an ordinal branch for pairwise data order supervision; and 3) Developing attention mechanism to extract relative feature responses for pair of images, thereby constructing relative visual embeddings conditioned by queried text prompts.
We evaluate our method four tasks spanning diverse numerical concepts, including facial aging estimation (Samek et al., 2017), object count sorting (Singh et al., 2024), image quality/aesthetics assessment (Hosu al., 2020; Murray, Naila and Marchesotti, Luca and Perronnin, Florent, 2012), and dating historical colored images (Palermo et al., 2012). Our approach consistently performs favorably against CLIP baselines and state-of-the-art methods in terms of ranking and retrieval qualities, even though these competing methods are fine-tuned for target tasks. Instead of focusing on enhancing CLIP's text encoder, our method unifies different ranking tasks in one framework followed by lightweight adapter with ranking-aware attention module to capture text-conditioned image differences, directly contributing to learning the distinction across ranking scores.
The main contributions of this work are as follows: 1) We introduce general ranking-related text prompt and integrate learning-to-rant into the CLIP model using a lightweight adapter for multi- image ranking. 2) We propose ranking-awars attention module to facilitate feature learning for the ranking purpose across images conditioned on the given text. 3) Without requiring additional task-specific pre- training,
validate method on various ranking tasks in benchmark datasets against fine-tuned CLIP baselines and other task-specific approaches, providing insight toward a unified framework for image ranking tasks.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Contrastive vision-language models. Learning and aligning multiple modality-specific representa- tions from large-scale image-text datasets have significantly advanced the performance of numerous multi-modal tasks (Alayrac et al., 2022; Chen et al., 2023; Kim et al., 2021; Pham et al., 2023; Radford et al., 2021; Singh et al., 2022; Yu et al., 2022; Lin et al., 2024). CLIP (Radford et al., 202 I), a pio- neering vision- rision-language model (VLM), aligns visual and textual encoders via pairwise contrastive objective and excels at zero-shot classification. Due to their robust and general representations, VLMs widely adopted as foundational models for variety of downstream tasks, such as segmentation (Lüddecke & Ecker, 2022; Rao et al., 2022), detection (Wu et al., 2023), and dense prediction (Auty & Mikolajezyk, 2023; Hu et al., 2024). While CLIP effectively captures global information, its training objective, matching image content to sentences, hinders its ability to discern relationships across images. Several approaches are proposed to address this issue, such task-dependent decoder training (Jiang et al., 2023; Ke et al., 2023) and fine-tuning with task-specific image-caption pairs (Liang et al., 2023; Paiss et al., 2023; Wang et al., 2023a:b). While these approaches augment VLMs with additional relational knowledge, they often necessitate task-specific modifications to model architectures or data preprocessing. limiting their generalizability to diverse downstream tasks.
In this paper, we introduce a framework that empowers CLIP to rank set of images based on the attribute specified by a given text, such as ranking images based on object counts, facial aging, or visual qualities, without requiring any task-specific pre-processing model modifications. Our method enables the model to learn differences from randomly paired images and correlate them with their ranking distances, thereby enhancing ranking performance and achieving promising results compared to task-specific methods.
Learning-to-rain (LTR) is pivotal in training models to effectively sort a list of items, making it essential to information retrieval and recommendation systems that strive to present the most relevant items in response to queries (Cao et al., 2007). LTR has a broad spectrum of applications, such as person/face re-identification (Chen et al., 2017; 2021), artwork retrieval (Yemelianenko et al., 2023), and storytelling evaluation (Hsu et al., 2022). LTR models are typically trained using ranked lists items, associated queries, and relevance scores, such as click-through data, user preference scores, or similarities. Ranking objectives such pairwise Hinge loss, triplet loss (Schroff et al., 2015), or listwise ranking loss (Burges et al., 2005) are commonly employed to optimize LTR models. While classification predicts discrete categories and regression predicts continuous values for individual items, LTR focuses on cross-item comparisons based on their relative relevance to a query. This emphasis on relative relevance encourages models learn quantitative representations that distinguish between items based on given query.
seariing-to-ranle
Recent works like OrdinalCLIP (Li et al., 2022), L2RCLIP (Wang et al., 2023d), and the concurrent work NumCLIP (Du et al., 2024) combine CLIP with LTR for ordinal regression tasks such as facial age estimation and image dating. They focus on designing the text encoder to map numerical labels to continuous space for improved image-text alignment. In this work, we also combine CLIP and L.R for ranking tasks but focus on optimizing text-conditioned image embedding distances between paired images to enhance ranking, a promising yet unexplored area in visual understanding.
PROPOSED METHOD
This paper proposes a method to equip a pre-trained VLM, e.g., CLIP, with ranking a set of images based on given textual query, such as one particular object count, abstract concept, or subjective preference. Figure 2 provides an overview of our framework, where we formulate this task as a learn-to-rank problem and derive lightweight adapter with ranking-ave aware attention module to extract relative feature responses across images so that the model can rank images.
3.1
LEARNING-TO-RANK FORMULATION
A straightforward approach to teach VLMs for image scoring based textual queries is to yield prompts by associating images with potential numerical indices, such as "A photo of [0, 1, 2, 3, ...] cats." for object counting (Paiss et al., 2023) or "A photo of [1, 2, 3, ...] years old face." for age estimation (Wang et al., 2023d). However, enumerating all numerical indices is impractical and computationally infeasible when dealing with large or
3

---

Published as a conference paper at ICLR 2025
Query
Ranking- aware Adapter
Cross A tiection Transformer Encoder
Image! 1 Image la Imago!!
CLIP Image Encoder
Regression Hoad
pug
bed conditindt visual embodding of
patch embedding 4
reletional taketas
Relational Ranking-
"Sort images by the number
CLIP Text Encoder
aware A Ulntion
Key/Value
pairwise embodding fiy
[2/ zf
Ranking Head
Frozen
text embedding with learnable text lokens W]
Trainable
Auxiliary Ranking Branch
Figure 2: Framework overview. We encode the given images and the query caption using the pre-trained CLIP model. The proposed ranking adapter compiles text-conditioned visual embedding {2/} via the transformer cross-attention mechanism, where patch embeddings {2:} serve as queries and text embeddings {w,) act key-value pairs. The ranking adapter comprises two heads: the regression and ranking heads. The former predicts image ranking scores using features of individual images. The latter employs a ranking-aware attention mechanism explore relative feature responses across images upon which these images are ranked.
even uncertain index ranges. Furthermore, tasks like object counting and facial age estimation frequently exhibit long-ta tail data distributions, making discrete captions less balanced. For continuous scoring targets like image preference or colorfulness estimation, discretizing scores into bins can be helpful (Hu et al., 2024; Auty & Mikolajezyk, 2023). However, determining an appropriate number of bins and cutoffs remains challenging and may vary depending on the applications.
To address these challenges of ranking images based on text guidance, we design universal instruc- tion template for all images containing keywords such as "Sort images by the number of [cats]. and use their query query-related values as our learning targets. Then, we adopt objective to optimize the model for image ranking rather than pairing each image with all potential text queries.
3.2 RANKING ADAPTER WITH RELATIONAL. ATTENTION
The proposed method aims to augment CLIP for sorting images based given textual expression, where our model comprises a frozen CLIP backbone and a lightweight adapter. The adapter is composed of cross-attention encoder and two decoders: one is regression head for predicting single-image ranking scores, and the other derived with pairwise supervision for cross-image ranking. Figure 2 depicts our proposed method.
Vision-language backbone. We employ a frozen pre-trained CLIP model as our feature extractor. To accommodate various tasks that may require either local global image features (such sorting by object counts or image brightness), we discard the pooling layers as suggested in Paiss et al. (2023) and Rao al. (2022). This retains the image token embedding 2; [Rpxd, i E where P = 100 denotes the number of patch tokens and B is the batch size, and the text embedding w, E R where = 77 is the number of text tokens. Both image and text embeddings share latent dimension of d = 768. Besides, given that our text prompts are formulated for ranking, which differs from the original captions, we append the additional t' 32 text tokens to adapt to the ranking-related text prompts.
B}.
Adapting CLIP to image sorting with the ranking-aware adapter. Inspired by Yu al. (2022), we utilize cross cross-attention to integrate textual information into image tokens to enhance their contextual richness for accurate image ranking. Specifically, each encoding block our design includes self-attention layer for image tokens, followed by cross-attention layer where image tokens serve as queries and text tokens as key-value pairs. Linear layers are then used for embedding projection Id + Rd", where d' = 512. This structure is repeated twice to effectively fuse text and image embeddings, resulting in the text-conditioned visual embedding z/. Subsequently, the output
4

---

Published as a conference paper at ICLR 2025
ont-condiiooeed
erubodding 4j
atbestion
Madoll & Scalt
Quety
painwise ambeddiing Mej
Aveage FPX
relational tokeus q
Pairwise diffecence O1
Ind-condiiioned visual emboddingg
Figure 3: Relational ranking-aware attention. First, the text-conditioned visual embeddings of two images {2{} and {z are concatenated to form pairwise embedding {z4y} This pairwise embedding is used as the key in the attention mechanism, with relational tokens {q} serving as the query. The resulting attention matrix A is split into two parts, {A,} and {A₂}. corresponding to the attention assigned to each image. Using these matrices, the attention outputs, {0,} and {O₂} are computed with {2/} and {2/} as values. Finally, the pair's difference computed through subtraction, averaged over relational tokens, and processed by FFN to generate the pairwise difference Or.j-
ferd-forward sub-networf (FFN) regressor that directly predicts the ranking score. The second branch is auxiliary ranking module with a relational-aware attention mechanism, which characterizes the discrepancies between an image pair, aligning with their corresponding ranking score differences.
embedding 2/ is routed to two parallel branches. The first branch
Relational ranking-awen aware attention. Given a pair of images, this attention mechanism is designed identify and highlight the visual discrepancies that correlate with their respective ranking scores. As depicted in Figure 3, the visual-text embeddings of two images are concatenated to form a pairwise embedding, which is subsequently processed through an attentional pooling (Lee al., 2019).
Specifically, given two images 1, and 1j. we concatenate their embeddings, z! and z/j, from the cross-attention transformer encoder to yield a pairwise representation: z, = | z/] E R2axdd ij 21, where p is the number of patch tokens, and d' is the embedding dimension.
We then initialize the relational tokens IR M xd', where M is the number of tokens, as the queries. The concatenated embedding z acts as key-value pairs. This allows us to generate relational representations. The attention matrix A is calculated using scaled dot-product attention via
A = Softmax( 9 (k: 4 kz) Softmax Vd'
KT
Vd
where K = E 2pxd' acts as keys. After computing A, divided into two parts: A and Aj. corresponding to the attention applied to a and z/. respectively. Here, zj and z/ serve as values, denoted as VÚ and V, for the second dot-produc, operation:
O, = A; V/2 and O = AA VV-
Finally, the relative difference between the outputs, O, and O,, computed:
O₂ FFN Em=1 (O... O..m)
M
where Os.m is aggregated responses from image i with the m-th relational token. The pairwise difference Oz, is averaged over M relational tokens and passed through feed-forward network, yielding an one-dimensional value for the pairwise ranking objective.
Unlike conventional attention mechanisms that process images individually, our proposed relational attention aggregates distinct features between pair of images. The resulting features capture differ- ences between images and hence are discriminative, leading to more accurate ranking predictions.
Training objective and loss functions. To optimize the model for both accurate ranking and score prediction, we combine two complementary loss functions: a regression loss and pairwise ranking loss. The regression branch predicts the ranking score for each image individually, and we employ Smooth L1 loss to minimize the difference between the predicted score s; and the ground truth y::
[ 1 (w: s;)², 11) s.l - 0.5, otherwise.
if [y: sıl < 1,
Lreg
5

---

Published as a conference paper at ICLR 2025
This encourages precise alignment between predicted scores and the query- -asedd ground truth values, particularly penalizing large errors.
The pairwise ranking branch focuses relative ordering, using pairwise ranking loss to ensure that for any two images, c.g., when ys > Yj. the model is is derived by the following loss function:
Lrank (y. Oi,)
Σ
max(0, 0z,))-
(6.,)lu:>,s)
This loss directly optimizes the model's ability to capture the correct relative differences between image pairs, which crucial for ranking tasks.
Finally, we combine the two objectives into a unified loss function:
L = aLreg + Lrank-
(6)
where a a hyperparameter (set to 0.2 in our experiments) to balance the importance of the regression and ranking objectives. By jointly optimizing both individual scores and pairwise ranking consistency, our method ensures robust performance across various ranking tasks.
EXPERIMENTAL RESULTS
4.1 TASKS AND DATASETS
Facial age estimation. Facial age estimation predicts the age of given face. We use the Adience dataset (Samek et al., 2017), which includes 13, 027 images labeled across 8 age groups, following the data split from Wang et al. (2023d). We use the text prompt "sort images by the person' age group." for this task.
Historical colored image dating. The historical colored image dataset (Palermo et al., 2012) is widely-used benchmark for predicting the decade of given historical colored image, consisting of 1, 325 images labeled across decades ranging from the 1930s to the 1970s. We follow the standard ordinal regression setting that in Wang al. (2023d). The text prompt we use is "sort images by the taken date of the photo. " for this task.
Image quality and aesthetics assessment. For ranking images based on subjective preference and objective properties, we employ the KonIQ- 10k dataset (Hosu et al., 2020) for assessing image qualities and the Aesthetic Visual Analysis (AVA) dataset (Murray, Naila and Marchesotti, Luca and Perronnin, Florent, 2012) for evaluating image aesthetics, using the task prompt "sort images by the image quality of [Property]." In addition to user voting or mean opinion score (MOS), include other image quality attributes such as colorfulness, contrast, and brightness. We evaluate models using the official splits, which have 2, 015 and 19, 930 test images in the KonIQ-10k and AVA datasets, respectively.
Object count sorting. We evaluate the performance of scoring images according to the quantity of queried objects using the text prompt instruction, "sort images by the number of [Ca (category]: ... The COCO-REM dataset (Singh et al., 2024), annotation revised version of the COCO dataset, serves as the test bed. We work on the 80 categories of the COCO dataset, comprising 118, 287 training and 5, 000 testing images.
Evaluation metrics. We evaluate facial age estimation and historical image dating using mean absolute error (MAE) and accuracy for comparison with prior works. We assess performance using Pearson's 's linear correlation coefficient (PLCC) and Spearman's 's rank correlation coefficient (SRCC) for image quality assessment and object count sorting.
4.2 EXPERIMENTAL SETTING
We use CLIP with ConvNeXt-L, containing a ConvNext--arge (Liu et al., 2022) image encoder with the image resolution set to 320 x 320 and the hidden dimension to di 768, as well transformer-base
text encoder of 12 layers with w = 77 text tokens and the hidden dimension set to d2 768. The model weights pre-trained on the LAION-5B dataset (Schuhmann et al., 2022) are adopted and frozen for both the image and text encoders. To adapt our model to the given text, concatenate leamable prompts s' of size 32 the text embedding produced by the CL.IP text

---

Published as a conference paper at ICLR 2025
Table Results on f'acial age estimation and histor- ical colored image dating. The mean and standard deviation are listed. We take numbers from Wang et (2023d) for the results of the reference methods,
Table 2: Results object count sorting. For BLIP-2, Flamingo, InstructBLIIP and VIMMVVII,, follow the text prompts for object counting as specified in their studies.
Adience
HCI Accuracy
Metbod
Fine- tuning
PLCC
SRCC
Method
Accuracy
MAB
MAE
Zero-shot
(3.6)
0.80 (0.02)
26.1 (0.6)
1.48 (0.03)
BLIP-2
0,284
0,252
CLIP
Flamingo (10-shot)
0.033
0.031
CoOp
(5.6(
0.50 (0.08)
51.9 (2.6)
0.76 (0.06)
InstruciBLIP
OrdinalCLIP 61.2
0.47 (0.06)
56.4 (1.7)
0.67 (0.03)
VLM-VILA
0.538
0.507
1.2CIIIP
(4.4)
0.36 (0.05)
67.2 (1.6)
0.43 (0.03)
NomCLIP
69.6 (2.0)
0.35 (0.03)
Zero-shot CLIP
0.026
0.001
CountingCLIP
InstrwiBLIP
63.7
0.41
30.9
0.96
Paiss (2023)
0.251
0.422
Ours
65.2 (2.9) 0.36 (0.03)
72.8 (2.6)
0.32 (0.03)
Ours
0.624
0.557
Table 3: Results the IQA/IAA task. We report the PLCC and SRCC of the mean opinion score
(MOS) to compare with baselines and the state-of of-the-art methods. Note that MUSIQ (Ke et al., 2021) only serves as the reference purpose since it is purely vision-based method optimized for this task. We take numbers from Ke et al. (2023) for the results of the reference methods.
KonIQ
Method
Task-relatee pertaining
Fine-tuning
PLCC (
SRCC (1)
PLCC
SRCC (f)
Purely vision-based (peciei))
MUSIQ (202l)
0.937
0.726
0.738
VLM--aaee (task-speciffic
VILA-P Ke (2023)
0.657
0.663
VILA-R Ke al. (2023)
0.919
0.932
0.774
0.774
CLIP (ine-tuned)
0.245
0,216
0,162
0.160
InstructBLIP IP Dai (2023)
0.211
0.163
0.229
0.226
CLIP-IQA Wang (2023c)
0.695
0.727
0.415
CLIP-IQA+ Wang et al. (2023c)
0.895
0.909
0.677
0.587
Hentschel (2022)
0.741
Ours
0.919
0.911
0.760
0.747
encoder. Following Jiang et al. (2023), we discard the pooling and projection of the visual encoder, leaving p = 100 patch tokens as inputs to our ranking adapter. The ranking adapter consists of cross-attention transformer encoder with two encoding blocks, followed by two parallel branches: 1) a regression head with two multi-layer perceptrons (MLP) blocks and 2) ranking head with contrastive ranking-aware attention using M = 16 relational tokens and three MLP blocks.
We take a random horizontal flipping as data augmentation and resize the images to 320 320 without cropping. We implement the ranking adapter upon the OpenCLIP (Ilharco et al., 2021) framework and optimize the model using combination of Smooth L1 loss and Hinge loss with an AdamW optimizer at learning rate of 1c-5, weight decay of 0.01, and batch size of 64. We use 220k steps for object counts sorting and 144k steps for image-quality assessment, facial age estimation, and dating historical image tasks. We conduct all experiments on one NVIDIA RTX-3090TI GPU.
4.3 QUANTITATIVE RESULTS
In the following, we evaluate the efficacy of our method by comparing it with baseline models on four quantitative tasks with various challenges.
Facial age estimation and historical colored image dating. We evaluate our method against zero-shot CLIP and four existing approaches for facial age estimation and historical color image (HCI) dating. The benchmarks include CoOp (Zhou et al., 2022), OrdinalCLIPP (Li et al., 2022), L2RCLIP (Wang et al., 2023d), and the concurrent NumCLIP (Du et al., 2024). As shown in Table I, our approach, which leverages visual differences between images, achieves competitive performance on facial age estimation in the Adience dataset, rivaling methods focusing on text embedding modulation. Note that while method performs slightly lower than L2RCLIP, the standard deviation indicates our approach's 's stability over L2RCLIP. We also validate our model on the UTKFaaee dataset, where it outperforms other approaches (see Table in Appendix A. Moreover,
7

---

Published as a conference paper at ICLR 2025
Facial age estimation (A dience)
Facial estimation (UTKF)
Predict elder
yrunietyounger
Predict soder
younger
D Dating historical images
Image aesthetics assessment
Predictclose to 1930s Image quality assessment (photography quality)
closer 1970s
mere favorable
Predictless favorable
Image quality assessment (brightness)
Predict Ligher quality
Powdict
Predict beighter
dredit
Figure 4: Qualitative examples of our model. We visualize the ranking performance facial age estimation, dating historical images, and image quality and aesthetics assessment.
Object crnant tartingted
orange
by apple
CLP (lrettree
Ours
boat
CLIP
Outs
IQ4/AA task
by moon opinion score (MOS)
image contrast
Ouss CLIPIQA+
Figure 5: Qualitative examples on object count sorting and IQA/IAA. The images are sorted from highest to lowest score according to the textual cues. The red cross (x) represents the wrong sorting position in the list. Al artworks accurately ranks images from simple to complex compositions with multiple object categories in real photos and artworks. For image sorting based quality properties, method outperforms fine-tuned CLIP-IQA (CLIP-IQA+).
generated by DALL-E (Betker al., 2023). Our method
our method records an MAE of 0.32 on the HCI dataset, surpassing the existing state-of-the-art by margin of 0.04 (more than 9% improvement). Figure 4 further demonstrates our method's ability to accurately rank images in various scenarios.
Object count sorting. For object count sorting, we compare our method with two CLIP baselines: the zero-shot CLIP model and fine-tuned Count-to-Tee (Paiss et al., 2023), along with four represen- tative VL.M models BLIP-2 (Li et al., 2023b), Flamingo (Alayrac et al., 2022), InstructBLIP (Dai et al., 2023), and VLM-VILA (Lin et al., 2024) - using visual-question saaual-uuestioonnnseriigg (VQA) protocol. image-text pairs to determine counts, with object numbers in
While the two CLIP baselines rely COCO-REM ranging from one to hundreds, we combine categories and numbers from O to 100 in the caption during fine-tuning and inference, e.g., "A photo of (7) (cat).: For the VLM benchmarks, we follow the text prompts for object counting as specified their studies and parse the number from the generated caption. As reported in Table 2, our proposed method achieves PLCCC
8

---

Published as a conference paper at ICLR 2025
Table Results of other image attributes on the AVA dataset.
MOS
Contrast
Colorfulness
Brightness
(1)
(1)
(1)
SRCC
PI
SRCC
PL,CC
SRCC
CLIP (fine-tuned)
0.162
0.100
-0.058
-0.071
0.289
0.308
0.087
0.061
CLIP-IQA
0,420
0.415
0.227
0.210
0.472
0.186
0.433
0.442
CLIP-IQA+
0.677
0.687
0.695
0.712
0.881
0,895
0.908
0.919
Ours
0.760
0.747
0.951
0.946
0.973
0.966
0.981
0.979
of 0.624 and an SRCC of 0.557, which is significantly better than the original CLIP, its task-specific fine-tuned variant, and large VLMs using the VQA protocol. Notably, our method improves by elimi- nating the need to exhaustively enumerate object-count combinations during training and inference, typically required in pairwise contrastive approaches.
In addition, our method supports training with multiple queries simultaneously, such as sorting different objects (e.g., cat, dog, boat). Most CLIP-based methods struggle with this because they require pairing numbers and queried objects simultaneously, leading overly complex text prompts. Figure 5 shows that our method can rank images based on the queried category for real and synthesized images containing objects of one or multiple classes.
Image quality and aesthetics assessment. We compare our method to three existing methods for image quality and aesthetic assessment (IQA/IAA), including the CLIP model, CLIP-IQA (Wang et al., 2023c), and VILA (Ke et al., 2023), on two benchmarks: the KonIQ-1Ok and AVA datasets. In Table our method performs favorably against CLIP-IQA and its fine-tuned variant, CLIP-IQA+, while maintaining competitive performance compared to the VILA variants that require pretraining on task-related datasets on the AVA caption dataset (Ghosal et al., 2019) (i.e., VILA-P) and fine-tuned with rank adapter (i.c., VILA-R) (see Figure 5 for more visual comparisons with CLIP- CLIP-IQA+).
Moreover, we explore our model's capacity to score images based on various image attributes, includ- ing contrast, colorfulness, and brightness. We compare the performance with CLIP and CLIP-IQA using the AVA dataset. To fine-tune CLIP, we discretize attribute values into bins and pair images with multiple captions describing the specific attribute values. For example, the caption "The contrast of the image is hi the than C.1 and lower than 0.2." matches an image employ antonym prompts, such as "Good photo."
with contrast of 0.15. For CLIP-IQA, versus "Bad photo." " for MOS, "Colorful photo." " versus "Dull photo." " for colorfulness, etc., to train the model on each image attribute using the official repository from Wang et al. (2023c).
In Table 4, our proposed method consistently outperforms task-specific fine-tuned CLIP and CLIP- IQA+. It suggests that our proposed method can single model to directly score images according to various attributes indicated in the caption. Without needing antonym prompts or pre-trate pre-trained text embedding as anchors, our model learns quantitative knowledge from the instructions alone.
4.4 ABLATION STUDIES
Our ranking-aware attention CLIP stands out by focusing on text-driven visual differences to score images. In this section, we examine two key components of our ranking adapter: the auxiliary ranking branch and the design of the lanking-aware module.
Effect of the auxiliary ranking branch. We validate whether additional supervision from visual differences enhances ranking performance. We compare three settings: (1) using only the regression head, (2) adding separate ranking head with embedding subtraction =2/) for ranking supervision, and (3) applying ranking-awan attention to the paired embedding (z/,) for the supervision.
As shown in Table 5, the regression head alone performs reasonably well (e.g., 0.402 MAE on HCI and 0.612 PLCC on object count sorting), demonstrating that reframing the ranking tasks as dearning-to-rant problem is effective compared to standard contrastive learning used for fine-tuning CLIP. Adding separate ranking head for pairwise supervision yields noticeable improvements, particularly on the HCI dataset, highlighting the value of additional ranking supervision. Notably, its improvement is limited in complex tasks like object count sorting, where the model requires to focus on the queried object within a cluttered image for image scoring. Incorporating munking-aware attention in the ranking branch further enhances performance, with a +21 14% gain HCI, 11.6% in PLCC, and +3.3%% in SRCC for object counting. These results indicate that additional ranking supervision benefits image ranking tasks.

---

Published as a conference paper at ICLR 2025
Table Ablation study on the design of the ranking adapter. The first row stands for fine-tuned CLIP, serving as a reference.
ICI
Object count sorting
Ranking-aware Atlention
LTR paradigm
Ranking Head
MAB (4)
PLCC (1)
SRCC (f)
1.113
0.251
0. 0.122
0.402
0.612
0.588
0.355
(+11.69%)
0.619
(+1.14%)
0.536
0..37%
0.317
(4-21.14%)
0.624
(+ (+1.96%)
0.557
(+3.53%%
Table 6: Ablation study on the components of ranking- attention.
HCI
Object count sorting
Component
Used
Ablated
MAE (4)
PLCC (?)
SRCC (1)
Ours
0.317
0.624
0.557
(I) Attention oulput computation
O A, Vi;0,
A, V,
O
A-V
0.355
0.602
0.542
(2) Output representation
O O2 -0
Og [O,: 0,l
0.347
0.609
0.540
Attention mechanism
Cross artention
Self attention
0.339
0.621
0.545
Ablation study on the design of ranking-aware attention. the attention module to capture the visual differences between image pairs. We validate design by swapping components within the ranking- aware module individually.
Several designs can potentially enable
In Table 6, we ablate (1) the split dot-product (A; V; A, Vj) by merging it into A V; (2) We compare explicit subtraction with concatenate for embedding fusion: (O; O, vs. [O;: OýD:: (3) We assess attentional pooling against self-attention for computing the attention matrix.
As shown in Table 6, both the split dot-product and explicit subtraction of output embeddings are essential, leading improvements in MAE, PLCC, and SRCC. Using merged dot-product for attention outputs may confuse the model about response sources, leading to performance degradation. Regarding the comparison between concatenation and explicit subtraction of output embeddings, our results show that explicit subtraction performs better, likely because it aligns more naturally with the ranking task. Finally, we observe that using cross-attention to aggregate responses from image pairs is more efficient regarding performance and GPU memory usage. In contrast, self- self-attention consumes about twice the GPU memory (due to the initial dot-product operation, Q K") and delivers worse performance. This result suggests that self-attention, which requires calculating relationships between all elements across image pairs, may lead to a higher computational burden and increased complexity, making training convergence more difficult.
CONCLUSIONS
This work presents an efficient and scalable framework for text-guided image ranking. By reframing CLIP's image-text contrastive learning into learning-to-ren tt-riin task and introducing ranking aware attention mechanism, our model effectively captures text-driven visual differences between image pairs without relying task-specific fine-tuning curated datasets. Experiments show the effecttve-- ness of our approach, surpassing CLIP baselines and achieving results comparable to state-of-the models tailored for specific tasks. Overall, our work highlights the potential of integrating VLMs with ranking tasks, utilizing text-guided visual distinctions for the sense of quantitative concepts.
Limitations and future works. Our ranking module can be trained on diverse text prompts covering single and multi-task properties. For example, it can sort images based on criteria such as the number of cats and image quality with a single model. As shown in Table 10 (Appendix B.3), our approach achieves performance on par with models trained for individual tasks. While our method shows convincing results in ranking images by two attributes simultancously (Appendix E). the lack of ground truth for multi-attribee rankings complicates evaluation. Besides, as we adapt CLIP to rank images by specified queries, our method retains its category bias, hindering its performance on apen-vocabulary tasks like counting "objects of whatever categories." The pairwise nature of our ranking- aware attention limits its compatibility with triplet-based ranking losses, such as the triplet or ranked list losses. Future research directions include exploring strategies and datasets for multi-attribute and open-vocabulary image ranking. Additionally, incorporating diverse ranking losses beyond pairwise comparisons could improve the model's robustness and adaptability.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGMENT
This work was supported in part by the National Science and Technology Council (NSTC) under grants 112-2221-E-A4--00--NNY 111-2628-E-4444025----- and I13-2654-F-006-000.
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: Visual Language Model for Few-Shot Learning. In NeurIPS, 2022. 3, 8
Dylan Auty and Krystian Mikolajczyk. Learning to Prompt CLIP for Monocular Depth Estimation: Exploring the Limits of Human Language. In ICCV Workshop, 2023. 3,4
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, al. Improving Image Generation with Better Captions. Computer Science. https://cdn. openai. com/papers/üull--3. pdf, 2023. 8
Christopher J. C. Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Gregory Hullender. Learning Rank Using Gradient Descent. In ICML, 2005. 3
Zhe Cao, Tao Qin, Tie Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to Rank: From Pairwise Approach to Listwise Approach. In ICML, 2007. 1,3
Xi Chen, Xiao Wang. Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V. Thapliyal, James Bradbury, and Weicheng Kuo. PaLI: A Jointly-Scaled Multilingual Language-Image Model. In ICLR, 2023.3
Yang Chen, Yong-Jin Liu, and Yu-Kun Lai. Learning Rank Retargeted Images. In CVPR, 2017.
Yiqiang Chen, Stefan Duffner, Andrei Stoian, Jean-Y Yves Dufour, and Atilla Baskurt. List-wise Learning-to-Rante with Convolutional Neural Networks for Person Re-identification. Machine Vision and Applications, 2021. 1,3
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning In NeurIPS, 2023. 7, 8, 14
Yao Du, Qiang Zhai, Weihang Dai, and Xiaomeng Li. Teach clip to develop a number sense for ordinal regression. arXiv preprint Ntl:2400.033544 2024. 3,7
Koustav Ghosal, Aakanksha Rana, and Aljosa Smolic. Aesthetic Image Captioning From Weakly-
Labelled Photographs. In ICCV Workshop, 2019. 9
Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A Dataset for Large Vocabulary Instance Segmentation. In CVPR, 2019. 19
Simon Hentschel, Konstantin Kobs, and Andreas Hotho. CLIP knows image aesthetics. Frontiers Artif. Intell., 2022.1
V. Hosu, H. Lin, T. Sziranyi, and Saupe. KonIQ- 10k: An Ecologically Valid Database for Deep Learning of Blind Image Quality Assessment. IEEE Transactions on Image Processing, 2020. 2,
Chi- Yai-Yang Hsu, Yun- Wei Chu, Vincent Chen, Kuan-Chich Lo, Chacha Chen, Ting-Hao Huang, and Lun-Wei Ku. Learning to Rank Visual Stories From Human Ranking Data. In ACL, 2022.
Xueting Hu, Ce Zhang, Yi Zhang, Bowen Hai, Ke Yu, and Zhihai He. Learning To Adapt CLIP for Few-Shot Monocular Depth Estimation. In WACV, 2024. 3, 4
11

---

Published as a conference paper at ICLR 2025
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021. URL https: http:://oo...rggg/// 5281/ zenodo. 51 133773.7
Ruixiang Jiang, Lingbo Liu, and Changwen Chen. CLIP-Count: Towards Text-Guided Zero-Shot Object Counting. In ACM MM, 2023. 3,
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. 14
Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang. MUSIQ: Multi-scale Image Quality Transformer. In ICCV, 2021.7 7
Junjie Ke, Keren Ye, Jiahui Yu, Yonghui Wu, Peyman Milanfar, and Feng Yang. VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretraining. In CVPR, 2023. 1, 9, 14
Wonjae Kim, Bokyung Son, and Ildoo Kim. ViLT: Essionandd-Laggual Transformer Without Convolution Region Supervision. In ICML, 2021. 3
Maksim Kuprashevich and Irina Tolstykh. Mivolo: Multi-input transformer for age and gender estimation. CoRR, 2023. 14
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation- invariant neural networks. In ICML, 2019. 5
Chunyi Li, Zicheng Zhang, Haoning Wu, Wei Sun, Xiongkuo Min, Xiaohong Liu, Guangtao Zhai, and Weisi Lin. AGIQA-3K: An Open Database for Al-Generated Image Quality Assessment. CoRR, 2023a. 19
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023b. 8
Wanhua Li, Xiaoke Huang, Zheng Zhu, Yansong Tang, Xiu Li, Jie Zhou, and Jiwen Lu. Ordinalclip: Learning rank prompts for language-guinn guided ordinal regression. In NeurIPS, 2022. 2, 3,7
Dingkang Liang, Jiahao Xie, Zhikang Zou, Xiaoqing Ye, Wei Xu, and Xiang Bai. CrowdCLIP: Unsupervised Crowd Counting via Vision-Language Model. In CVPR, 2023. 1,3
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. VILA: On Pre-training for Visual Language Models. In CVPR, 2024. 3,
Zhuang Liu, Hanzi Mao, Chao-Yur Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. arXiv preprint wwXiv:2201.035455 2022.
Timo Lüddecke and Alexander S. Ecker. Image Segmentation Using Text and Image Prompts. In CVPR, 2022.
Murray, Naila and Marchesotti, Luca and Perronnin, Florent. AVA: A Large-scale Database for Aesthetic Visual Analysis. In CVPR, 2012. 2, 6
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada. Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching CLIP to Count to Ten. In ICCV, 2023. 1, 4, 7,
Frank Palermo, James Hays, and Alexei A. Efros. Dating historical color images. In ECCV, 2012.
Hieu Pham, Zihang Dai. Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Thang Luong, Yonghui Wu, Mingxing Tan, and Quoc Le. Combined scaling for zero-shot transfer leaming. Neurocomputing, 2023.
Yu, Vi-Ting Chen, Minh-Thang
12

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In ICML, 2021. 1,
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. DenseCLIP: Language-Guidee Dense Prediction with Context-Aware Prompting. In CVPR, 2022.
Wojciech Samek, Alexander Binder, Sebastian Lapuschkin, and Klaus-Robert Müller. Understanding and comparing deep neural networks for age and gender classification. In ICCV, 2017. 2,
Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: A Unified Embedding for Face Recognition and Clustering. In CVPR, 2015. 3
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski. Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-SB: An open large-scale dataset for training next generation image-text models. In Neur/PS, 2022.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kicla, FLAVA: Foundational Language And Vision Alignment Model. In CVPR, 2022. 3
Shweta Singh, Aayan Yadav, Jitesh Jain, Humphrey Shi, Justin Johnson, and Karan Desai. Bench- marking Object Detectors with COCO: A New Path Forward. arXiv preprint arXiv:2403.1887, 2024. 2, 6
Alex Jinpeng Wang, Pan Zhou, Mike Zheng Shou, and Shuicheng Yan. Position-guided Text Prompt for Vision-Language Pre-training. In CVPR, 2023a. 1,3
Hualiang Wang, Yi Li, Huifeng Yao, and Xiaomeng Li. CLIPN for Zero-Shot OOD Detection:
Teaching CLIP to Say No. In ICCV, 2023b. 3
Jianyi Wang, Kelvin C. K. Chan, and Chen Change Loy. Exploring CLIP for Assessing the Look and Feel of Images. In AAAI, 2023c. 7,9
Rui Wang, Peipei Li, Huaibo Huang, Chunshui Cao, Ran He, and Zhaofeng He. Learning-to- rank meets language: Boosting Lanuage-driver ordering alignment for ordinal classification. In NeurIPS, 2023d. 2, 3, 6, 7, 15
Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. CORA: Adapting CLIP for Open- Vocabulary Detection with Region Prompting and Anchor Pre-Matching. arXiv preprint 2r31v:2303.13076 2023. 3
Tetiana Yemelianenko, Iuliaa Tkachenko, Tess Masclef, Mihaela Scuturici, and Serge Miguet. Learn- ing to Rank Approach for Refining Image Retrieval in Visual Arts. In ICCV Workshop, 2023.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.
CoCa: Contrastive Captioners are Image-Tex: Foundation Models. TMLR, 2022. 3,
Zhang, Zhifei, Song, Yang, and Qi Hairong. Age progression/reegresion by conditional adversarial autoencoder. In CVPR, 2017. 14
Renrui Zhang, Ziyao Zeng, Ziyu Guo, and Yafeng Li. Can Language Understand Depth? In ACM MM, 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. IJCV, 2022.
13

---

Published as a conference paper at ICLR 2025
A EVALUATE ON OTHER DATASETS
A.1 FACIAL AGE ESTIMATION ON THE UTKFACE DATASET.
Table 7 shows additional results facial age estimation using FTKeaace dataset (Zhang et al., 2017). Following the preprocessing and data split described Kuprashevich & Tolstykh (2023), we train our ranking-awan adapter for 20k steps with batch size of 64, learning rate of 5e 5, and a weight decay of 0.01. Our method outperforms MiVOLO (Kuprashevich & Tolstykh, 2023), designed to predict a person's 's age based on facial or full-body images.
Table 7: Facial Age Estimation Results the UTKFace Dataset. We take numbers from Kupra- shevich Tolstykh (2023) for the results the reference methods.
Method
MAE (4)
CORAL
5.39
Randomized Bins
4.55
MWR
4.37
MIVOLO
4.23
Ours
3.83
A.2 OBJECT COUNT SORTING ON ON CLEVR DATASET.
The CLEVR dataset (Johnson et al., 2017) is often used to assess complex relational understanding. such as answering questions like "Are there an equal number of large things and metal spheres?" in visual question-answering context. Here, we focus simple yet general counting task by counting instances that match the queried attribute. Specifically, we train and evaluate the model's ranking ability based the counts of three attributes: color (8 colors), material (2 materials), and shape (3 shapes), using single model. For example, we use the prompt "Sort images by the number of objects in red." order images by the number of red objects. To the best of our knowledge, no existing benchmark aligns with our setting, so we use CLIP and InstructBLIP (Dai al., 2023) for comparison. As demonstrated in Table 8, our approach effectively ranks images based the queried attributes, outperforming both the CLIP baseline and InstructBLIP. Figure 6 further illustrates the quantitative results for queries based on different attributes.
B
MORE EXPERIMENTAL DETAILS
B.1 OBJECT COUNT SORTING RELATIVE TO SPECIFIC COUNTS
Sorting images based on the counts of a queried object is straightforward way to assess model performance. However, our method is viable for sorting images by relevance to other specified counts of queried object.
To validate this, we conduct an experiment where, during training, we randomly sample a reference count n and adjust the target values to reflect the distance between the actual object count and n. For instance, given batch with images containing [0, 3, 5, 10] dogs and a reference count of 4, the target values for ranking become [4, 1,
Figure shows that with a query of dogs, 'images without dogs are ranked highest, followed by images with 2, 4, and dogs. Similarly, for queries like "3 dogs" or "6 dogs," the sorting results match our expectations.
B.2 COMPARISONS WITH GENERAL
MODELS
Peneral-purposs languaee models (VLMs) have demonstrated strong capabilities in solving tasks via a welgquusstionnnnsweriie paradigm. In this study, we evaluate their performance on ranking tasks, specifically testing InstructBLIP (Dai et al., 2023) across all tasks and VILA- VLM (Ke et al., 2023) on the object count sorting task (Table 2). The text prompt used for each task as follows:
14

---

Published as a conference paper at ICLR 2025
Table 8: Object Count Sorting Results on the CLEVR Dataset.
Color
Material
Shape
Method
Backbone
PLCC
SRCC
PL.CC
SRCC
PL.C
SRCC
CLIP (baseline)
ConvNext-L
0.263
0.258
0.267
0.256
0.255
0.247
InstructBLIP
ViT-g/14
0.194
0.170
0.332
0.315
0.584
0.548
Ours
ConvNext-L
0.992
0.836
0.992
0.981
0.993
0.966
color in yellow
KIS
112
color in purple
shape in cube
naterial in metal
RAb
RW5
Figure 6: Quantitative examples of querying objects by different attributes. Images are ordered from high matching score to low matching score according to related query attributes.
Object count sorting: "How many (category) in the image? Answer in a number."
Image quality assessment: "How is the photography quality of the image? Answer in a number from to 10."
Facial age estimation:
"How old is the person in the image? Answer in number."
Historical colored image dating: "When was this image taken (from 1930s number."
to 1970s) Answer
For facial age estimation and HCI datasets, where ground truth labels are ordinal labels, post- process the generated text into bins as defined by Wang et al. (2023d). As shown in Table 9, while InstructBLIP performs well in facial age estimation, it struggles with tasks such as image quality assessment and determining a photo's decade.
B.3 TRAINING THE RANKING ADAPTER ACROSS MULTIPLE TASKS
Our method uses a learning-to-rann framework, leveraging text-driven visual differences for image ranking, which enables training across multiple tasks with diverse text queries. As shown in Table 10, the model trained on multiple tasks simultancously achieves competitive performance compared to those trained on individual tasks. While there is trade-off between task generalization and task- specifc optimization where diverse queries could slightly reduce individual task performance this is balanced by the model's flexibility and its ability handle multiple attributes across various
15

---

Published as a conference paper at ICLR 2025
images by relevance
Sort mages
images by relevance dog.
imgges relevance cat.
Sortimages by relevance dog.
Sort relevance catt.
Quantitative examples of queried by specific category relevant specific counts. Images are ordered from high matching score to low matching score with respect to the text query.
Figure 7:
Table 9: Comparison with general VLM methods.
Object count sorting
KonIQA-10K
Adience
HCI
Method
Backbone
PLCC
SRCC
PLCC
SRCC
MAE
MAE
CLIP (baseline)
ConvNext-L
0.251
0.422
0.245
0.216
0.80
1.48
InstructBLIP
ViT-g/14
0.509
0.485
0.211
0.163
0.41
0.96
Ours
ConvNext-L
0.624
0.557
0.919
0.911
0.36
0.32
Table 10: Comparison between training on individual tasks and simultaneous training on multiple tasks. Red and blue numbers denote the top two scores.
HCI
Adience
KeatQ-10k
Object count sorting
Training
MAB
MAE (L)
PLCC (t)
SRCC
PLCC (t)
SRCC (t)
individual (HCI)
0.31
1.50
0.013
-0.069
-0.046
individual (Adieenc)
2.25
0.36
0.044
0.061
-0.001
0.036
individual (KonIQ-10k 10k)
2.57
individual (Object count sorting)
2.99
4.09
0.044
0.065
0.624
0.587
mixed
0.36
0.43
0.881
0.876
0.637
0.573
Table 11: Ablation for the different number of relational tokens in the ranking- module.
IICI
KontQ 10k
Object count sorting
Atokens
MAE
PLOC
SRCC
PLCC
SRCC
0.9182
0.9104
0.6003
0.5211
0.343
0.9172
0.9087
0.6197
0.5483
0.321
0.9173
0.9087
0.6167
0.5451
0.313
0.9194
0.9107
0.6242
0.5571
domains. This offers broader quantitative understanding across diverse scenarios and enables us to explore image ranking by multiple attributes simultancously.
B.4 EFFECT OF THE NUMBER OF RELATIONAL. TOKENS IN RANKING-AWARE ATTENTION
Table 11 presents ablation study the impact of using different numbers of relational tokens M in the ranking-aware attention module of our ranking adapter. The results indicate that M = 16 yields the best average performance, and use this as the default setting for all other experiments.
16

---

Published as a conference paper at ICLR 2025
Table 12: Ablation for the different visual backbones and image resolution.
Object count sorting
KunIQA 10k (MOS)
HCI
Adhience
Backbone
Sparams
Image resolution
PLCC
SRCC
PLCC
SRCC
MAE
MAE
VIT-B/16
87M
224
0.574
0.497
0.929
0.911
0.45
0.39
ComNNext-B B
89M
224
0.546
0.480
0.926
0.904
0.44
0.39
ConNNext-L.
198M
224
0.592
0.520
0.920
0.900
0.41
0.40
ConvNext-L
198M
320
0.624
0.557
0.919
0.911
0.32
0.36
B.5 EFFECT OF DIFFERENT VISUAL BACKBONE, AND IMAGE RESOLUTION
Table 12 presents the effect of using different visual encoders and image resolutions among different tasks. The results demonstrate that our proposed method performs well across different backbones and image resolutions, with larger backbones and higher resolutions yielding improved performance.
C CHALLENGES AND FAILURE CASES
Although the COCO-REM dataset refines the annotations of the COCO dataset, labeling densely packed objects remains challenging, often resulting in incorrect counts. A "crowded" flag is used to indicate densely packed objects. Figure 8 displays several challenging cases for ranking adapter. Our model accurately assigns higher scores to images with a "crowded" flag for the object count sorting task, indicating accurate counts far exceeding the annotated counts. Such a "crowded" flag is typical in categories such as "Person"", "Food", and "Animals", etc. This result highlights the advantage of using a ranking paradigm for quantitative comprehension, as better tolerates noisy labels without directly pairing incorrect values with images. However, our model struggles with identifying icons or paintings as objects, such as cats painted on a bus or apples on a tablecloth, as in Figure 8(a).
In the image quality/aesthetics assessment task in Figure 8(b), the model's ranking results closely align with subjective "MOS" scores. Nonetheless, some images with similar content receive significantly different subjective scores, while our model will have similar scores to these images, such as the flower examples in Figure 8(b).
D MORE QUALITATIVE EXAMPLES
As shown in Figure we evaluate our ranking adapter on unseen categories using the LVIS dataset, which includes more categories than the COCO dataset. The model maintains its ability to rank target objects, such as butterflies, when the target categories are not coupled with others in the dataset. Performance drops drastically for categories like deer, which often appears with zebra giraffe, or lemon, which frequently found with apple or orange. Moreover, our model struggles with counting objects in images taken with depth of field, such as those containing pillows.
For image aesthetics assessment, we apply our ranking adapter to the AGIQA-3k dataset, which includes generated artworks from various image generative models using the same prompt. As shown in Figure 10, we sample five images from the highest to lowest MOS at the same interval, finding that model shows high agreement with subjective scores.
E QUERY BY MULTIPLE ATTRIBUTES
Our proposed method can rank images on different tasks and with various attributes, making evalu- ating its performance with multiple attributes intriguing. However, combining two attributes into single caption can cause issues: 1) it is difficult to determine which attribute the model focuses on more, and 2) how to determine the ground truth is unclear. Still, we explore the potential of our pro- posed method by evaluating images on the AVA dataset and AGIQA-3k dataset qualitatively. When combining multiple attributions, scores, normalize these scores, and then multiply them to rank images by both attributes. As shown in Figure 11, the top-5 and bottom-5 images retrieved by our method highly reasonable. Additionally,
separate captions for each attribute to generate individual
17

---

Published as a conference paper at ICLR 2025
by cat
by person
Ground Truth
n-20
Ours
by apple
by bottle
Ground Truth
0142
n-20
0-20
Ours
(a) Object Count Sorting Task
AVA dataset
Ground Truth
Ours
KonIQ-10k
Ground Truth
Ours
(b) Image Quality/A esthetics Task
Figure 8: Examples of challenging cases.. The top row shows images ordered by annotated counts, and the bottom row shows images ordered by our model. a) For images in the CÓCO-REM dataset, icons (e.g., cat painting on the bus) may confuse the model, whereas it accurately scores crowded objects as seen the examples of "Person". "Apple", and "Bonle". b) For the MOS, as the general order is similar, some images that have similar patterns (e.g., flowers) may have different preference scores.
when retrieving images based on an object category and an image property, the results align well with the query. By adjusting the weight given to different attributes. it is possible to tailor the retrieval results to favor one attribute over the other, highlighting the potential for future applications.
18

---

Published as a conference paper at ICLR 2025
By butterfly
by deer
Grand Truth
Ous
by egg
by lemon
Grand Truhh
Outs
by pillow
by helicopter
Grand
Truth
Ours
Figure 9: Examples the LVIS dataset (Gupta et al., 2019). Evaluation of our model examples from unseen categories during training. The top row shows images ordered by annotated counts, and the bottom row shows images ordered by our model.
Painted portrait of intense woman, anime style
Everything dream
Ground Truth
ISP
Outs
A cosmic universe
A colour portrait of a king, realistic style
Ground Truhh
Panda chef in a kitchen
Logo design of a full moon
Ground
Trub
Ovaroun
Ous
Figure 10: Examples on AGIQA-3k (Li et al., 2023a). Comparison of Al-generated artworks in the AGI3Q dataset based on MOS. The top row shows images ordered by subjective scores from high to low, and the bottom row shows images ordered by our model.
19

---

Published as a conference paper at ICLR 2025
MoS/Coloee Colorfulress
Top-5 reizieval
Lasb-5 retrieval
MOSS Conbast
Top-5 retzieval
Last: 5 retrivaal
Controst/ Colorfulness
Top-5 retzival
Last-5 retrieval
Figure 11: Queried by multiple IQA attributes. The scatter plot shows the normalized scores of two attributes, images with those normalized values above the median highlighted in red and those below in blue. Examples for the top-5 and last-5 retrievals of kos/Colorfulnese. MOS/Cont rast. and cont
AVA Dataset
AGIQA-3k
MOS / Person
AVA Dataset
AGIQA-3k
(b) Colorfulness/ Person
Figure 12: Queried by multiple attributes across tasks. Examples for the top-5 images retrieved using two attributes: (a) by MOS and by the number of "persol "person" and (b) by Colorfulness and by the number of "person", on the AVA and AGIQA-33 3k datasets.
20


Reviews:
[{"review_id": "S3nDE6fQXX", "reviewer": "Reviewer_a2Bs", "summary": "The paper introduces a **Ranking-Aware Adapter for Text-Driven Image Ordering with CLIP**, designed to enhance CLIP’s capability in text-guided image ranking. Key points include:\n\n1. **Objective**: The method reframes CLIP into a learning-to-rank (LTR) task, using a lightweight adapter to improve ranking tasks across image attributes (e.g., facial age, object count, image quality).\n\n2. **Ranking Adapter Design**: The adapter consists of:\n   - **Learnable Prompts** for adapting CLIP to new ranking instructions.\n   - **Auxiliary Ranking Branch** that uses ranking-aware attention to focus on text-conditioned visual differences.\n   - **Two Parallel Heads** for regression (individual image ranking score) and pairwise ranking supervision (to capture relative differences across images).\n\n3. **Performance**: The adapter consistently outperforms fine-tuned CLIP on tasks such as facial age estimation, image dating, image quality assessment, and object count sorting.\n\n4. **Evaluation and Ablation Studies**: The model is validated on multiple benchmarks, showing improved ranking accuracy, particularly as dataset complexity grows. Ablation studies highlight the contributions of each component, confirming the benefit of ranking-aware attention for pairwise comparisons.\n\nThis method enhances CLIP’s ability to rank images without exhaustive task-specific tuning, aiming to generalize across ranking tasks. Let me know if you'd like to dive into specific aspects like strengths, weaknesses, or contributions.", "questions": "Please see the weaknesses section on questions.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- *Generalizable Across Ranking Tasks*: The ranking-aware adapter is designed to handle multiple text-driven ranking tasks (e.g., age estimation, object counting, image quality assessment) without requiring extensive task-specific tuning. This flexibility suggests that the approach could generalize across diverse ranking applications, potentially making it adaptable for other vision-language tasks where relative comparisons matter.\n\n- *Improved Performance on Benchmarks*: the proposed method improves performance on tasks like object count sorting, facial age estimation, and image aesthetics assessment, which could indicate its effectiveness in capturing ranking relationships across different domains.", "weaknesses": "- **Clarity and Reproducibility Issues**: The paper is challenging to follow, with several instances where the context is unclear, and symbols (e.g., Eq(5) and ΔO) are introduced without proper explanation. This lack of clarity makes it difficult to fully understand the method and poses challenges for reproducing the results. In particular, additional context is recommended regarding:\n   - The specific role and application of the ranking score across different tasks.\n   - Which of the two heads (regression or ranking) produces the final output, especially for each individual task.\n   - How MAE is computed in experiments like facial age estimation and image dating, given that the model primarily performs ranking rather than explicit regression or classification.\n   - What exactly is being ranked in each experiment, as this varies by task and isn’t sufficiently explained.\n\n- **Ambiguity in Terminology and Methodology**: The claim that the paper “reframes the CLIP model into a learning-to-rank task” is ambiguous and could be misleading. A model cannot be reframed into a task; it can be adapted or extended to handle a task. In my understanding, the approach here is simply to add an adapter on top of the existing CLIP model, with CLIP’s image and text encoders used as frozen feature extractors. The authors are encouraged to clarify what they aim to achieve in reframing CLIP for ranking and to describe more precisely how this method differs from other approaches to ranking with CLIP features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WAYaHJ5cim", "reviewer": "Reviewer_FmxD", "summary": "The paper proposes an efficient approach to enhance the CLIP model for image ranking tasks by reframing it as a learning-to-rank problem. It introduces a lightweight adapter with learnable prompts and a ranking-aware attention branch to leverage text-conditioned visual differences. This method consistently outperforms fine-tuned CLIP models across various tasks, achieving competitive results in facial age estimation and image quality assessment. By focusing on image ranking with a single instruction, the approach offers a generalized way to learn from visual differences without relying heavily on extensive text prompts.", "questions": "1. The paper utilizes CLIP with ConvNeXt-L, which differs from the backbone networks in existing methods. Does this affect the fairness of experimental comparisons?\n\n2. Does the model require separate adapter networks for different tasks, and how might this impact its generalization to unseen scenarios?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is clear and well-structured, making the content easy to follow. The logical flow helps readers grasp the key concepts without difficulty.\n\n2. The motivation behind the study is explained exceptionally well. It highlights that existing research often centers on reasoning from a single image and relies heavily on text prompts. This approach restricts the ability to achieve a comprehensive understanding when multiple images are involved. By addressing these limitations, the study aims to enhance multi-image reasoning capabilities.\n\n3. The method diagram is presented with clarity, making it easy for readers to comprehend the proposed approach. This visual aid effectively supports the explanation of complex processes, ensuring that the methodology is accessible to a broad audience.\n\n4. The experimental results show impressive performance across several datasets. This demonstrates the robustness and effectiveness of the proposed methods, indicating their potential for broader application in various contexts.", "weaknesses": "1. Comparison with Existing Methods: There is a need to clearly delineate the core differences between OrdinalCLIP, L2RCLIP, and NumCLIP compared to existing methods, which might not be fully addressed.\n\n2. State-of-the-Art Comparisons: The article does not adequately compare the proposed models to state-of-the-art multi-modal large language models (LLMs), which could provide a more comprehensive evaluation of their performance.\n\n3. Performance on Complex Benchmarks: The performance on complex counting benchmarks like TallyQA and CLEVR is not thoroughly evaluated, leaving questions about the models' capabilities in more challenging scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ea4pDzvoUe", "reviewer": "Reviewer_AayG", "summary": "This paper introduces a modified CLIP model for text-guided image ranking, using a lightweight adapter with ranking-aware attention to enhance understanding of visual differences across multiple images. By incorporating learnable prompts for ranking instructions, the approach reduces dependence on extensive text prompting. The method outperforms fine-tuned CLIP on various tasks and rivals specialized models in areas like facial age estimation and image quality assessment.", "questions": "- Are the weights of the relational ranking-aware attention also frozen? If they are trainable, a trainable symbol should be indicated in Figure 2.\n- How does the relational ranking-aware attention handle multiple images (i.e., more than two)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The motivation is clear: while previous methods require generating multiple captions for input images, this approach only needs a single rank-related text prompt.\n2. The proposed ranking-aware adapter achieves superior performance over fine-tuned CLIP and is competitive with specialized models for tasks like facial age estimation and image quality assessment, offering a versatile solution for text-guided image ranking.", "weaknesses": "1. Although the motivation of this paper is clear, the technical contribution does not appear strong enough from my perspective. I will wait to see other reviewers’ comments on this aspect.\n2. In Equation (2), the symbols $V_i$ and $V_j$ are not explained. Adding the shapes or dimensions of certain symbols in Equation (2) would enhance clarity.\n3. Regarding the experiments: why does ranking-aware attention utilize three MLP blocks?\n\nMinor Issues:\n- Consider adding count numbers to the results in Figures 5 and 6 for clarity, as done in Figure 7(a).\n- The citation format in Lines 417, 309, 160, and 161 appears inconsistent. I suggest the authors review the full paper and adjust citation formatting for consistency with Line 424 if needed.\n- The symbols between Figure 3 and the main text are inconsistent. For example, the symbol for relation tokens is $\\mathbf{q}$ in Figures 2 and 3 but $q$ in Equation (1).\n- Some symbols in Figure 3 are unclear. $A_i$ is a matrix, but $P$ represents dimension. What does $A_i(1-P)$ signify?\n- The column margins in Tables 1 and 2 are too wide, causing the tables to appear cramped together, and the right edge of the table extends beyond the page, which affects visual appeal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6btiGFBIUv", "reviewer": "Reviewer_DJCM", "summary": "This paper introduces a light-weight rank adapter which allows using an existing pre-trained CLIP model for any ranking task. The authors propose using learnable text prompts along with a light-weight cross-attention based ranking module which can measure visual differences between pairs of images. This approach is much simpler than some of the preceding works as it is able to be trained for the learning-to-rank task using a single text prompt such as \"sort by number of [category]\" compared to some of the prior works which would try to use multiple text prompts followed by contrastive learning to teach CLIP how to rank. Their solution is intuitive, elegant and can be readily adapted to any existing CLIP type framework without the need of fine-tuning the CLIP encoders. They show SoTA performance on a variety of learning-to-rank tasks such as facial age estimation, colored image dating, object counting etc. They also validate all their design choices in the ranking module with ablations and show abundant qualitative examples of their method. The appendix also covers interesting future directions such as ranking based on multiple attributes and analyzing how the transferability of ranking works across domains. Overall I think the paper is quite well written and provides a very elegant solution to the learning to rank problem and is well substantiated with experiments and ablations.", "questions": "1) Have you tried any experiments in ranking images by the number of objects in general, irrespective of the category of objects? For example just sorting on the \"number\" of whatever object is present in the image. That might help gauge how well the architecture is able to abstract out the \"count\" of objects.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1) The paper is very well written and it is clear to understand the method section. The idea of replacing multiple rank specific prompts in prior work with a single 'sort by x' is simple and well motivated and simplifies the architecture quite a bit. The use of cross-attention in both the ranking adapter and the auxiliary ranking module makes intuitive sense to obtain text-conditioned image features and to get \"pairwise image\" differences respectively. \n2) The experimental section is well detailed and results are shown on a variety of different benchmarks for ranking. The appendix also covers more examples of tasks. The ablation section also validates the design choices for the ranking modules quite well. Overall the results are impressive beating all existing methods on a wide range of tasks. \n3) The method proposed is light-weight as it does not require fine-tuning of CLIP encoders and only of the ranking modules and all the experiments were run on a single 3090Ti GPU, which is impressive. \n4) The paper discusses abundant qualitative examples across the different benchmarks and also discusses interesting future directions such as ranking across multiple attributes and the generalizability of ranking across benchmarks.", "weaknesses": "1) The paper only shows results on a single image backbone in the CLIP architecture (ConvNeXt-L). Also, it resizes images to 320X320 whereas most of the prior work (LR-CLIP, Ordinal-CLIP, NumCLIP) uses an image size of 224X224 and they use the ViTB/16 image encoder. It would be good to show results with this setting as it will ensure a fair comparison with prior baselines and it will also provide some insight into the generalizability of the ranking modules across different architecture types. \n2) The paper only shows results with the pairwise ranking loss. It might be interesting to look at how the results vary by using other losses like triplet loss or the ranked list loss as presented by Wang et al (2021). Using a ranked list loss might also allow to not use the auxiliary ranking module which computes the pairwise attention matrix for the image features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)