Learning-to-Rank Meets Language: Boosting Language-Driven Ordering Alignment for Ordinal Classification
Authors: Rui Wang
Pei Pei Li
Huaibo Huang
Chunshui Cao
Ran He
Zhaofeng He
Venue: NeurIPS 2023 poster (2023)
Keywords: Ordinal Classification
Representation Learning
Vision-Language
Prompt Learning
TL;DR: a novel approach to ordinal classification with language-driven ordering alignment

Abstract: We present a novel language-driven ordering alignment method for ordinal classification. The labels in ordinal classification contain additional ordering relations, making them prone to overfitting when relying solely on training data. Recent developments in pre-trained vision-language models inspire us to leverage the rich ordinal priors in human language by converting the original task into a vision-language alignment task. Consequently, we propose L2RCLIP, which fully utilizes the language priors from two perspectives. First, we introduce a complementary prompt tuning technique called RankFormer, designed to enhance the ordering relation of original rank prompts. It employs token-level attention with residual-style prompt blending in the word embedding space. Second, to further incorporate language priors, we revisit the approximate bound optimization of vanilla cross-entropy loss and restructure it within the cross-modal embedding space. Consequently, we propose a cross-modal ordinal pairwise loss to refine the CLIP feature space, where texts and images maintain both semantic alignment and ordering alignment.  Extensive experiments on three ordinal classification tasks, including facial age estimation, historical color image (HCI) classification, and aesthetic assessment demonstrate its promising performance.

Learning-to-Rank Meets Language: Boosting Language-Driven Ordering Alignment for Ordinal Classification
Rui Wang', Peipei Li Huaibo Huang", Chunshui Cao², Ran He", Zhaofeng He²
BBijjng University of Posts and Telecommunications *CRIPAC&MAIS, Institute of Automation, Chinese Academy of Sciences
"WATRIX.AI
[wr_ bupt, lipeipei, zhaofenghe| @ @bupt.edd.co
huaibo. @ eripac.ia.acccom chunshui. cao @ watrix.ai ai, rhe @nlpr.ia.c.cc
Abstract
We present a novel language-driven ordering alignment method for ordinal classi- fication. The labels in ordinal classification contain additional ordering relations, making them prone to overfitting when relying solely on training data. Recent developments pre-trained lisiua-language models inspire us leverage the rich ordinal priors in human language by converting the original task into vision- language alignment task. Consequently, we propose L2RCLIP, which fully utilizes the language priors from two perspectives. First, we introduce a complemen- tary prompt tuning technique called RankFormer, designed to enhance the or- dering relation of original rank prompts. It employs token-level attention with residual-style prompt blending the word embedding space. Second, to further incorporate language priors, we revisit the approximate bound optimization of vanilla cross-entropy loss and restructure it within the cross-modal embedding space. Consequently, propose cross-modal ordinal pairwise loss to refine the CLIP feature space, where texts and images maintain both semantic alignment and ordering alignment. Extensive experiments three ordinal classification tasks, including facial age estimation, historical color image (HCI) classification, and aesthetic assessment demonstrate its promising performance. The code is available at attnss/gggioo..oommmyaaag3331111
Introduction
Ordinal classification aims to predict labels that are related in natural or implied order, which can be considered as special case of ordinal regression after label discretization, i.e. discretize the continuous labels and each bin is then treated as a class. Common examples of such tasks are facial age estimation (e.g., estimating the facial age from to 100), historical color image classification (e.g.. assigning time period to color photographs, ranging from the 1930s to the 1970s), and aesthetics assessment(e.g. rating image quality on scale from "unacceptable" to "exceptional").
Compared with common classification, ordinal property of labels need to be additionally considered in ordinal classification. Many algorithms 35 12 employ the ordinal classification framework. which trains set of classifiers or integrates probabilistic priors to directly predict rank labels. However, these methods exhibit suboptimal performance as result of insufficiently harnessing the ordering properties. Order learning algorithms [48] [39 26 [2] demonstrate competitive performance by
effectively capturing relative ordering relationships. These approaches determine the target order of
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

novel instance by contrasting it with well-defined reference instances. Nonetheless, the algorithm'ss 's performance can be substantially compromised by inadequate indexing quality of the reference instances.
Furthermore, many metric learning techniques 41 R3
49 have been developed construct
ordinal embedding space in which the distances between diverse features effectively represent the differences in their respective ranks. However, all these methods learn ranking concepts depending solely on training data, which renders them vulnerable to overfitting 29
CLIP Feuture Space
CLIP Feature Space
CLIP Fosture Space
Text Features
R
Kank Templiite
Image Fentures
Rank Former
V/
R
Interpolation
R
R4
semantic ordering
semantic ordering
X
semantic ordering
Runk: 73 < K
- <35 Ki
Rank: < N
(a) CoOp
(b) OrdinalCLIP
(c) Ours
Figure 1: Comparison with CoOp(a) and OrdinalCLIP(t), where represents the image features and R r represents the rank templates word embedding space and CLIP feature space, respectively. (a) CoOp aligns each rank template with its corresponding images via contrastive loss but vanilla CLIP fails to ensure ordering aligmment. (b) OrdinalCLIP considers additional interpolation to explicitly maintain ordering alignment. However, the interpolation term not preserve semantic aligmment in CLIP feature space. (c) Our method enhance the ordering relation of vanilla rank templates while ensuring the semantic alignment of CLIP space.
Fortunately, recent developments in large pre-trained vision-language models offer insights for various visual tasks [32 27 29 25 56]. Compared with visual content, human language contains highly abstract concepts and rich semantic knowledge [29 32). Motivated by we attempt borrow knowledge from language domain by two major observations. Firstly, rank templates inherently contains ordinal information, c.g. "a sixty years old face" "a ten years old face", which is also demonstrated in Figure 3(a). Secondly, inspired by [25,56], information of numerous image features within well-aligned modal feature space, which can be considered as robust prior for cross-modal metric learning.
(6), rank features encapsulate the average
Hence, we propose L2RCLIP boost learning-to-rann of CLIP-based models for ordinal classifica- tion. Specifically, we first introduce complementary prompt tuning method, termed RankFormer to enhance the ordering relation of original rank templates. Specifically, RankFormer employs bisen-wise attention layer and performs residual-style prompt blending with the original templates for prompt tuning. Moreover, inspired by pairwise metric learning (2), we propose modsl ordinal pairwise loss to ensure both semantic and ordering alignment in the CLIP feature space. Concretely, we revisit the approximate bound optimization of conventional cross-entropy loss and reformulate in the cross- modsl embedding space with ordinal language priors. OrdinalCLIP [29] also incorporates language priors model ordering alignment, demonstrating impressive performance. However, our method distinguishes itself from previous works, as depicted in Figure 1 CoOp achieves semantic alignment through contrastive loss but fails to maintain ordering alignment. OrdinalCLLIP addresses ordering alignment at the cost of weakened semantic alignment. Conversely, by leveraging Rank- Former and cross- modal ordinal pairwise loss, our approach simultancously considers both semantic alignment and ordering alignment.
The contributions of this paper can be summarized follows: (1) We incorporate learning-to- -toning-to-rank into vision-languall pre-training model for ordinal classification, in which we present RankFormer enhance the ordering relation of vanilla language prompts. (2) We explicitly utilize the language priors and further propose cross-modal ordinal pairwise loss to refine CLIP embedding space, in which image features and text features maintain both semantic and ordering alignment.
(3) Extensive experiments demonstrate the competitive performance of L,RCLIP on age estimation, aesthetics assessment and historical image dating, as well as improvements in few-shot and distribution shift experiments.

---

2 Related Work
Ordinal Classification Ordinal classification attempts solve classification problems in which not all wrong classes are equally wrong. Early techniques [4, 45) adopted the classification framework and train set of classifiers performance due to ignoring the ordering relation. By incorporating probabilistic priors, Geng et al. [12] firstly proposed label distribution learning and assigned a Gaussian or Triangle distribution for an instance. The -ariince loss was introduced in [39] for learnable label distribution and penalizes the learned variance of estimated distribution to ensure a sharp distribution. Probabilistic embedding (28) was developed to model the data uncertainty for ordinal regression. Liu et al. (33 proposed predicting the ordinal targets that fall within a certain interval with high confidence. These methods learn better rank concepts and significantly reduce the model's overconfidence toward incorrect predictions. In contrast to them, our L2RCLIP only focuses on enhancing vision-language alignment without complicated probabilistic distribution assumption.
directly estimate the rank labels. These methods got degraded
solve the ordinal classification task from the perspective of metric learning. These methods exploit the pairwise ordering relation in embedding space, where the distance between different features reflects ordinal information. For example, Ordinal log-loss (OLL) was presented in [3] with an additional distance-awar weighting term for ordinal classification. RankSim [3] proposed a regularization loss to ensure the ordinal consistency between label and feature. Suárez et al. [49] ranked the embedding distances between pairwise rank differences of features. Another way for ordinal classification is order learning 31 [9]. which learns ordering relation by comparison between instances. It usually show more promising results as learning relative ordering relation is much easier than learning absolute ordering relation. Lim et al. (31 firstly proposed this concept and determined the ordinal information of an unseen instance by compared to some known reference instances. Lee& Kim el al. [9]] improved the quality of indexing to boost the performance. MWR 48) further proposed to learn continuous regression score from reference instance. However, these methods often depend solely on training data to learn rank concepts, potentially leading to overfitting. To mitigate these issues, we leverage rich priors in human language to learn language-driven rank concepts.
Furthermore, many techniques 3 41
49
23
24
22]
Vision-language Learning Vision-language pre-training(VLP) has significantly improved the per- formance on many downstream tasks by text and image matching, including segmentation 43 g ob-
ject detection [14, image retrieval 36 1. generation tasks [8] 40
25 5 and ordinal regression (2)].
CLIP 42) and ALIGN [5] proposed to embed images and texts into the same representation space with two separate encoders in a contrastive-baser approach. The experimental results show that the impressive "zero-shot" performance on downstream tasks, which show the power of language prior. Inspired by the recent advances in NLP, prompts and adapter- based tuning becomes prevalent in improving ability of CLIP. CLIP-Adapte [11 adds a right-weight module on top of image and text encoder. CoOp (55) proposes learn context prompts for image classification. Due to lack of ordinal property, these methods lead to degraded performance in ordinal classification. Considering the great potential of language prior, we propose to incorporate learning-to-rant into CLIP for ordinal classification.
3 Proposed Approach
3.1 Problem Formulation
Ordinal classification is a unique case of image classification where labels possess ordinal properties.
Mathematically, let 2; E X' denote the i-th input instance with i = 1, 2, ..., N,, y: E {ri, r2. ..., T'M with ordered ranks r> > r-s-1 > > r11 denote the ground truth value and ÿ represent the predicted rank by the network, where N represents the total number of instances, M represents the number of ranks, and > indicates the ordering between different ranks. Analogous to normal classification, ordinal classification aims to recover yi by encoding the image to feature with encoder & and then using a classifier (=) to compute probability p;- The predicted label 5s is the result with the highest probability py:- The classification probability can be calculated by:
((x:))
3

---

Rerio-ppeelfic Prompt
Ny P
CLIP Foate Space
Images
RyA phott your fxce
CdlP
Aavetioe
C concut
Proppie
intoyphaee
Enhaanee undering relation of vanilla rank prompts
(b) Ensure hoth semantic alignment and ordering aligument
Figure 2: An overview of the proposed I. L.RRLIP. We incorporate shaminngtt--rant into CLIP from two perspectives. First, RankFormer performs a token-level attention mechanism to enhance the ordering relation of vanilla rank prompts. Then, refined rank-specific prompts and randomly initialized context prompts are concatenated in the word embedding space and are sent to a text encoder to extract the corresponding text features. Moreover, we present two types of losses refine CLIP feature space by attraction and repulsion, respectively. Attraction refers to an asymmetrical contrastive loss and a tightness term attract paired images and text features while repulsion refers to a reweighting diversity term to ensure the ordering alignment.
P: exp(w: 23) 15=1 exp(w] zi)
To exploit ordinal information in language, ordinal classification can transformed into a vision- language alignment task. Specifically,
pre-trained CLIP image feature extractor Image(-) to extract features from input images: V = 2, = Image(x;)) For text features, we first construct hard rank templates R {R1, R2...., RM for given ordinal classification task. For each template, we convert into fixed-length tokens and then map them into 51 512-dimensional word embeddings. The language feature extractor Text(-) encodes the embeddings classifier weight r;. The process can be formulated as: F; W/; Text (Takeniezee Finally, we can calculate the prediction probability pa for rank with Eq
3.2 Proposed Method
Our objective is to integrate learning-to-rant into CLIP and learn language-driven rank concepts for ordering alignment while preserving original semantic alignment. Inspired by [16 27 we firstly learn rank concepts in the word embedding space and subsequently refine the CLIP feature space to maintain both semantic and ordering alignment. Specifically, we introduce RankFormer to enhance the ordering relation of the original language prompts. RankFormer employs token-wise attention layer for rank prompt tuning. To better utilize language prior, we reformulate the approximate bound optimization within cross-modal embedding space. Furthermore, propose a cross-modal ordinal pairwise loss to ensure ordering alignment. Moreover, randomly initialized global context prompts and an asymmetrical contrastive loss are adopted to ensure semantic alignment. The overall framework is illustrated in Figure 2
Rank-specific Prompts. As shown in Figure 3(a), rank templates in the vanilla CLIP contain some degree of ordinal information. However, substantial proportion, nearly half of the pairs, lack clear ordinality. One intuitive approach to enhance this ordinal information to further fine-tune these rank templates. However, implementing such strategy introduces two primary challenges. First, the performance of different ranks for ordinal classification varies significantly driven by the imbalanced training data, and certain ranks cannot even be trained due to insufficient training data in extreme cases. Second, the contrastive loss introduces difficulties in enhancing the ordering relation during the training [53]. To address the first challenge, OrdinalCLIP [2 trains only small number of base rank templates and generates other rank templates through explicit interpolation, which confines the ordinal information vanilla rank templates. this paper, we train RankFormer with token-wise attention to enhance the ordinal information of the fixed rank templates. Specifically, we denote tokenized rank templates as R R MxnxD where M and D represent the number of templates and word embedding channels, respectively. Note that we only consider the ranking tokens with length of n. Subsequently, we perform liken-wise attention for prompt tuning and apply residual-style prompt blending with the original prompts. The process can be formulated as: R' (1 - a) R + fEFN (fassffMaaaa (R)), where represents the residual ratio.

---

Cross-modal Ordinal Pairwise Loss. Since cross-entropy loss ignored the ordinal information during training [53]. we turn solution that recovers the ordering relation while maintaining semantic alignment. Inspired by the pairwise metric learning, we firstly revisit the vanilla cross- entropy loss from the perspective of approximate bound optimization. Boudiaf et al. 2 proved that minimizing the cross-entropy loss accomplishes by approximating lower bound pairwise cross-entropy loss LPCE. LPCE contains a tightness term and a diversity term, as follows:
N
N
K
Dj=1Ppk 2j
K
LPCE = 2AN
Σ
Σ 2:31 yt
2
log
erp
Σ
Hall (2)
N
AN
2A
TIGHTNESS
DIVERSITY
where 2, represents the image feature in the embedding space, Pij represents the softmax probability of point 2; belonging class j. C Ei=1 Pi represents the soft mean of class k and À R. is to make sure I.E a convex function with respect to encoder +w..
By incorporating the language priors, turn Eq. 2) to cross-modal pairwise cross-entropy loss. Specifically, as human language contains rich prior knowledge, text features can be considered as both hard mean ты N E and soft mean т N Ei=1 Pact; of image features at class k; where N& represents the sample number of class k and N represents the total sample number. Then, we reformulate the original pairwise cross-entropy loss LPCE as cross-modal pairwise cross-entropy loss LCPCE:
N
N
K
K
LOPCE = 2AN
log
erp
U
Tk
(3)
N
2A
k=1
k=1
TIGHTNESS
EISIIT
Inspired by we use meanNN entropy estimator The process is formulated as:
10
to estimate the diversity term in Eq.
D
diversity CPCE
Σ
(4)
N(N 1)
izi
N
CCCEE
tightness
x
1
Σ
Tya
(5)
where represents the feature dimensions. recover ordering relation, we propose an additional weighting term. Intuitively, each rank template will have a high similarity score with images of close rank and a low similarity score with images of distant rank. As such, we opt weight the cross-modal features with Wij where Wij are the distances in the label space. The final cross-modal ordinal pairwise loss is defined follows:
B
Leop(yi
Wysj (By, + yry)) - vyy Tup
jtt
where y controls the strength of the rank within rank templates and B represents the batchsize. To further refining the CLIP feature space, we also propose a simplified cross-modal ordinal pairwise loss Loopp with language-related parameters frozen(i.e. y 0).
Global Context Prompts. Given that global context prompts significantly surpass manually de- signed discrete prompts in vision tasks [55 27 54 we integrate them with our complementary rank-specific prompts in RankFormer to enhance semantic alignment. Specifically, we randomly initialize I. global context prompts, denoted as G = {G, G2, Gr). and concatenate them with rank-specific prompts the word embedding space of CLIP.
Asymmetrical Contrastive Loss. In vanilla CLIP 42 models are optimized using the standard contrastive loss, including text-image contrastive loss T.22 and an image-text contrastive loss Lyze- In this work, we replace the original contrastive loss with an asymmetrical contrastive loss due to
5

---

many tany image-text mappings within batch. In other words, different images in batch may have the same rank, and the same rank may have the same description. Therefore, we replace t, with ly in both text-image contrastive loss Te2; and image-text contrastive loss T,21- Specifically, our improved asymmetrical contrastive loss defined as follows:
exp(u (
(ue)
Σ
log
(3,))
E1=1 crp(u) /
sEZ(8.)
where 7 is the temperature parameter and Z(y+) = f E... :: yy²--
3.3 Loss functions
We employ two-stage training scheme refine approach following 27 In the first stage, focus to enhance the ordinal information of the vanilla rank templates. We utilize cross-modal ordinal pairwise loss Leop and the asymmetrical contrastive losses Lez and Ięze- In the second stage, our objective is to simultaneously improve both semantic alignment and ordering alignment within the CLIP latent space. We use the simplified cross-modal ordinal pairwise loss Locop and the cross-entropy loss Lee. More detail can be found in Sec. 4.1
Experiments
4.1
Implementation details
We adopt the ViT-B/16 visual encoder and the text encoder from CLIP [42] as the backbone for our image and text feature extractor. All training data is resized 224 224 and subjected to random horizontal flipping. For all experiments, we employ the Adam 17 optimizer with default settings. The learning rates for the RankFormer and the visual backbone are 3.5 x 10 and 10 respectively. The model is trained for 20 epochs in the first stage and 40 epochs in the second stage, with the learning rate decayed by factor of 0.1 in epoch 30. We L 5 most of our experiments. For MORPH and CL.AP201 we set T.22, L,zt, and Teoop with weights of 0.03, 0.03, and 3 for the first stage, and Teee and Tssop with weights of and for the second stage. For other datasets, we set L,2i, Lizi, and Loop with weights of 0.1, 0.1, and 3 for the first stage, and Lee and T.scop with weights of and for the second stage. All experiments are conducted on a single NVIDIA 3090 GPU.
4.2 Age Estimation
Table Results on MORPH and CLAP2015.
Table 2: Results on Adience dataset.
Morph
CLAP2015
Adience
Methods
Methods
MAE())
MAE())
Accuracy (%,
MAE())
AGEn 50
2.52
2.94
OR-CNN 37
56.7
0.54
BridgeNet
60
2.38
2.87
CNNPOR 34
57.4 + 5.8
0.55 0.08
AVDI 52
2.37
GP-DNNOR 35
57 4
+ 5.5
0.54 + 0.07
POE 28
2.35
SORD
59.6 + 3.6
0.49 0.05
DRC-ORID 19
2.26
POE 28
1 4.4
0.47 0.06
PML 6
2.15
2.91
MWR 48
62.6
0.45
MWR 488
2.13
2.77
GOL 20
62.5
0.43
Vanilla CLIP 42
6.91
4.66
Vanilla CLIP 42
43.3 - 3,6
0.80 + 0.02
CoOp 55
2.39
2.75
CoOp 55
60.6 + 5.5
0.50 1: 0.08
OridinalCLIP
20
2.32
OridinalCLIP 29
61.2 + 4.2
0.47 1 0.06
L2RCLLIP(OurS) 2.13
2.62
1.2CCLI((Our)
66.2
0.36 0.05
Datasets. Age estimation aims to predict the age of given facial image. We train and evaluate our
method on the widely-used MORPH [4 dataset, CLAP2015 99] dataset, and Adience [2]] dataset.
MORPH II 44 is one of the largest and most commonly used longitudinal face databases and we
follow the widely adopted training and evaluation following [55] 28 48 47]. CLAPP015 [9] used
for apparent age estimation, where each image rated by at least 10 annotators, and the mean rating

---

is set as the ground truth. Following 48
we split it into 2,476 for training. 1,136 for validation, and 1,079 for testing Adience [2] contains discrete labels annotated with eight age groups. We adopt the same five-fold cross volsdvvlliotion protocol used in [2I]. For evaluation metric, we use the mean average error (MAE) to measure the absolute differences between the ground truth labels and the predicted ones. Classification accuracy is additionally adopted for Adience. For detailed experimental settings, please refer to the supplementary material.
Comparison with State-of-the-art Methods. In Table 1, our L2RCLIP outperforms both con- ventional algorithms and language-power powered algorithms in all tests. For results on the Morph II, compared with conventional algorithms, L2RCLIP achieves state-of-the-an performance MAE at 2.13, verifying the significance of leveraging rich priors human language. Furthermore, compared to methods utilizing language priors, our L2RCLIP exhibits substantial performance improvements. CoOp significantly enhances the vanilla CLIP's performance by learning global context prompts for semantic alignment. Subsequently, OrdinalCLIP further reduces MAE by 0.07 through fixed interpolation for ordering aligmment. Nonetheless, a considerable margin remains compared to our methods, which validates the effectiveness of approach.
Table also compares the results on CLAP2015, Due to the great challenge, many previous methods However, without using such schemes, 1.2RCIIP outperforms all conventional algorithms. Compared with language-guis guidded models, our L2RCLIP also shows much better results on MAE, specifically significant MAE margin of 0.15 evaluated on the test set. Table 2 shows the comparison results on Adience [2I] using the metrics of MAE and Accuracy. Compared with Morph II and CLAP2015, Adience is used for age group estimation. The underline indicates the mean value reported in the original paper. Our method outperforms the state-of- of-the-art algorithms by significant gaps of 5.7% in accuracy and 0.07 in MAE.
adopt additional boosting schemes 30 50
(a) Vanilla CLIP
(b) CoOp
(e) OrdinalCLIP
(d) (2RCLiP(OUN))
Figure 3: The similarity matrices of rank templates in different methods. The redder, the more similar the pair of rank templates. The percentages of templates pairs that obey the ordinality are: 55.36% 59.92%, 65.94 65.94% and 71.87%, respectively.
Ordinality of Learned Rank Templates. Following 29
report the ordinality score by mea- suring the cosine similarity between rank templates. The ordinality scoreJis the percentage of rank template pairs that obey the ordinal property. Figure regureBreports reports the ordinality scores in different meth- ods. Vanilla CLIP in Figure 3(a) contains a certain degree of ordering relations, where more than half of the rank template pairs show correct ordering relation. Moreover, CoOp Figure 3(b) introduces additional global context prompts and improves the ordinality score by 4.56%. OrdinalCLIP in Figure 3(c) adopts an explicit interpolation strategy and further improves the ordinality by 6.02%. However, there is noticeable "red stripe" in the upper right corner, indicating that this section of rank template pairs significantly violates ordinality. We believe that explicit interpolation effectively ensures local ordinal properties, but may fail to preserve global ordinal properties. Our methods greatly alleviate this problem and achieve higher ordinality score. The "red blob" in the lower right corner results from insufficient samples for old individuals. Therefore, avoid the model's overconfidence towards incorrect predictions, this part of rank templates should exhibit higher similarity while ensuring ordering alignment. By comparison, our method outperforms previous methods.
Few-shot Learning. Table 4 demonstrates the generalization ability of L2RCLIP for few-shot learning. Following (29]. the full dataset is split into 80% for training and 20% for testing. The entire test set used for validation, and only 02//4/8/16/32/64 samples in the training set from each class of labels are chosen for training. We observe that introducing rank information in CLIP
i.c., OS(%) = EM EM- T(a.. a1,+11 (MM x (M - 1)/2), where the number templates and st represents the cosine similarity of each pair of templates.
7

---

Table 3: We report the MAE results under few-shot settings on the MOPRH II dataset.
Shots
32
64
CoOp 55
5.09
4.50
3.81
3.57
3.23
2.87
2.61
OrdinalCLI 29
4.94
4.36
3.55
3.31
3.07
2.76
2.57
L2RCLIP(Ours)
4.54
3.92
3,40
3.28
2.81
2.55
2.38
actually benefits few-shot learning tasks. Compared with CoOp. both OrdinalCLIP and our L.2RCLIP achieve significant improvements across all settings, which verifies the effectiveness of ordering alignment. Moreover, our L2RCLIP further reduces the average MAE by 0.24, which demonstrates the effectiveness of the proposed learning-to-rann method in CLIP.
Table 4:
The MAE results under the distribution shift setting on the MOPRH II. "re cls" denotes the number of reduced classes, and "re smp" means the percentage of reduced sampled one class.
cls smp
10-80
10-90
20-80
20-90
30-80
30-90
40-80
40-90
CoOp 55
2.71
2.85
2.98
3.51
3.06
3.36
2.99
3.30
OrdinalCLIP 29
2.61
2.67
2.77
3.06
2.86
3.21
2.84
3.12
L2R)L
2.28
2.30
2.37
2.43
2.51
2.61
2.68
2.79
Distribution Shift. Following 29] we conduct data distribution shift experiments on the MORPH II dataset for generalization. We use the same setting as the general regression setting, For the training set, we randomly choose several rank labels, e.g., 10, 20, 30, and 40. Then, those classes, we randomly discard some portion of training data, e.g., 80 and 90. We report our experiments in Table For the most severe settings, CoOp. OrdinaICLIP. and our L2RCLIP methods incur performance losses of 42.24%, 34.49% and 30.98%, respectively, which demonstrates that our approach exhibits superior robustness when faced with data distribution shift problems. For all shift settings, our method shows better performance than OrdinalCIIIP which highlights the effectiveness of the proposed learning-to- taaming-to-rank method CLIP compared with interpolation.
Table 5: Ablation Study of L2RCLIP on the MORPH and CLAP2015.
Ablation
Choices
RankFormer
Leop
Lueop
MAF(L)
2.50
2.48
2.37
2.38
2.21
2.29
2.31
2.13
Morph II
OS(%.1)
55.89
57.58
66.15
55.89
66.15
71.87
57.58
71.87
MAE(()
2.75
2.68
2.71
2.71
2.70
2.65
2.66
2.62
CLAP2015
OS(%.1)
53.33
54.77
57.33
53.33
57.33
67.55
55.25
67.55
4.3 Analysis
Ablation Study. In this section, conduct ablation study to examine the respective roles of each component in L2RCLIP, and the results are reported in Table 5. Three key observations be drawn from Table First, each proposed component demonstrates improvements over the baseline model and complements one another. Second, although RankFormer marginally enhances the ordinal information of vanilla rank templates, its performance remains unsatisfactory compared to other methods when only contrastive loss is utilized for training. This result further highlights the importance of our cross-modal pairwise loss refining the CLIP feature space. Third, fine-tuning rank templates without RankFormer significantly impairs performance on CLAP2015, potentially due to imbalanced training for certain rank templates. This finding underscores the effectiveness of RankFormer in modeling the ordering relation derived from the complete set of rank templates.
we aim to enhance the ordering relation vanilla rank templates, essential to examine the effects of initialization. We report our results in Table 6 Although our
Initialization Impact.

---

Table 6: Initialization Impact Analysis.
Initialization
MAE(U)
OS(%, 1)
photo of [age] years old face. Age extimation: person the age of [age]-
2.13
71.87
2.14
72.07
The age of the person in the portrait is {age].
2.12
71.32
The age of the person is [age]-
2.16
71.52
The age of the face is (age).
2.14
71.08
Mcan/Sid
2.14/0.01
71.57/0.40
methods fix the original rank templates, different initializations lead to similar convergence and
performance with low standard deviation value of 0.014. This observation further substantiates the robustness of our methods against diverse initializations.
Table 7: Results on Morph, CLAP2015 and Adience datasets.
Morph (MAE, 4)
CLAP2015 (MAE, 4)
Adience
Methods
(Aceuracy, t)
(MAE, 4
1.2RCLIP-I
2.19
2.78
62.9 + 5.5
0.42 0.06
L2RCLIP (Ours)
2.13
2.62
68.2 + 7.2
0.36 + 0.05
Compared with interpolation- based method. To further prove the effectiveness our proposed method, compare L2RCLIP with previous interpolation-base -aased method (e.g. OrdinalCLIP([9] We adopt the same setting except the process ordinality learning and we term it as L2RCLIP-L. We report the results the aging dataset. More experiments for few-shot learning and distribution shift and the implementation details setting for L2RCLIP-I can be found in appendix.
As illustrated in Table 7 our method outperforms interpolation-basee methods with significant margin in experiments involving a large number of rank categories. This outcome attributable to the challenge posed by direct interpolation methods in modelling complex ordering relationships. Our approach continues to surpass interpolation basee methods even experiments with a smaller number of rank categories. Collectively, these experiments corroborate the effectiveness of the methods proposed in this study.
4.4 Image Aesthetics Assessment
Datasets.
CrowdBeauty [46 consists of 13,929 available Flickr photos across four categories: nature, animal, urban, and people. The aesthetic quality each image evaluated using five absolute rating scales: "unacceptable"" "flawed", "ordinary", "professional" and "exceptional".. Following previous methods [29 20), we select 80% the images for training and the rest for testing. Five-fold cross-validation is employed for fair comparisons. Both the mean MAE and accuracy are reported.
Table 8: Results on Image Aesthetics dataset.
Accuracy X, f)
MAR(L)
Methods
Nature
Animal
Urban
People
Overall
Nature
Animal
Urban
People
Overall
CNNPOR 24
71.86
69.32
69.09
69.94
70.05
0.294
0.322
0.325
0.321
0.316
SORD 7
73.59
70.29
73.25
70.59
72.03
0.271
0.308
0.276
0.309
0.290
POE 28
73.62
71.14
72.78
72.22
72.44
0.273
0.299
0.281
0,293
0.287
GOL. 20
73.8
72.4
74.2
69.6
72.7
0.27
0.28
0.26
0.31
0.28
Vanilla CLIP 42
65.24
45.67
58.78
53.06
55.68
0.461
0.557
0.468
0.524
0.502
CoOp 555
72.74
71.46
72.14
69.34
71.42
0.285
0.298
0.294
0.330
0.302
OridinalCLIP
29
73.65
72.85
73.20
72.50
73.05
0.273
0.279
0.277
0.291
0,280
(22CCIIP(Our))
73.51
75.26
77.76
78.69
76.07
0.267
0,253
0.216
0.246
0.245
Results. Table g presents our results on the CrowdBeauty dataset. Aesthetic score prediction is challenging due to the subjectivity and ambiguity of aesthetic criteria; however, our 1,2RIIIP demonstrates sfate-of-the-art performance on most experimental settings by fully exploring learning- to-rank with language prior. Compared with OrdinalCLIP, L2RCLIP improves accuracy by 3.02% and

---

reduces MAE by 0.35 overall, which further verifies the effectiveness of our proposed learning-to-ee to-rank method. When compared with the best approach without language prior, L2RCLIP improves overall MAE by 0.35 and overall accuracy by 3.37%. Consistent improvements are observed across all categories compared to previous methods, showeasing the effectiveness of our proposal exploiting language ordinal information.
4.5 Historical Image Dating
Datasets.
The historical image dating dataset 38 serves as a benchmark for automatically predict- ing the decade of historical colored images. The dataset comprises five-decade categories, ranging from the 1930s to the 1970s. Following [34 38 we adopt the same train-test split and ten-fold cross-validation. Both the mean and standard deviation for MAE and accuracy metrics are reported.
Results. Table 9 showcases improvements com- pared to other state-of-the art models using the full dataset. Initially, zero-shot CLIP exhibits subopti- mal performance resulting from inadequate seman- tic and ordinal alignment. CoOp enhances the re- sults by incorporating global context prompts to fa- cilitate semantic alignment. Furthermore, Ordinal- CLIP exploits interpolation to improve the average MAE by 0.09. Our L2RCLIP further advances the average MAE by 0.25 with a lower standard devia- tion. In comparison with the previous conventional model, L2RCLIP achieves new state-of- the-art per- formance with an MAE of 0.43 and an accuracy of 67.22%, thereby validating the effectiveness of proposed method.
Table 9; Results on HCI dataset.
HCI
Methods
MAB(L)
Accuracy(%6 f)
CNNPOR 34
0.82 = 0.05
50.12 1. 2.65
GP-DNNOR 33
0.76 0.05
46.60 + 2.98
POE 28
0.76 = 0.04
54.68
3.21
MWR 48
0.58
57.8
GOL 20
0.55
56.2
Vanilla CLIP
42
1.01 = 0.03
30.41
3.32
CoOp 55
0.76 = 0.06
51.94 + 2.60
OridinalCLIP 29
0.67 = 0.03
56.44 = 1.66
L2RCLIP(OUrs)
0.43 = 0.02
67.22 1.59
5 Discussions and Conclusions
In this paper. we propose L2RCLIP to boost leamning-to-re to-rnkkkto-a of CLIP for ordinal classification. Specifically, we introduce a complementary prompt tuning method, termed RankFormer, to enhance the ordering relation the original rank prompts. performs token-level attention and residual-style prompt blending for prompt tuning. Additionally, we revisit the approximate bound optimization of cross-entropy and reformulate it in the cross-modal embedding space by incorporating the language knowledge. To additionally recover the ordinal information, we further introduce cross-modal ordinal pairwise loss to refine the ordering alignment of CLIP feature space. Extensive experiments demonstrate the effectiveness of our approach on various ordinal classification tasks, including facial age estimation, historical image dating, and image aesthetics assessment. Furthermore, L2RCLIP outperforms in the challenging few-shot learning and data distribution shift learning scenarios. Lastly, we conduct a comprehensive ablation study to verify the effectiveness of each component of our proposed method.
Broader Impacts. As a versatile approach, L2RCLIP can be applied to any ordinal classification tasks such image aesthetics assessment other rank assessment. However, these tasks may pose risk of unlawful surveillance or invasion of privacy if abused. Meanwhile, as L2RCLIP is based on a large-scale vision-language model, addressing demographic biases in pre-trained vasion-language models is of significant importance. Therefore, we emphasize that 1,RCLIPP represents a research proof of language-driver learning and is not appropriate for real-world usage without strict technical controls.
Acknowledgement. This research is sponsored by National Natural Science Foundation of China (Grant No. 62306041, 62006228), Beijing Nova Program (Grant No. 20230484488, 20230484276), and Youth Innovation Promotion Association CAS (Grant No. 2o.2022132).
106,
10

---

References
[1] Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Effective conditioned and composed image retrieval combining clip-based features. In CVPR, 2022.
[]] Malik Boudiaf. Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and
Ismail Ben Ayed. unifying mutual information view of metric learning: cross-cntropy vs. pairwise losses.
In ECCV, 2020.
[3] François Castagnos, Martin Mihelich. and Charles Dognin. A simple log-based loss function for ordinal
text classification. In COLING, 2022.
[4] Shixing Chen, Caojin Zhang, Ming Dong, Jialiang Le, and Mike Rao. Using ranking-cnn for age estimation.
In CVPR, 2017.
[5] Xing Cui. Zekun Li. Peipei Li. Yibo Hu, Hailin Shi, and Zhaofeng He. 12edit: Towards multi-turn
interactive image editing via dialogue. 2023.
[6] Zongyong Deng, Hao Liu, Yaoxing Wang. Chenyang Wang. Zekuan Yu, and Xuchong Sun. Pml: Progres- sive margin loss for long-tailed age classification. In CVPR, 2021.
[7] Raul Diaz and Amit Marathe. Soft labels for ordinal regression. In CVPR, 2019.
[8] Xiaoyi Dong. Yinglin Zheng. Jianmin Bao, Ting Zhang, Dongdong Chen, Hao Yang. Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, et al. Maskclip: Masked self-distillation advances contrastive language-image pretraining arXiv preprint arXiv:2208. 12262, 2022.
[9] Sergio Escalera, Junior Fabian, Pablo Pardo, Xavier Baró, Jordi Gonzalcz, Hugo Escalante, Dusan Misevic, UIrich Steiner, and Isabelle Guyon. Chalcam looking at people 2015: Apparent age and cultural event recognition datasets and results. In ICCVW, 2015.
[10] Lev Faivishevsky and Jacob Goldherger. Ica based on a smooth estimation of the differential entropy. In NewrIPS, 2008
[11] Peng Gao, Shijic Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint or arXiv:2 2021.
[12] Xin Geng, Chao Yin, and Zhi- ZhiiHua Zhou. Facial age estimation by learning from label distributions. TPAMI, 2013.
[13] Yu Gong, Greg Mori, and Frederick Tung. Ranksim: Ranking similarity regularization for deep imbalanced regression. In ICML., 2022.
[14] Xiuye Gu. Tsung-Y Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint A00iv:2104. 13321, 2021.
[15] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling visual and vision-language representation learning with noisy text supervision. In /CML, 2021.
[16] Bingyi Kang, Saining Xie, Marcus Rohrhach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalan- tidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arviv 1910.09217, 2020.
[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.666 1412.6980, 2014.
[18] Gihyun Kwon and Jong Chul Ye. Clipstyler: Image style transfer with single text condition. In CVPR, 2022.
[19] Seon-Ho Lee and Chang-Su Su Kim. Deep repulsive clustering of ordered data based on order-identity decomposition. In ICLR, 2021.
[20] Seon-Ho Lee, Nyeong Ho Shin, and Chang-Su Kim. Geometric order learning for rank estimation. In NewrIPS, 2022.
[21] Gil Levi and Tal Hassncer. Age and gender classification using convolutional neural networks. In CVPRW, 2015.
[22] Peipei Li, Yibo Hu, Ran He, and Zhenan Sun. Glohal and local consistent wavelet-domain age synthesis. TIFS. 14(11):2943-295 2019.
[23] Peipci Li, Yibo Hu, Xiang Wu, Ran Hc, and Zhenan Sun. Deep label refinement for age estimation. PR, 2020.
[24] Peipei Li, Huaibo Huang, Yiho Hu, Xiang Wu, Ran He, and Zhenan Sun. Hierarchical face aging through disentangled latent characteristics. In ECCV, 2020.
[25] Peipei Li, Rui Wang, Huaibo Huang, Ran He, and Zhaofeng He. Pluralistic aging diffusion autoencoder. In ICCV, 2023.
[26] Qiang Li, Jingjing Wang, Zhaoliang Yao, Yachun Li, Pengju Yang, Jingwei Yan, Chunmao Wang. and Shiliang Pu. Unimoodal-comeentraued loss: Fully adaptive label distribution learning for ordinal regression.
In CVPR, 2022.
[27] Siyuan Li, Li Sun, and Qingli Li. Clip-reid: Exploiting vision-language model for image re-identification without concrete text lahels. arXiv preprint arXiv:22/ 1. 13977, 2022.
[28] Wanhua Li, Xiaoke Huang, Jiwen Lu, Jianjiang Feng. and Jie Zhou. Learning probabilistic ordinal
embeddings for uncertainty-aware regression. In CVPR, 2021.
[29] Wanhua Li, Xiaoke Huang, Zheng Zhu, Yansong Tang, Xiu Li, Jie Zhou, and Jiwen Lu. Ordinalclip:
Learning rank prompts for language-guided ordinal regression. In NeurIPS, 2022. [30] Wanhua Li. Jiwen Lu, Jianjiang Feng. Chunjing Xu, Jie Zhou, and Qi Tian. Bridgenet: A continuity-awar probabilistic network for age estimation. In CVPR, 2019.

---

[31] Kyungsun Lim, Nyeong-Ho Shin, Young- Yoon Lee, and Chang-Su Su Kim. Order learning and its application to age estimation. In ICLR, 2020.
[32]
Zhiqiu Lin, Samuel Yu, Zhiyi Kuang, Deepak Pathak, and Deva Ramanan. Multimodality helps unimodality: Cross-modal few-shot learning with multimodal models. In CVPR, 2023.
[33] Liang Liu, Hao Lu, Haipeng Xiong, Ke Xian, Zhiguo Cao, and Chunhua Shen. Counting objects by blockwise classification. TCSVT. 2019.
[34] Yanzhu Liu, Adams Wai Kin Kong, and Chi Keong Goh. constrained deep neural network for ordinal
regression. In CVPR, 2018.
[35] Yanzhu Liu, Fan Wang. and Adams Wai Kin Kong. Probabilistic deep ordinal regression based on gaussian
processes. In ICCV, 2019.
[36] Haoyu Ma, Handong Zhao, Zhe Lin, Ajinkya Kale, Zhangyang Wang, Tong Yu, Jiuxiang Gu, Sunav Choudhary, and Xiaohui Xie. Fi-clip: Entity-aware interventional contrastive learning for e-commerce modal retrieval. In CVPR, 2022.
[37] Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. Ordinal regression with multiple output
enn for age estimation. In CVPR, 2016.
[38] Frank Palermo. James Hays, and Alexei A Efros. Dating historical color images. In ECCV, 2012.
[39] Hongyu Pan, Hu Han. Shiguang Shan, and Xilin Chen. Hean-variance loss for deep age estimation from a face. In CVPR, 2018.
[40] Or Patashmik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven
manipulation of stylegan imagery. In ICCV, 2021.
[41] Gorkem Polat, Ilkay Ergene, Haluk Tarik Kani, Yesim Ozen Alahdab, Ozlen Atug, and Alptekin Temizel. Class distance weighted cross-entropy loss for ulcerative colitis severity estimation. arXiv preprint arXiv:2202.08 /67, 2022.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML, 2021.
[43] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In CVPR, 2022.
[44] Karl Ricanek and Tamirat Tesafaye. Morph: A longitudinal image database of normal adult age progrrrressio..
In FGR, 2006.
[45] Rasmus Rothe, Radu Timofte, and Luc Van Gool. Dex: Deep expectation of apparent age from a single image. In ICCVW. 2015.
[46] Rossano Schifanella, Miriam Redi, and Luca Maria Aicllo. An image is worth more than thousand favorites: Surfacing the hidden beauty of flickr pictures. In AAAI, 2015. [47] Wei Shen, Yilu Guo, Yan Wang. Kai Zhao, Bo Wang, and Alan I. Yuille. Deep regression forests for age estimation. In CVPR, 2018
[48] Nycong-Ho Shin, Soon-Ho Lee, and Chang-Su Kim. Moving window regression: a novel approach to ordinal regression. In CVPR, 2022.
[49] Juan Luis Suárez, Salvador García, and Francisco Herrera. Ordinal regression with explainable distance metric learning based on ordered sequences: Extended abstract. In DSAA, 2021.
[50] Zichang Tan, Jun Wan, Zhen Lei, Ruicong Zhi, Guodong Guo, and Stan Z Li. Efficient group-n n encoding
and decoding for facial age estimation. TPAMI, 2017.
[51]
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008.
[52] Xin Wen, Biying Li, Haiyun Guo, Zhiwei Liu, Guosheng Hu, Ming Tang, and Jinqiao Wang. Adaptive
variance based label distribution learning for facial age estimation. In ECCV, 2020.
[53] Shihao Zhang, Linlin Yang. Michael Bi Mi, Xiaoxu Zheng, and Angela Yao. Improving deep regression with ordinal entropy. arXiv preprint arXiv:2301. 08915, 2023.
[54] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for
vision- laiguala models. In CVPR, 2022.
[55] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 2022.
[56] Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan L.,, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui
Xu, and Tong Sun. Lafite: Towards language-free training for text-to-image generation. In CVPR, 2022.
12

---

Appendix
This supplementary material begins with a comprehensive visualization of the datasets central to our study. The specifics of our experimental settings are subsequently outlined in Section 6.2 Section 6. features an expanded analysis, including results from ablation studies. A key highlight of this section is the visual interpretation of the CLIP image features facilitated by t-SNE [5]]. Concurrently, comparative analysis is conducted, comparing the efficacy of interpolation- strategies with our learning-based methods(i.e. L2RCLIP).
6.1 More Analysis of L2RCLIP
(a) Vanilla CLIP
(b) *RankFomerr
(c) FRankFormer & heop
Leop & Lygoop
(c)
Figure 4: Visualizing the ablation effects on the MORPH II dataset: t-SNE visualizations of 512D
spaces in CLIP latent space.
Additional Ablation Study. Figure presents the embedding spaces corresponding to various ablation settings, mapped from the original 512D embedding spaces via t-SNE (51). In Panel (a), the vanilla CLIP implementation reveals sub-optimal ordering relationship among images of distinet ranks. Feature demarcations across different ranks ambiguous, displaying considerable overlap. The incorporation of RankFormer with global context prompts [55]. illustrated in Panel (b), aids improving and consolidating the order alignment of these image features. As shown in Panel (c), the implementation of our proposed loss function, Leop, further enhances this order alignment. Conversely, Panel (d) indicates that without the RankFormer module, the order alignment between the lowest and highest ranks falls short of the desired outcome. These findings substantiate the efficacy of the modules proposed in this study, with each component playing a sigmificant role in the overall model performance. This underscores the criticality of their synergistic implementation.
Table 10: Batchsize Analysis.
Batchsize
16
32
64
96
128
MAB()
2.15
2.13
2.13
2.17
2.17
OS(%, t)
67.21
68.55
71.87
71.71
75.42
We also explore the role of different batchsize settings. We report the result in Table 10 the result, choose batchsize with 64 to conduct other experiments.
According
Table 11:
The MAE results under the distribution shift setting and few shot setting on the MOPRH II.
re cls re smp
10-90
20-80
20-90
30-80
30-90
40-80
40-90
1.2RCLIP-I
2.39
2.45
2.50
2.57
2.70
2.73
2.93
L2RCLIP(Ours)
2.30
2.37
2.43
2.51
2.61
2.68
2.79
#Shots
#2
#8
#16
#32
#64
L2RCLIP-I
4.31
4.02
3.63
3.48
3.13
2.80
2.62
L2RCLIP(Ours)
4.54
3.92
3.40
3.28
2.81
2.55
2.38
Compared with Interpolation-based method. technique for our basic rank templates, which we term 1.2RCIIP-I. Two distinguishing characteristies set L2RCLIP-I apart from OrdinalCLIP [29]. Firstly, utilize the VIT-B/16 visual backbone of
In this study, we also propose an interpolation
13

---

CLIP for image feature extraction, whereas OrdinalCLIP employs pre-trained VGG-16 network supplemented by a linear projection layer. Secondly, our method relies on two-stage training strategy, in contrast to the one-stage approach adopted by OrdinalCIIP Importantly, both training strategies require a comparable time commitment. The results are reported Tables and 1T
Local ordinality score of L2RCLIP. To further prove our L2RCLIP have learned better ordering relationship, follow OrdinalCLIP(29 and use the local ordinality score. The formula is defined as: - Σ}= [[s.. st,+i) // (K (K 1)/2), where is the size of local window and j represents the cosine similarity of each pair of templates. We propose that the locally linear manifold can be preserved within a fixed small window size. Therefore, we calculate the local ordinality score using window sizes of 2, 4, 8, 16, and 32. The results of the local ordinality score shown in T2
Table 12: The local ordinality score results on the MORPH II dataset.
# window size
2
4
8
16
32
Vanilla CLIP
100.00
83.33
78.57
70.83
60.08
OrdinalCLIP((9) 100.00
100.00
100.00
96.19
L2RCLIP(Ours)
100.00
100.00
100.00
100.00
97.78
Table 13: Ablation study of global context prompts and architecture.
Method
Morph(MAE)
Morph(OS%)
CLAP2015(MAE) CLAP2015(OS%)
Vanilla CLIP
6.91
55.36
4.66
52.51
w/o context prompt
2.23
65.46
2.76
67.17
RankFormer-> MLP
2.27
67.48
L2RCLIP(Ours)
2.13
71.87
2.62
67.55
More ablation study of L2RCLIP To avoid the effect of token mixing and the type of architecture of RankFormer, we conduct more detailed ablation study. The results are shown in TableT3 Our proposed method is complementary to previous prompt tuning methods and our L2RCLIP can achieve comparable performance with OrdinalCIIIP even without context prompt. We have also compared I.2RCLIP performance with an ML.P-based architecture avoid effects driven by extra computation. Note that both RankFormer and MIP have similar training parameters. The results show that our token-wise RankFormer can enhance the ordinality between input rank templates.
Table 14: Additional results on Morph II.
Methods
Setting A
Setting B
Setting C
Setting D
DRC-ORID 19
2.26
2.51
2.58
2.16
OLBII
2.41
2.75
2.68
2.22
MWR-G
48
2.24
2.55
2.61
2.16
GOL
2.17
2.60
2.51
2.09
1.2RCLIP(Ours)
2.13
2.53
2.56
1.95
More results on Morph II datasets We have conducted experiments on the other three settings of Morph II. The results are presented in Table 14 The details for each settings follows:
Setting A: A total of 5,492 images of Caucasians are sampled and then randomly divided into training and test sets with a ratio of
Setting B: Approximately 21,000 images of Caucasians and Africans are randomly selected,
ensuring a balanced ratio
between Caucasians and Africans, as well as a ratio of 3 between females and males. The dataset is then divided into three subsets (S1, $2 S3). The training and testing process is repeated twice: 1) training on SI and testing on S2+S3, and 2) training on S2 and testing S1+S3.
14

---

The entire dataset is randomly partitioned into five folds, with the constraint that images of the same person belong only one fold. The 5-fold cross-validation is then performed.
Setting C:
Setting D: The entire dataset is randomly divided into five folds without any restrictions. The 5-fold cross-validation is then performed.
6.2 Experiment settings
Dataset Details. In the scope of this study, we only utilize publicly available data. To provide a
comprehensive understanding of the tasks at hand, illustrate selection of random samples from the image aesthetics assessment dataset (Figure 5 and the historical image dating dataset (Figure 6) To further enhance exposition, Figure depicts both the original and adjusted distributions of the MORPH II dataset.
(a) Exceptional.
(d) Flawed. Figure 5: Samples from the urban collections of the aesthetics dataset.
(b) Professional.
(c) Ordinary.
(e) Unacceptable.
U.S.N
(a) 1930s.
(a) 1940s.
(a) 1950s.
(a) 1960s.
(a) 1970s.
Figure Samples from the historical image dating dataset.
Original
0.08
0.08
0.08
test
test
test
0.06
0.06
0.06
Froquercy
Prqqurrry
0.01
00
0.01
0.00
0.00
20
30
40
50
60
70
30
20
30
40
50
60
70
80
20
30
40
50
60
70
80
Aga
Age
Age
20_
Recis_20 pp_0.9
Recls_30
0.00
0.00
0.05
lest
levil train
lest
0.36
0.06
0.05
train
froouenc
Frequency
0.30
0.00
0.00
20
30
40
50
50
70
50
20
30
40
50
60
70
80
20
30
40
50
60
70
80
Age
Aige
Age
p.
Rects,
0.08
0.08
0.05
test
test
test
0.00
tran
0.08
train
0.05
train
Frepancy
Freq
0.02
0.00
0.00
20
30
60
50
00
70
30
20
30
40
50
60
70
80
20
30
40
50
60
70
80
Agge
Age
Age
Figure 7: Original and shifted distributions of the MORPH II dataset.
15


Reviews:
[{"review_id": "yyQQzaVPbp", "reviewer": "Reviewer_VQC9", "summary": "This paper proposes a language-driven ordering alignment method for ordinal classification. For the language prompt, this paper introduces the RankFormer, which uses Transformer to learn token-wise attention over a set of rank templates. For the loss function, this paper presents a cross-modal ordinal pairwise loss under the pairwise cross-entropy loss formulation. Experimental results show the effectiveness of the proposed method.", "questions": "The key design has flaws. There are many typos and errors. This paper is highly unpolished. ", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tConsidering language prior to ordinal regression is a promising direction.\n\n2.\tNew SOTA is achieved as shown in the experiments.\n", "weaknesses": "1.\tThe design of RankFormer doesn’t make any sense. RankFormer is essentially learning R’ = f_W(R), where W and R are learnable parameters. As R’ is only conditional on W and R, learning R and R’ are completely equivalent mathematically. Any R’ learned by the proposed method is in the solution space of R. They are not fundamentally different, and any difference in performance between the two could be due to the randomness of the network.\n\n2.\tIt’s unclear what is the total loss used in this paper. \n\n3.\tIn Line 178, the meaning of “To further refine the CLIP feature space, we also propose a simplified cross-modal ordinal pairwise loss L_scop with language-related parameters frozen.” is unclear. What’s the purpose of L_scop? There is no significant difference between L_cop and L_scop. Why can’t merge them and simply use one Loss? Mathematically, you can achieve the same effect by changing the value of \\lambda.\n\n4.\tLine 187: “Many to many image-text mappings within a batch”. No, it’s one-to-many mapping. For each image, it only has one label. For each category, there may exist multiple hits. \n\n5.\tSome results are confusing. In Table 2, the variance of L2RCLIP in terms of Accuracy is the highest (7.2) while the MAE variance is relatively low (0.05). \n\n6.\tThere are many inconsistent in the formulation and equations. For example, in line 147, M represents the class numbers and k is the length of rank templates. But in Eq 2, K is the class number and k is the class index. Again, in Line 182, k becomes the length of global context prompts. \n\n7.\tThe equations from (3)-(6) are pretty messy. The sign in Eq 4 is wrong.\n\n8.\tTypos: Line 135, there is no Fig. 3.2.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer VQC9\nWe appreciate the reviewer's insightful comments. However, there seems to be some misunderstanding. We would like to clarify the issues and address the questions as follows.\n\n**Q1: The design of RankFormer.**  \n\n**[Reply]** First, we want to clarify the misunderstanding regarding the training parameters in RankFormer. As described in Line 145-147, we **fix the parameters of rank prompts, with the $W$ being learnable while the $R$ is fixed**. Second, our ablation study demonstrates that our RankFormer outperforms the baseline when we make $R$ learnable parameters.\n\n**Q2: The total loss used during training.**  \n\n**[Reply]** Due to page limitations, we have included this section in the supplementary materials. We utilize the cross-modal ordinal pairwise loss $L_{cop}$ and asymmetrical contrastive loss $L_{t2i}$ and $L_{i2t}$ to learn reliable rank prompts. In the second stage, we employ the cross-entropy loss $L_{ce}$ and simplified cross-modal ordinal pairwise loss $L_{scop}$ to fine-tune the image encoder. Further details can be found in Supp. Line 17-25.\n\n**Q3: Our setting for $L_{cop}$ and $L_{scop}$.**  \n\n**[Reply]** When finetuning the image encoders, we fix the learned rank prompts so that the entropy estimation of these text embeddings can be ignored, which is equivalent to setting $\\lambda=0$. It is worth noting that $L_{scop}$ is indeed a special case of $L_{cop}$, as you have suggested.\n\n**Q4: Many-to-many mapping.**  \n\n**[Reply]** There seems to be a misunderstanding. We are employing pairwise contrastive learning, similar to CLIP. This means that the <category, image> and <image, category> pairs should have a similar many-to-many relationship from a row-wise or column-wise perspective. Thus, both the category and image may exist multiple hits.\n\n**Q5: Check the result on Adience dataset.**  \n\n**[Reply]** We apologize for the oversight. We have rechecked the test pipeline for the Adience dataset, and the accurate results are as follows:\n\n|  L2RCLIP(Ours)   | Total  | 5-fold 01 | 5-fold 02 | 5-fold 03 | 5-fold 04 | 5-fold 05 |\n|  ----  | ----  | ----  | ----  | ----  | ----  | ----  |\n| MAE | 0.36±0.05 | 0.39 | 0.28 | 0.39 | 0.35 | 0.42 |\n| Accuracy  | 66.2%±4.4% | 64.8% | 73.5% | 64.6% | 68.0% | 60.1% |\n\n**Q6-8: The typos, formula and notation.**  \n\n**[Reply]** We will ensure to avoid repeated notation and correct any mistakes in our revised paper.\n\nWe have made every effort to address your comments faithfully in the revised paper. If you have any additional comments, please let us know. Thank you once again for your constructive feedback. We truly appreciate it."}, {"review_id": "Tb7tLecxeM", "reviewer": "Reviewer_GKDm", "summary": "This paper proposes L2RCLIP, which features two modules for ordinal classification with vision-language models, i.e., CLIP. The first is a token-wise attention module called RankFormer to tune the rank prompts. And the second is a pairwise ordinal loss to inject rank information into the supervision. Synergically, the two modules achieves competitive performance across age estimation, aesthetics assessment and historical image dating benchmarks, as well as improvements in few-shot and distribution shift experiments.", "questions": "1.\tThe authors might give qualitative or quantitative analysis of the rank prompts to support that rank information is learned and preserved by RankFormer.\n2.\tThe authors might give more elaboration on the synergy of the two proposed modules in Table 5, e.g., explain what necessitates the use of a pairwise loss to unleash the power of RankFormer and how / whether the pairwise loss can be regarded as an indispensable part of the RankFormer? \n3.\tSome details need the authors’ clarification:\n1)\tWhy OrdinalCLIP does not preserve semantic alignment as it has also learned a set of context prompts other than the rank prompts?\n2)\tAs the rand templates in an Mxkxc tensor in the token-wise attention (line147), does the attention operate on the 1st dimension (M) or on the 2nd dimension (k)?\n3)\tWhat does “language-related parameters frozen” mean in line 178? Does it mean the prompting part (including context and rank prompts) is frozen or only the text encoder is frozen.\n4)\tAs Lscop is a simplified special case of Lcop, how would the two co-exist in Table 5.\n\n", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed two modules are simple and effective in improving ordinal classification ability of CLIP models.\n2.\tThe token-wise attention in prompt tuning is interesting.\n3.\tThe writing of this paper is of good quality.\n\n", "weaknesses": "1.\tConsidering rank information in the loss has been a common practice in ordinal classification methods as depicted in the related works (line80-line92). It is unclear how language priors are applied in the proposed pairwise loss (Eq.6), and thus distinguish this loss from existing methods.\n2.\tAlthough the token-wise attention is new and intuitively incorporates the information of different rank prompts, the ordinal properties with strictly ordered ranks are not assured in the attention process.\n3.\tSome definition of variables may cause confusion, e.g., The ‘T’s in Eq.3,4,5 represent the output embeddings of text encoder while the ‘T’s in Figure2(a) seems representing the prompt embeddings. And it’s confusing that whether ‘k’s in line147 and line149 are for the same thing.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer GKDm\nWe thank the reviewer for the valuable feedback and a positive assessment of our work. We are happy the reviewer finds the paper well-organised and our method interesting, valuable, and innovative with good performance. Below we detail our response to the review concerns.\n***\n**Q1: The qualitative and quantitative analysis of rank information**\n\n**[Reply]** We use three kinds of method to verify the rank information learned by our proposed modules. **First**, we follow OrdinalCLIP(Li et al, NeurlPS2022) and adopt **ordinality score** to measure the distance of normalized rank templates quantitatively and qualitatively. We outperform the previous method by over 5.93%. **Second**, we conduct **comprehensive ablation study** on two different datasets to verify the effectiveness of our proposed method. **Finally**, we **visualize the embedding space** for the ablated methods in the Supp. Fig.4. We think these qualitative and quantitative analysis will support the rank information learned by our proposed method.\n\n**Q2: Explaination of proposed two modules** \n\n**[Reply]** We aim to improve the performance of ordinal classification by focusing on two key aspects.\n**First**, through our experiments, we have observed that vanilla text prompts already possess a certain degree of ordinal property. Building upon this observation, we have designed a **token-wise RankFormer module** to further enhance the ordering alignment within these prompts. This module specifically focuses on capturing and reinforcing the correct ordering relationships between different tokens. **Second**, taking inspiration from previous work on metric learning, we have **incorporated language knowledge** into a lower bound of cross-entropy loss. Additionally, we have introduced **an additional distance weighting term** to effectively model the embedding space with better ordering alignment. This helps to ensure that the learned representations exhibit the desired ordinal properties.\n\nFrom the perspective of experiment, we have conducted a comprehensive ablation study. This study allows us to individually assess the impact of each module and examine how their effects can be combined. Furthermore, we have visualized the embedding space of each ablated model, providing additional insights into the behavior and performance of our proposed approach. These visualizations can be found in Supp. Fig. 4.\n\n\n\n**Q3.1: The semantic alignment in OrdinalCLIP by context prompts**\n\n**[Reply]** We understand your points regarding the semantic alignment of OrdinalCLIP. We agree that context prompts can enhance semantic alignment, as evidenced by previous works like CoOp and other prompt tuning methods.\n\nHowever, we would like to emphasize two important aspects. **First**, rule-based interpolation does not guarantee that the interpolated results will always adhere to the correct semantic alignment. This can potentially lead to suboptimal performance on downstream tasks. \n**Second**, we consider semantic alignment can be measured using metrics such as MAE when CLIP is adopted for classification. Our methods not only demonstrate better performance on several benchmark tests but also, as shown in Table 1, exhibit promising results in both semantic alignment and ordering alignment, even without the use of global context prompts.\n\nTable 1. Ablation study of global context prompts.\n\n|Method\t|Morph (MAE)\t|Morph (OS)\t|CACD (MAE)\t|CACD (OS)|\n|---|---|---|---|---|\n|Vanilla CLIP\t|6.91\t|55.36%\t|4.66\t|52.51%|\n|CoOp(Variant)\t|2.39\t|59.92%\t|2.75\t|53.33%|\n|OrdinalCLIP(w context prompt)\t|2.32\t|65.94%\t|—\t|—|\n|w/o context prompt\t|2.23\t|65.46%\t|2.76\t|67.17%|\n|L2RCLIP(Ours)\t|2.13\t|71.87%\t|2.62\t|67.55%|\n\n\n**Q3.2: Token-wise attention in RankFormer**\n\n**[Reply]** The attention mechanism in RankFormer operates on the first dimension. RankFormer handles three different types of tokens.\n**First**, for special tokens like [EOS], RankFormer keeps them the same during training. These special tokens are not optimized.\n**Second**, for normal tokens, RankFormer functions similarly to linear layers.\n**Lastly**, for rank tokens, RankFormer employs a token-level attention mechanism to further enhance the ordinal property. \n\n**Q3.3&Q3.4 : Training detail about language-related parameters and $L_{cop}$/$L_{scop}$**\n\n**[Reply]** We fix the text encoder and image encoder and only train global context prompts and RankFormer in the first stage and only finetune the image encoder in the second stage.  As you suggested, $L_{scop}$ is the special case of $L_{cop}$. We firstly use $L_{cop}$, $L_{t2i}$ and L_{i2t} to learn reliable rank prompts. Then, we use $L_{scop}$ and cross-entropy loss $L_{ce}$ to finetune the image encoder for better performance. **See more details in Supp Line 17-25.** In the second stage, all language-related parameters are frozen. So entropy estimation of last term in Eq(4) can be ignored, which is equivalent to setting $\\lambda =0$. Hope the misunderstanding can be cleared by our explanation.\n\n\n\n**W1: The application of lanuage prior and novelty compare with previous method**\n\n**[Reply]** As far as we are aware, there have been limited studies that specifically focus on incorporating language priors into metric learning techniques. Compared with language-powered method, e.g. OrdinalCLIP, we propose RankFormer and cross-modal ordinal pairwise loss to boost performance of ordinal classification and achieve the SOTA performance on several test benchmarks.\n\n**W2: The learned ordinal properties by RankFormer**\n\n**[Reply]** We explain W2 carefully in Q1. Please refer to Q1 for more details.\n\n**W3: Some typos mistakes**\n\n**[Reply]** We will ensure to avoid repeated notation and correct any mistakes in our revised paper.\n\nEvery attempt has been made to address your comments faithfully in the revised paper. If you have any additional comments, please let us know. Thank you again for your positive and insightful comments. We do appreciate them."}, {"review_id": "ZKvHhoiJ2H", "reviewer": "Reviewer_yU8S", "summary": "The paper proposed to leverage vision-and-language models to improve ordinal classification. This is a follow-up work on the previous OrdinalCLIP paper. The major contribution is RankFormer, which is designed to enhance the ordering of the original rank prompts. Also a cross-modal ordinal pairwise loss is proposed to refine the CLIP feature space. Experimental results are presented on three ordinal classification tasks, including facial age estimation, historical color image classification, and aesthetic  assessment.", "questions": "See above.", "limitations": "See above.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The intuition behind the proposed method makes sense to me. The overall idea is simple and straightforward, and should be easy to reproduce. The experimental results seem to be extensive and can demonstrate the effectiveness of the proposed method.\n", "weaknesses": "The following are more detailed comments and suggestions about the paper.\n\n1, In Line 10-11, if the goal is to incorporate language priors, why use the CLIP model? The text encoder in CLIP is not very strong. Existing LLMs can provide much better language priors.\n\n2, The paper claims that the proposed model is designed to learn both semantic and ordering based on Figure 1. It is better to provide some examples or analysis about how the semantic and ordering are learned simultaneously. The visualization in Figure 3 is not very helpful.\n\n3, The paper may want to provide more details about RankFormer in Line 145-152. Why call it token-wise attention? It seems to be the basic attention mechanism. Maybe illustrate the architecture of RankFormer as well.\n\n4, In Figure 2, why fix the CLIP_{Text} encoder but fine tuning the CLIP_{Image} encoder? The “C” in the left part means “concatenation”?\n\n5, In Line 147-149, “k is the length of rank templates”, so k is smaller than M? How to pick k in this formulation?\n\n6, The writing of the paper can be much better:\nIn Line 149, is the the length of rank templates…\nWhere is Fig. 3.2 in Line 135?\n“k” is used everywhere in the paper: in Line 182, for the number of global context prompts; in Figure 2 for the index of rank templates; in Line 149, for the length of rank templates.\n\n7, In Line 184, the paper proposed to use asymmetrical contrastive loss to handle many-to-many image-text mapping within the batch. It is unclear to me why this asymmetrical loss can better handle many-to-many mapping? Please elaborate more.\n\n8, As a follow-up paper of OrdinalCLIP, the paper is an incremental improvement over OrdinalCLIP, and the novelty of the paper seems to be limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer GKDm\nWe would like to thank the reviewer for the valuable comments. However, we feel there is some misunderstanding. We clarify the issues and address the questions accordingly as described below.\n***\n**Q1: Choice of using powerful language model**\n\n**[Reply]** That may be a promising research direction to further improve performance in ordinal classification. Due to limited computational resources, we have chosen the CLIP text encoder to provide language priors. Despite this limitation, we have found that CLIP performs well in three tasks of ordinal classification: age estimation, historical color image classification, and aesthetic assessment. Our proposed method has achieved state-of-the-art (SOTA) performance on several test benchmarks. As you suggested, we believe our method can provide new insights for further LLM-based ordinal classification.\n\n**Q2: Validation of learned ordinal property and semantic alignment**\n\n**[Reply]** **For ordinal property**, we use three kinds of method to verify it learned by our proposed modules. **First**, we follow OrdinalCLIP(Li et al, NeurlPS2022) and adopt ordinality score to measure the distance of normalized rank templates quantitatively and qualitatively. We outperform the previous method by over 5.93%. **Second**, we conduct comprehensive ablation study on two different datasets to verify the effectiveness of our proposed method quantitatively. **Finally**, we visualize the embedding space for the ablated methods in the Supp. Fig.4. We think these qualitative and quantitative analysis will support the rank information learned by our proposed method.\n\n**For semantic alignment**, we think the semantic alignment can be measured using metrics such as MAE when CLIP is adopted for classification. Our method performs the best in 15 out of 16 benchmark tests, which can be used to prove that our method use better semantic alignment. Moreover,   we conduct additional ablation study on global context prompts to further prove our RankFormer and proposed  cross-modal ordinal pairwise loss can achieve both semantic alignment and ordering alignment compared with previous method. The results are shown in Table 1.\n\nTable 1. Ablation study of global context prompts.\n\n|Method\t|Morph (MAE)|\tMorph (OS)\t|CACD (MAE)|\tCACD (OS)|\n|---|---|---|---|---|\n|Vanilla CLIP\t|6.91\t|55.36%\t|4.66\t|52.51%|\n|CoOp(Variant)\t|2.39\t|59.92%\t|2.75|\t53.33%|\n|OrdinalCLIP(w context prompt)\t|2.32\t|65.94%\t|—\t|—|\n|w/o context prompt\t|2.23\t|65.46%\t|2.76\t|67.17%|\n|L2RCLIP(Ours)\t|2.13\t|71.87%\t|2.62\t|67.55%|\n\n\n\n\n**Q3: Explaination of token-wise attention in RankFormer**\n\n**[Reply]** Given an input tensor $x\\in R^{M\\times N\\times C}, the normal attention operates on the second dim while token-wise attention on the first dim since we want to enhance the ordinal property in the vanilla rank prompts. In fact,  RankFormer handles three different types of tokens. **First**, for special tokens like [EOS], RankFormer keeps them the same during training. These special tokens are not optimized. **Second**, for normal tokens, RankFormer functions similarly to linear layers. **Lastly**, for rank tokens, RankFormer employs a token-level attention mechanism to further enhance the ordinal property. The detailed architecture of RankFormer will be included in our revised version.\n\n**Q4: Training parameters in CLIP text encoders and image encoders**\n\n**[Reply]** Since we only have coarse rank templates, the performance of text encoders may be significantly degraded after full-parameter fine-tuning. Therefore, we only fine-tune a minimal number of parameters in the text branch. Additionally, \"C\" represents concatenation, and we will include this correction in our revised framework.\n\n**Q5: Explaination of meaning of notations**\n\n**[Reply]** Apologies for any confusion caused. In fact, $k$ here is usually smaller than the max token length (which is 77 in the most of case) in CLIP. $M$ is the number of rank templates or ordinal categories in ordinal classification, e.g. $M$=101 if the range of age estimation is [0,100]. We choose $k$ based on our pre-defined rank templates. We will exclude special token in practice.\n\n**Q6: Revision of some confusing typos**\n\n**[Reply]** We will ensure to avoid repeated notation in our revised paper. Thanks for your advice.\n\n**Q7: Explaination of asymmetrical loss**\n\n**[Reply]** In contrast to the normal contrastive learning in CLIP, where each image has only one target label, our cases involve images with multiple target labels within a batch. Directly adopting the symmetrical loss used in CLIP would be suboptimal in this scenario. We need to take into account all the target labels in a batch. Therefore, we group the correct matches in the similarity map and compute the loss by taking the mean.\n\n**Q8: Limited novelty compared with OrdinalCLIP**\n\n**[Reply]** We do not agree. The analysis is as follows:\nBoth OrdinalCLIP and our L2RCLIP aim to leverage language priors for the ordinal classification task. The main difference lies in the design of the ranking mechanism. OrdinalCLIP manually designs interpolation rules and applies interpolation to a few learnable rank prompts, achieving good results on certain test benchmarks.\nHowever, we argue that this explicit interpolation may involve a tradeoff between semantic alignment and ordering alignment, as the interpolated result may not guarantee correct semantic alignment. In response to this, we have designed a new token-wise RankFormer and a novel cross-modal ordinal pairwise loss. This allows us to learn a more complex ranking mechanism while preserving semantic alignment. Our method has achieved SOTA) performance. Overall, our L2RCLIP outperforms in 15 out of 16 benchmark tests.\n\nEvery attempt has been made to address your comments faithfully in the revised paper. If you have any additional comments, please let us know. Thank you again for your valuable comments. We do appreciate them."}, {"review_id": "Cssn785qg4", "reviewer": "Reviewer_XrDm", "summary": "In this paper, a novel ordinal regression framework based on CLIP is proposed. The proposed algorithm, which is called L2RCLIP, exploits language priors together with image features. \nIt encourages that image features at each class locate around the text feature of that class in the embedding space. To this end, L2RCLIP uses RankFormer to obtain text features and CLIP image encoders to obtain image features.\nThe network is optimized via cross-modal ordinal pairwise loss. Extensive experiments on various ordinal regression tasks show that the proposed algorithm outperforms the previous SOTA, ordinalCLIP.\n", "questions": "Please see the weakness section for my questions on the paper. In overall, I'm leaning to accept because the proposed algorithm is clearly described, and technically sound. Also, it achieves the good scores on various benchmark tests. However, I will see the other reviewer's opinion and the author response too. ", "limitations": "The authors have addressed the potential negative social impact in the main paper but I was not able to find specific discussion about the limitation. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is easy to follow. It describes the proposed algorithm clearly and seems like to be reproducible.\n\n2. The proposed algorithm is simple but technically sound. It also achieves the best scores in most tests.\n\n3. Experiments are diverse and solid enough to evaluate the performances properly. It also provides extensive ablation studies for better understanding on each part of the proposed algorithm\n\n", "weaknesses": "\n* Major\n1. It would be better to discuss about the many loss functions in Eq(2)~Eq(6) more deeply. It lacks the how the loss function operates to encourage network training to be performed into the desirable direction.\n\n2. Are global context prompts learnable parameters as well?\n\n3. If so, it would be interesting to see the ablation result for global context prompts. It may be used as the auxiliary role for the language priors, but it may reduce the impact of the rank template encoding. \nIn such a case, without the global context prompts, the results in Table 6 may be changed meaningfully. \n\n4. MORPH II has 4 widely used evaluation settings. In the paper, the evaluation on the most simple setting is provided only. It would be helpful to compare the performances on the other challenging settings.\n\n* Minor\n\n1. L135, Fig. 3.2 -> Figure 2\n\n2. In Eq (1):  z_j -> z_i?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer XrDm\nWe sincerely appreciate your positive review and valuable comments. Please find our responses below.\n***\n\n**Q1: To prove the effectiveness of proposed losses**\n\n**[Reply]** Thank you for your suggestion. We provide a detailed analysis of Eq(2) to Eq(6) as follows:\n\n- **Eq(2)->Eq(3)** Compared to images, language contains a higher density of information. Intuitively, the information conveyed by **\"a 20-year-old person\"** and a large number of images containing 20-year-old people is similar. Therefore, we attempt to combine language features with pairwise loss in a mean form.\n\n- **Eq(3)->Eq(4)** Previous work has addressed the diversity term through meanNN-based entropy estimation.\n\n- **Eq(3)->Eq(5)** The tightness term can be straightforwardly transformed into a loss objective function.\n\n- **Eq(4),Eq(5)->Eq(6)** We introduce a simple distance term to further enhance the ordering relation.\n\n\n**Q2: The role of global context prompts**\n\n**[Reply]** The results of the related ablation study are presented in Table 1:\n\n**Table 1.  Ablation study of global context prompts.**\n|Method\t|Morph (MAE)\t|Morph (OS)\t|CACD (MAE)\t|CACD (OS)|\n|  ----  | ----  | ----  | ----  | ----  |\n|Vanilla CLIP\t|6.91\t|55.36%\t|4.66\t|52.51%|\n|CoOp(Variant)\t|2.39\t|59.92%\t|2.75\t|53.33%|\n|w/o context prompt\t|2.23\t|65.46%\t|2.76\t|67.17%|\n|L2RCLIP(Ours)\t|2.13\t|71.87%\t|2.62\t|67.55%|\n\n**Q3: Results on all settings of Morph II**\n\n**[Reply]** We have conducted experiments on the other three settings of Morph II. The results are presented in Table 2.\n\n**Table 2. Additional results on Morph II**\n|Method\t|SettingA\t|SettingB\t|SettingC\t|SettingD|\n|  ----  | ----  | ----  | ----  | ----  |\n|MWR-G(2022, CVPR)\t|2.24\t|2.55\t|2.61\t|2.16|\n|GOL(2022, NeurlPS)\t|2.17\t|2.60\t|**2.51**\t|2.09|\n|L2RCLIP(Ours)\t|**2.13**\t|**2.53**\t|2.56\t|**1.95**|\n\n**Q4: Some minor typos error**\n\n**[Reply]** Thank you for pointing out the typos. We have corrected the mentioned errors in our revised paper.\n\nEvery attempt has been made to address your comments faithfully in the revised paper. If you have any additional comments, please let us know. Thank you again for your positive and insightful comments. We do appreciate them.\n\n"}, {"review_id": "kf06Glgm5G", "reviewer": "Reviewer_6B9T", "summary": "The paper presents L2RCLIP, a novel language-driven ordering alignment method for ordinal classification. The authors propose to leverage the rich ordinal priors in human language by converting the original task into a vision-language alignment task. The method introduces a complementary prompt tuning technique called RankFormer, designed to enhance the ordering relation of original rank prompts. In addition, the authors propose a cross-modal ordinal pairwise loss to refine the CLIP feature space, where texts and images maintain both semantic alignment and ordering alignment. The method is evaluated on three ordinal classification tasks, including facial age estimation, historical color image (HCI) classification, and aesthetic assessment, showing promising performance.\n", "questions": "Can the authors provide a fair comparison by evaluating L2RCLIP using a similar backbone to previous works?\nHow can the ordinality score be better evaluated to provide a more comprehensive understanding of the method's performance?\nCould the authors provide experimental evidence or a more detailed explanation to support the claim of Rank-specific prompts being effective in enhancing the ordering relation?\nCan the authors describe the final loss composition in the main text, providing a clear explanation of how the different components of the loss function are combined?", "limitations": "The authors need to analysis the limitations of their work.", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed loss function is an interesting contribution to the field, as it provides a new perspective on viewing cross-entropy loss within the context of ordinal regression. The authors' approach to incorporating language priors and restructuring the cross-modal embedding space using cross-modal ordinal pairwise loss is innovative and well-presented.\nThe method demonstrates impressive performance on various ordinal classification tasks, outperforming previous state-of-the-art methods. This indicates the potential of L2RCLIP in addressing real-world problems related to ordinal classification, including facial age estimation, historical image dating, and image aesthetics assessment.\n", "weaknesses": "The comparison with previous methods seems unfair, as the authors use a much stronger image backbone from CLIP, while previous papers use VGG16 as their backbone. This could be a significant factor contributing to the improved performance of L2RCLIP. It would be beneficial for the authors to provide a fair comparison by also evaluating their method using a similar backbone to previous works.\nIn Figure 3, some unexpected spikes around age 10 and some cube-like patterns are observed, indicating that the ordinality score is not as smooth as expected, despite being higher than previous methods. The authors could consider reducing the window of comparisons and evaluating the ordinality score more comprehensively. This would provide a better understanding of the method's performance in terms of ordinality.\nThe claim of Rank-specific prompts being effective in enhancing the ordering relation is too strong, and there is no experimental evidence to support it. While token mix may increase computation and information flow, it does not necessarily guarantee the ordinal property of rank features or the final textual features. The authors should provide experimental evidence or a more detailed explanation to support this claim.\nThe paper could be improved by describing the final loss composition in the main text, which is currently missing. Providing a clear explanation of how the different components of the loss function are combined would help the reader better understand the proposed method.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 6B9T\nWe sincerely appreciate your positive review and valuable comments. Please find our responses below.\n***\n**Q1: Compred with previous method with the same architecture**\n\n**[Reply]** Our method doesn't show promising result using VGG16, which may be reasonable since our two key designs based on the well-aligned text-image latent space. For fairness, we keep the same experiment setting and retrain the interpolation-based method based on OrdinalCLIP. The results and training details are as follows:\n\n**Table 1. Results on Morph, CLAP2015 and Adience datasets**\n|Method| Morph (MAE) | CLAP2015 (MAE) | Adience (MAE) | Adience (Acc.)\n|---|---|---|---|---|\n|L2RCLIP-I|2.19|2.78| 0.42 ± 0.06|62.9 ± 5.5|\n|L2RCLIP(ours)|**2.13**|**2.62**|  **0.36 ± 0.05** |**66.2±4.4**|\t\n\n**Table 2. The MAE results under the distribution shift setting on the MOPRH II**\n|re cls - re smp| 10-90| 20-80| 20-90| 30-80| 30-90| 40-80| 40-90|\n|---|---|---|---|---|---|---|---|\n|L2RCLIP-I| 2.39| 2.45 |2.50 |2.57| 2.70| 2.73| 2.93|\n|L2RCLIP(ours)| **2.30**| **2.37**|**2.43**|**2.51** |**2.61**| **2.68**| **2.79**|\n\n**Table 3. The MAE results under few shot setting on the MOPRH II**\n|#shots |#1| #2| #4| #8 |#16| #32| #64|\n|---|---|---|---|---|---|---|---|\n|L2RCLIP-I| **4.31** |4.02| 3.63 |3.48| 3.13| 2.80| 2.62|\n|L2RCLIP(ours)| 4.54| **3.92** |**3.40** |**3.28**| **2.81**| **2.55**| **2.38**|\n\nFor fairness, we use the official code of OrdinalCLIP for interpolation. We conduct three groups of experiment to verify the effectiveness of  our methods. **First**, as illustrated in Table 1, our method outperforms interpolation-based methods with a significant margin in experiments involving a large number of rank categories. This outcome is attributable to the challenge posed by direct interpolation methods in modelling complex ordering relationships. **Second**, our methods  exhibits superior performance in the majority of few-shot learning tasks and distribution shift tasks, when compared to interpolation-based methods. Collectively, these experiments corroborate the effectiveness of the methods proposed in this study.\n\n**The detail for L2RCLIP-I**: Firstly, we utilize the ViT-B/16 visual backbone of CLIP for image feature extraction, whereas OrdinalCLIP employs a pre-trained VGG-16 network supplemented by a linear projection layer. Secondly, our method relies on a two-stage training strategy, in contrast to the one-stage approach adopted by OrdinalCLIP.\n\n**Q2: The ordinality score with less ranks.**\n\n**[Reply]** Thanks for your suggestions. We reduce the number of ranks for better visualization. **The results are shown in the .pdf file.**\n\n**Q3: The ordinal property learned by RankFormer.**\n\n**[Reply]** To avoid token mixing effect, we conduct global context prompt ablation study. The results are as follows.\n\n**Table 4. Ablation study of global context prompts.**\n|Method\t|Morph (MAE)|\tMorph (OS)|\tCACD (MAE)|\tCACD (OS)|\n|---|---|---|---|---|\n|Vanilla CLIP\t|6.91\t|55.36%|\t4.66|\t52.51%|\n|CoOp(Variant)\t|2.39\t|59.92%|\t2.75\t|53.33%|\n|w/o context prompt|\t2.23|\t65.46%\t|2.76\t|67.17%|\n|L2RCLIP(Ours)|\t2.13|\t71.87%\t|2.62\t|67.55%|\n\nThe above result have shown that our porposed methods actually help to learn ordinal property. Additionally, we visualize the embedding space by t-SNE. **The results are shown in the .pdf file.**\n\n**Q4: The training detail about final loss used in L2RCLIP**\n\n**[Reply]** Due to page limitations, we have included this section in the supplementary materials. We utilize the cross-modal ordinal pairwise loss $L_{cop}$ and asymmetrical contrastive loss $L_{t2i}$ and $L_{i2t}$ to learn reliable rank prompts. In the second stage, we employ the cross-entropy loss $L_{ce}$ and simplified cross-modal ordinal pairwise loss $L_{scop}$ to fine-tune the image encoder. **Further details can be found in Supp. Line 17-25.**\n\nEvery effort has been made to address your comments faithfully in the revised paper. If you have any additional comments, please let us know. Thank you again for your positive and insightful comments. We do appreciate them.\n\n"}, {"review_id": "Nl7bp8ZRRu", "reviewer": "Reviewer_v94e", "summary": "The paper presents a novel language-driven ordering alignment method called L2RCLIP for ordinal classification. The authors leverage pre-trained vision-language models to incorporate rich ordinal priors from human language. They propose RankFormer, a prompt tuning technique that enhances the ordering relation of rank prompts using token-level attention and residual-style prompt blending. Additionally, they introduce a cross-modal ordinal pairwise loss to refine the CLIP feature space, ensuring semantic and ordering alignment between texts and images. The proposed method is evaluated on facial age estimation, historical color image classification, and aesthetic assessment tasks, showing promising performance.\n", "questions": "N/A", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel method that leverages language priors to address the overfitting issue in ordinal classification.\n2. The experimental results indicate that the proposed method achieves promising performance on the evaluated tasks, suggesting its effectiveness in addressing the overfitting problem.\n", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to the Reviewer v94e\nComment:\nThank you for your positive review and constructive feedback."}]

Author rebuttal:
# Response to All Reviewers
Thank you for your valueble review and insightful suggestions. **The .pdf file includes extra figures. Please download it if needed.**
We have made every attempt to address your comments in the revised manuscript and hope that you find this revision satisfactory. If you have additional concerns, please let us know. We do appreciate them.

Meta-review: Five out of six reviewers recommended the paper for acceptance (one of the positive reviews was mentioned in the discussion, but not updated in the system). The AC reviewed the paper, reviews, and rebuttals, and agrees with the consensus. The paper presents a novel ordering alignment method for ordinal classification leveraging strong priors derived from human language. The paper provides a thorough experimental validation, clearly outperforming prior state-of-the-art methods, and demonstrating the effectiveness of the proposed procedure.
Decision: Accept (poster)