Title: Learning-to-Rank Meets Language: Boosting Language-Driven Ordering Alignment for Ordinal Classification
Venue: NeurIPS 2023
Accepted: yes
Keywords: Ordinal Classification

Representation Learning

Vision-Language

Prompt Learning
Abstract: We present a novel language-driven ordering alignment method for ordinal classification. The labels in ordinal classification contain additional ordering relations, making them prone to overfitting when relying solely on training data. Recent developments in pre-trained vision-language models inspire us to leverage the rich ordinal priors in human language by converting the original task into a vision-language alignment task. Consequently, we propose L2RCLIP, which fully utilizes the language priors from two perspectives. First, we introduce a complementary prompt tuning technique called RankFormer, designed to enhance the ordering relation of original rank prompts. It employs token-level attention with residual-style prompt blending in the word embedding space. Second, to further incorporate language priors, we revisit the approximate bound optimization of vanilla cross-entropy loss and restructure it within the cross-modal embedding space. Consequently, we propose a cross-modal ordinal pairwise loss to refine the CLIP feature space, where texts and images maintain both semantic alignment and ordering alignment.  Extensive experiments on three ordinal classification tasks, including facial age estimation, historical color image (HCI) classification, and aesthetic assessment demonstrate its promising performance.

Learning-to-Rank Meets Language: Boosting Language-Driven Ordering Alignment for Ordinal Classification

Abstract

We present a novel language-driven ordering alignment method for ordinal classification. The labels in ordinal classification contain additional ordering relations, making them prone to overfitting when relying solely on training data. Recent developments in pre-trained vision-language models inspire us to leverage the rich ordinal priors in human language by converting the original task into a visionlanguage alignment task. Consequently, we propose L2RCLIP, which fully utilizes the language priors from two perspectives. First, we introduce a complementary prompt tuning technique called RankFormer, designed to enhance the ordering relation of original rank prompts. It employs token-level attention with residual-style prompt blending in the word embedding space. Second, to further incorporate language priors, we revisit the approximate bound optimization of vanilla cross-entropy loss and restructure it within the cross-modal embedding space. Consequently, we propose a cross-modal ordinal pairwise loss to refine the CLIP feature space, where texts and images maintain both semantic alignment and ordering alignment. Extensive experiments on three ordinal classification tasks, including facial age estimation, historical color image (HCI) classification, and aesthetic assessment demonstrate its promising performance. The code is available at https://github.com/raywang335/L2RCLIP.

1 Introduction

Ordinal classification aims to predict labels that are related in a natural or implied order, which can be considered as a special case of ordinal regression after label discretization, i.e. discretize the continuous labels and each bin is then treated as a class. Common examples of such tasks are facial age estimation (e.g., estimating the facial age from 1 to 100), historical color image classification (e.g., assigning a time period to color photographs, ranging from the 1930s to the 1970s), and aesthetics assessment(e.g., rating image quality on a scale from "unacceptable" to "exceptional").

Compared with common classification, ordinal property of labels need to be additionally considered in ordinal classification. Many algorithms [4, 35, 12] employ the ordinal classification framework, which trains a set of classifiers or integrates probabilistic priors to directly predict rank labels. However, these methods exhibit suboptimal performance as a result of insufficiently harnessing the ordering properties. Order learning algorithms [48, 39, 26, 12] demonstrate competitive performance by effectively capturing relative ordering relationships. These approaches determine the target order of a

novel instance by contrasting it with well-defined reference instances. Nonetheless, the algorithm's performance can be substantially compromised by inadequate indexing quality of the reference instances.

Furthermore, many metric learning techniques [3, 41, 13, 49] have been developed to construct an ordinal embedding space in which the distances between diverse features effectively represent the differences in their respective ranks. However, all these methods learn ranking concepts depending solely on training data, which renders them vulnerable to overfitting[29].

CLIPFeatureSpace

CLIPFeatureSpace

CLIPFeatureSpace

R1

TextFeatures

R1

r1

r3

r1

r1

R2

V3

Image Features

r2

V3

R2

RankFormer

V3

R3

V2

r2

r4

R1

Interpolation

V2

r'3

T4

R3

V2

r3

T4

R4

R3

R4

R5

semantic

ordering

r5

Rs

semantic

ordering

r5

Rs

semantic

ordering

r5

Rank:r<r2<r3<r4<rs

Rank:r<r2<r<r4<rs

Rank:r<r2<r<r4<rs

(a) CoOp

(b) OrdinalCLIP

(c) Ours

Fortunately, recent developments in large pre-trained vision-language models offer new insights for various visual tasks [32, 27, 29, 25, 56]. Compared with visual content, human language contains highly abstract concepts and rich semantic knowledge [29, 32]. Motivated by it, we attempt to borrow knowledge from language domain by two major observations. Firstly, rank templates inherently contains ordinal information, e.g. "a sixty years old face" ≻ "a ten years old face" , which is also demonstrated in Figure 3(a). Secondly, inspired by [25, 56], rank features encapsulate the average information of numerous image features within a well-aligned cross-modal feature space, which can be considered as a robust prior for cross-modal metric learning.

Hence, we propose L2RCLIP to boost learning-to-rank of CLIP-based models for ordinal classification. Specifically, we first introduce a complementary prompt tuning method, termed RankFormer, to enhance the ordering relation of original rank templates. Specifically, RankFormer employs a token-wise attention layer and performs residual-style prompt blending with the original templates for prompt tuning. Moreover, inspired by pairwise metric learning [2], we propose cross-modal ordinal pairwise loss to ensure both semantic and ordering alignment in the CLIP feature space. Concretely, we revisit the approximate bound optimization of conventional cross-entropy loss and reformulate it in the cross-modal embedding space with ordinal language priors. OrdinalCLIP [29] also incorporates language priors to model ordering alignment, demonstrating impressive performance. However, our method distinguishes itself from previous works, as depicted in Figure 1. CoOp achieves semantic alignment through contrastive loss but fails to maintain ordering alignment. OrdinalCLIP addresses ordering alignment at the cost of weakened semantic alignment. Conversely, by leveraging RankFormer and cross-modal ordinal pairwise loss, our approach simultaneously considers both semantic alignment and ordering alignment.

The contributions of this paper can be summarized as follows: (1) We incorporate learning-to-rank into vision-language pre-training model for ordinal classification, in which we present RankFormer to enhance the ordering relation of vanilla language prompts. (2) We explicitly utilize the language priors and further propose cross-modal ordinal pairwise loss to refine CLIP embedding space, in which image features and text features maintain both semantic and ordering alignment. (3) Extensive experiments demonstrate the competitive performance of L2RCLIP on age estimation, aesthetics assessment and historical image dating, as well as improvements in few-shot and distribution shift experiments.

2 Related Work

Ordinal Classification Ordinal classification attempts to solve classification problems in which not all wrong classes are equally wrong . Early techniques [4, 45] adopted the classification framework and train a set of classifiers to directly estimate the rank labels. These methods got degraded performance due to ignoring the ordering relation. By incorporating probabilistic priors, Geng et al. [12] firstly proposed label distribution learning and assigned a Gaussian or Triangle distribution for an instance. The mean-variance loss was introduced in [39] for learnable label distribution and penalizes the learned variance of estimated distribution to ensure a sharp distribution. Probabilistic embedding [28] was developed to model the data uncertainty for ordinal regression. Liu et al. [33] proposed predicting the ordinal targets that fall within a certain interval with high confidence. These methods learn better rank concepts and significantly reduce the model's overconfidence toward incorrect predictions. In contrast to them, our L2RCLIP only focuses on enhancing vision-language alignment without complicated probabilistic distribution assumption.

Furthermore, many techniques [3, 41, 13, 49, 23, 24, 22] solve the ordinal classification task from the perspective of metric learning. These methods exploit the pairwise ordering relation in embedding space, where the distance between different features reflects ordinal information. For example, Ordinal log-loss (OLL) was presented in [3] with an additional distance-aware weighting term for ordinal classification. RankSim [13] proposed a regularization loss to ensure the ordinal consistency between label and feature. Suárez et al. [49] ranked the embedding distances between pairwise rank differences of features. Another way for ordinal classification is order learning [31, 48, 19], which learns ordering relation by comparison between instances. It usually show more promising results as learning relative ordering relation is much easier than learning absolute ordering relation. Lim et al. [31] firstly proposed this concept and determined the ordinal information of an unseen instance by compared to some known reference instances. Lee&Kim et al. [19] improved the quality of indexing to boost the performance. MWR [48] further proposed to learn a continuous regression score from reference instance. However, these methods often depend solely on training data to learn rank concepts, potentially leading to overfitting. To mitigate these issues, we leverage rich priors in human language to learn language-driven rank concepts.

Vision-language Learning Vision-language pre-training(VLP) has significantly improved the performance on many downstream tasks by text and image matching, including segmentation [43, 8], object detection [14], image retrieval [36, 1], generation tasks [18, 40, 25, 5] and ordinal regression [29]. CLIP [42] and ALIGN [15] proposed to embed images and texts into the same representation space with two separate encoders in a contrastive-based approach. The experimental results show that the impressive "zero-shot" performance on downstream tasks, which show the power of language prior. Inspired by the recent advances in NLP, prompts and adapter-based tuning becomes prevalent in improving ability of CLIP. CLIP-Adapter [11] adds a light-weight module on top of image and text encoder. CoOp [55] proposes to learn context prompts for image classification. Due to lack of ordinal property, these methods lead to degraded performance in ordinal classification. Considering the great potential of language prior, we propose to incorporate learning-to-rank into CLIP for ordinal classification.

3 Proposed Approach

3.1 Problem Formulation

Ordinal classification is a unique case of image classification where labels possess ordinal properties. Mathematically, let x i ∈ X denote the i -th input instance with i = 1 , 2 , ..., N , y i ∈ { r 1 , r 2 , ..., r M } with ordered ranks r M ≻ r M -1 ≻ · · · ≻ r 1 denote the ground truth value and ˆ y i represent the predicted rank by the network, where N represents the total number of instances, M represents the number of ranks, and ≻ indicates the ordering between different ranks. Analogous to normal classification, ordinal classification aims to recover y i by encoding the image to feature z i = Φ( x i ) with encoder Φ and then using a classifier f ω ( · ) to compute probability p i . The predicted label ˆ y i is the result with the highest probability p y i . The classification probability can be calculated by:

RankFormer

Rank-specificPrompts

R<R<R

R<RK<Rj

Rank

CLIPFeatureSpace

Images

Text Features

Image Features

R:A photo of 1years old face

CLIPText

CLIPimage

Token-wise Attention

concat

GlobalContextPrompts

G2

GL-1

GL

Attraction

Ordinal Property

Repulsion

(a)Enhanceorderingrelation ofvanillarankprompts

(b)Ensureboth semantic alignment and orderingalignment

To exploit ordinal information in language, ordinal classification can be transformed into a visionlanguage alignment task. Specifically, we use a pre-trained CLIP image feature extractor Image ( · ) to extract features from input images: v i = z i = Image ( x i ) . For text features, we first construct hard rank templates R = { R 1 , R 2 , ..., R M } for a given ordinal classification task. For each template, we convert it into fixed-length tokens and then map them into 512-dimensional word embeddings. The language feature extractor Text ( · ) encodes the embeddings as a classifier weight r i . The process can be formulated as: r i = w i = Text ( Tokenizer ( R i )) . Finally, we can calculate the prediction probability p i for rank i with Eq.(1).

3.2 Proposed Method

Our objective is to integrate learning-to-rank into CLIP and learn language-driven rank concepts for ordering alignment while preserving original semantic alignment. Inspired by [16, 27], we firstly learn rank concepts in the word embedding space and subsequently refine the CLIP feature space to maintain both semantic and ordering alignment. Specifically, we introduce RankFormer to enhance the ordering relation of the original language prompts. RankFormer employs a token-wise attention layer for rank prompt tuning. To better utilize language prior, we reformulate the approximate bound optimization within cross-modal embedding space. Furthermore, we propose a cross-modal ordinal pairwise loss to ensure ordering alignment. Moreover, randomly initialized global context prompts and an asymmetrical contrastive loss are adopted to ensure semantic alignment. The overall framework is illustrated in Figure 2.

Rank-specific Prompts. As shown in Figure 3(a), rank templates in the vanilla CLIP contain some degree of ordinal information. However, a substantial proportion, nearly half of the pairs, lack clear ordinality. One intuitive approach to enhance this ordinal information is to further fine-tune these rank templates. However, implementing such a strategy introduces two primary challenges. First, the performance of different ranks for ordinal classification varies significantly driven by the imbalanced training data, and certain ranks cannot even be trained due to insufficient training data in extreme cases. Second, the contrastive loss introduces difficulties in enhancing the ordering relation during the training [53]. To address the first challenge, OrdinalCLIP [29] trains only a small number of base rank templates and generates other rank templates through explicit interpolation, which confines the ordinal information of vanilla rank templates. In this paper, we train a RankFormer with token-wise attention to enhance the ordinal information of the fixed rank templates. Specifically, we denote tokenized rank templates as R ∈ R M × n × D , where M and D represent the number of templates and word embedding channels, respectively. Note that we only consider the ranking tokens with length of n . Subsequently, we perform token-wise attention for prompt tuning and apply residual-style prompt blending with the original prompts. The process can be formulated as: R ′ = (1 -α ) · R + α · f FFN ( f MSA ( f LN ( R ))) , where α represents the residual ratio.

Cross-modal Ordinal Pairwise Loss. Since cross-entropy loss ignored the ordinal information during training [53], we turn to a solution that recovers the ordering relation while maintaining semantic alignment. Inspired by the pairwise metric learning, we firstly revisit the vanilla crossentropy loss from the perspective of approximate bound optimization. Boudiaf et al. [2] proved that minimizing the cross-entropy loss accomplishes by approximating a lower bound pairwise cross-entropy loss L PCE . L PCE contains a tightness term and a diversity term, as follows:

where z i represents the image feature in the embedding space, p ij represents the softmax probability of point z i belonging to class j , c k = ∑ N i =1 p ik z i represents the soft mean of class k and λ ∈ R is to make sure L CE is a convex function with respect to encoder Φ w .

By incorporating the language priors, we turn Eq.(2) to a cross-modal pairwise cross-entropy loss. Specifically, as human language contains rich prior knowledge, text features can be considered as both hard mean r k = 1 N k ∑ N k i =1 v k and soft mean r k = 1 N ∑ N i =1 p ik v i of image features at class k , where N k represents the sample number of class k and N represents the total sample number. Then, we reformulate the original pairwise cross-entropy loss L PCE as cross-modal pairwise cross-entropy loss L CPCE :

Inspired by [53], we use meanNN entropy estimator [10] to estimate the diversity term in Eq.(3). The process is formulated as:

̸

where D represents the feature dimensions. To recover ordering relation, we propose an additional weighting term. Intuitively, each rank template will have a high similarity score with images of a close rank and a low similarity score with images of a distant rank. As such, we opt to weight the cross-modal features with w ij , where w ij are the distances in the label space. The final cross-modal ordinal pairwise loss is defined as follows:

̸

where γ controls the strength of the rank within rank templates and B represents the batchsize. To further refining the CLIP feature space, we also propose a simplified cross-modal ordinal pairwise loss L scop with language-related parameters frozen(i.e. γ = 0 ).

Global Context Prompts. Given that global context prompts significantly surpass manually designed discrete prompts in vision tasks [55, 27, 54], we integrate them with our complementary rank-specific prompts in RankFormer to enhance semantic alignment. Specifically, we randomly initialize L global context prompts, denoted as G = { G 1 , G 2 , ..., G L } , and concatenate them with rank-specific prompts in the word embedding space of CLIP.

Asymmetrical Contrastive Loss. In vanilla CLIP [42], models are optimized using the standard contrastive loss, including a text-image contrastive loss L t 2 i and an image-text contrastive loss L i 2 t . In this work, we replace the original contrastive loss with an asymmetrical contrastive loss due to

many-to-many image-text mappings within a batch. In other words, different images in a batch may have the same rank, and the same rank may have the same description. Therefore, we replace t i with t y i in both text-image contrastive loss L t 2 i and image-text contrastive loss L i 2 t . Specifically, our improved asymmetrical contrastive loss is defined as follows:

where τ is the temperature parameter and Z ( y i ) = { z ∈ 1 ...B : y z = y i } .

3.3 Loss functions

We employ a two-stage training scheme to refine our approach following [27]. In the first stage, our focus is to enhance the ordinal information of the vanilla rank templates. We utilize cross-modal ordinal pairwise loss L cop and the asymmetrical contrastive losses L t 2 i and L i 2 t . In the second stage, our objective is to simultaneously improve both semantic alignment and ordering alignment within the CLIP latent space. We use the simplified cross-modal ordinal pairwise loss L scop and the cross-entropy loss L ce . More detail can be found in Sec. 4.1 .

4 Experiments

4.1 Implementation details

We adopt the ViT-B/16 visual encoder and the text encoder from CLIP [42] as the backbone for our image and text feature extractor. All training data is resized to 224 × 224 and subjected to random horizontal flipping. For all experiments, we employ the Adam [17] optimizer with default settings. The learning rates for the RankFormer and the visual backbone are 3 . 5 × 10 -4 and 1 × 10 -5 , respectively. The model is trained for 20 epochs in the first stage and 40 epochs in the second stage, with the learning rate decayed by a factor of 0.1 in epoch 30. We set L = 5 in most of our experiments. For MORPH and CLAP2015, we set L t 2 i , L i 2 t , and L cop with weights of 0.03, 0.03, and 3 for the first stage, and L ce and L scop with weights of 1 and 1 for the second stage. For other datasets, we set L t 2 i , L i 2 t , and L cop with weights of 0.1, 0.1, and 3 for the first stage, and L ce and L scop with weights of 1 and 1 for the second stage. All experiments are conducted on a single NVIDIA 3090 GPU.

4.2 Age Estimation

Datasets. Age estimation aims to predict the age of a given facial image. We train and evaluate our method on the widely-used MORPH II [44] dataset, CLAP2015 [9] dataset, and Adience [21] dataset. MORPH II [44] is one of the largest and most commonly used longitudinal face databases and we follow the widely adopted training and evaluation following [45, 28, 48, 47]. CLAP2015 [9] is used for apparent age estimation, where each image is rated by at least 10 annotators, and the mean rating

is set as the ground truth. Following [48], we split it into 2,476 for training, 1,136 for validation, and 1,079 for testing. Adience [21] contains discrete labels annotated with eight age groups. We adopt the same five-fold cross-validation protocol used in [21]. For evaluation metric, we use the mean average error (MAE) to measure the absolute differences between the ground truth labels and the predicted ones. Classification accuracy is additionally adopted for Adience. For detailed experimental settings, please refer to the supplementary material.

Comparison with State-of-the-art Methods. In Table 1, our L2RCLIP outperforms both conventional algorithms and language-powered algorithms in all tests. For results on the Morph II, compared with conventional algorithms, L2RCLIP achieves state-of-the-art performance on MAE at 2.13, verifying the significance of leveraging rich priors in human language. Furthermore, compared to methods utilizing language priors, our L2RCLIP exhibits substantial performance improvements. CoOp significantly enhances the vanilla CLIP's performance by learning global context prompts for semantic alignment. Subsequently, OrdinalCLIP further reduces MAE by 0.07 through fixed interpolation for ordering alignment. Nonetheless, a considerable margin remains compared to our methods, which validates the effectiveness of our approach.

Table 1 also compares the results on CLAP2015. Due to the great challenge, many previous methods adopt additional boosting schemes [30, 50]. However, without using such schemes, L2RCLIP outperforms all conventional algorithms. Compared with language-guided models, our L2RCLIP also shows much better results on MAE, specifically a significant MAE margin of 0.15 evaluated on the test set. Table 2 shows the comparison results on Adience [21] using the metrics of MAE and Accuracy. Compared with Morph II and CLAP2015, Adience is used for age group estimation. The underline indicates the mean value reported in the original paper. Our method outperforms the state-of-the-art algorithms by significant gaps of 5.7% in accuracy and 0.07 in MAE.

只

1.00

0

1.0

1.0

Q

1.0

10

86°0

10

0.9

10

0.8

10

0.9

20

0.96

20

0.8

20

20

0.8

0.6

0.7

30

0.94

30

0.7

30

30

0.6

40

0.92

40

0.6

40

0.4

40

0.5

50

- 0.90

50

0.5

50

0.2

50

0.4

60

0

(a)Vanilla CLIP

10

20

30

40

50

60

09

10

(b) CoOp

20

30

40

0S

60

0.4

60

0

(c)OrdinalCLIP

10

20

30

40

50

60

60

0

OT

(d) L2RCLIP(Ours)

20

30

40

50

60

E'0

Ordinality of Learned Rank Templates. Following [29], we report the ordinality score by measuring the cosine similarity between rank templates. The ordinality score is the percentage of rank template pairs that obey the ordinal property. Figure 3 reports the ordinality scores in different methods. Vanilla CLIP in Figure 3(a) contains a certain degree of ordering relations, where more than half of the rank template pairs show correct ordering relation. Moreover, CoOp in Figure 3(b) introduces additional global context prompts and improves the ordinality score by 4.56%. OrdinalCLIP in Figure 3(c) adopts an explicit interpolation strategy and further improves the ordinality by 6.02%. However, there is a noticeable "red stripe" in the upper right corner, indicating that this section of rank template pairs significantly violates ordinality. We believe that explicit interpolation effectively ensures local ordinal properties, but may fail to preserve global ordinal properties. Our methods greatly alleviate this problem and achieve a higher ordinality score. The "red blob" in the lower right corner results from insufficient samples for old individuals. Therefore, to avoid the model's overconfidence towards incorrect predictions, this part of rank templates should exhibit higher similarity while ensuring ordering alignment. By comparison, our method outperforms previous methods.

Few-shot Learning. Table 4 demonstrates the generalization ability of L2RCLIP for few-shot learning. Following [29], the full dataset is split into 80% for training and 20% for testing. The entire test set is used for validation, and only 1/2/4/8/16/32/64 samples in the training set from each class of labels are chosen for training. We observe that introducing rank information in CLIP

actually benefits few-shot learning tasks. Compared with CoOp, both OrdinalCLIP and our L2RCLIP achieve significant improvements across all settings, which verifies the effectiveness of ordering alignment. Moreover, our L2RCLIP further reduces the average MAE by 0.24, which demonstrates the effectiveness of the proposed learning-to-rank method in CLIP.

Distribution Shift. Following [29], we conduct data distribution shift experiments on the MORPH II dataset for generalization. We use the same setting as the general regression setting. For the training set, we randomly choose several rank labels, e.g., 10, 20, 30, and 40. Then, in those classes, we randomly discard some portion of training data, e.g., 80 and 90. We report our experiments in Table 4. For the most severe settings, CoOp, OrdinalCLIP, and our L2RCLIP methods incur performance losses of 42.24%, 34.49%, and 30.98%, respectively, which demonstrates that our approach exhibits superior robustness when faced with data distribution shift problems. For all shift settings, our method shows better performance than OrdinalCLIP, which highlights the effectiveness of the proposed learning-to-rank method in CLIP compared with interpolation.

4.3 Analysis

Ablation Study. In this section, we conduct an ablation study to examine the respective roles of each component in L2RCLIP, and the results are reported in Table 5. Three key observations can be drawn from Table 5. First, each proposed component demonstrates improvements over the baseline model and complements one another. Second, although RankFormer marginally enhances the ordinal information of vanilla rank templates, its performance remains unsatisfactory compared to other methods when only contrastive loss is utilized for training. This result further highlights the importance of our cross-modal pairwise loss in refining the CLIP feature space. Third, fine-tuning rank templates without RankFormer significantly impairs performance on CLAP2015, potentially due to imbalanced training for certain rank templates. This finding underscores the effectiveness of RankFormer in modeling the ordering relation derived from the complete set of rank templates.

Initialization Impact. As we aim to enhance the ordering relation vanilla rank templates, it is essential to examine the effects of initialization. We report our results in Table 6. Although our

methods fix the original rank templates, different initializations lead to similar convergence and performance with a low standard deviation value of 0.014. This observation further substantiates the robustness of our methods against diverse initializations.

Compared with interpolation-based method. To further prove the effectiveness of our proposed method, we compare our L2RCLIP with previous interpolation-based method (e.g. OrdinalCLIP[29]). We adopt the same setting except the process of ordinality learning and we term it as L2RCLIP-I. We report the results on the aging dataset. More experiments for few-shot learning and distribution shift and the implementation details setting for L2RCLIP-I can be found in appendix.

As illustrated in Table 7, our method outperforms interpolation-based methods with a significant margin in experiments involving a large number of rank categories. This outcome is attributable to the challenge posed by direct interpolation methods in modelling complex ordering relationships. Our approach continues to surpass interpolation-based methods even in experiments with a smaller number of rank categories. Collectively, these experiments corroborate the effectiveness of the methods proposed in this study.

4.4 Image Aesthetics Assessment

Datasets. CrowdBeauty [46] consists of 13,929 available Flickr photos across four categories: nature, animal, urban, and people. The aesthetic quality of each image is evaluated using five absolute rating scales: "unacceptable", "flawed", "ordinary", "professional", and "exceptional". Following previous methods [29, 20], we select 80% of the images for training and the rest for testing. Five-fold cross-validation is employed for fair comparisons. Both the mean MAE and accuracy are reported.

Results. Table 8 presents our results on the CrowdBeauty dataset. Aesthetic score prediction is challenging due to the subjectivity and ambiguity of aesthetic criteria; however, our L2RCLIP demonstrates state-of-the-art performance on most experimental settings by fully exploring learningto-rank with language prior. Compared with OrdinalCLIP, L2RCLIP improves accuracy by 3.02% and

reduces MAE by 0.35 overall, which further verifies the effectiveness of our proposed learning-to-rank method. When compared with the best approach without language prior, L2RCLIP improves overall MAE by 0.35 and overall accuracy by 3.37%. Consistent improvements are observed across all categories compared to previous methods, showcasing the effectiveness of our proposal in exploiting language ordinal information.

4.5 Historical Image Dating

Datasets. The historical image dating dataset [38] serves as a benchmark for automatically predicting the decade of historical colored images. The dataset comprises five-decade categories, ranging from the 1930s to the 1970s. Following [34, 38], we adopt the same train-test split and ten-fold cross-validation. Both the mean and standard deviation for MAE and accuracy metrics are reported.

Results. Table 9 showcases improvements compared to other state-of-the-art models using the full dataset. Initially, zero-shot CLIP exhibits suboptimal performance resulting from inadequate semantic and ordinal alignment. CoOp enhances the results by incorporating global context prompts to facilitate semantic alignment. Furthermore, OrdinalCLIP exploits interpolation to improve the average MAE by 0.09. Our L2RCLIP further advances the average MAE by 0.25 with a lower standard deviation. In comparison with the previous conventional model, L2RCLIP achieves a new state-of-the-art performance with an MAE of 0.43 and an accuracy of 67.22%, thereby validating the effectiveness of our proposed method.

5 Discussions and Conclusions

In this paper, we propose L2RCLIP to boost learning-to-rank of CLIP for ordinal classification. Specifically, we introduce a complementary prompt tuning method, termed RankFormer, to enhance the ordering relation of the original rank prompts. It performs token-level attention and residual-style prompt blending for prompt tuning. Additionally, we revisit the approximate bound optimization of cross-entropy and reformulate it in the cross-modal embedding space by incorporating the language knowledge. To additionally recover the ordinal information, we further introduce cross-modal ordinal pairwise loss to refine the ordering alignment of CLIP feature space. Extensive experiments demonstrate the effectiveness of our approach on various ordinal classification tasks, including facial age estimation, historical image dating, and image aesthetics assessment. Furthermore, L2RCLIP outperforms in the challenging few-shot learning and data distribution shift learning scenarios. Lastly, we conduct a comprehensive ablation study to verify the effectiveness of each component of our proposed method.

Broader Impacts. As a versatile approach, L2RCLIP can be applied to any ordinal classification tasks such as image aesthetics assessment or other rank assessment. However, these tasks may pose a risk of unlawful surveillance or invasion of privacy if abused. Meanwhile, as L2RCLIP is based on a large-scale vision-language model, addressing demographic biases in pre-trained vision-language models is of significant importance. Therefore, we emphasize that L2RCLIP represents a research proof of language-driven learning and is not appropriate for real-world usage without strict technical controls.

Acknowledgement. This research is sponsored by National Natural Science Foundation of China (Grant No. 62306041, 62006228), Beijing Nova Program (Grant No. Z211100002121106, 20230484488, 20230484276), and Youth Innovation Promotion Association CAS (Grant No.2022132).

References

Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Effective conditioned and composed image retrieval combining clip-based features. In CVPR , 2022.

Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In ECCV , 2020.

François Castagnos, Martin Mihelich, and Charles Dognin. A simple log-based loss function for ordinal text classification. In COLING , 2022.

Shixing Chen, Caojin Zhang, Ming Dong, Jialiang Le, and Mike Rao. Using ranking-cnn for age estimation. In CVPR , 2017.

Xing Cui, Zekun Li, Peipei Li, Yibo Hu, Hailin Shi, and Zhaofeng He. I2edit: Towards multi-turn interactive image editing via dialogue. 2023.

Zongyong Deng, Hao Liu, Yaoxing Wang, Chenyang Wang, Zekuan Yu, and Xuehong Sun. Pml: Progressive margin loss for long-tailed age classification. In CVPR , 2021.

Raul Diaz and Amit Marathe. Soft labels for ordinal regression. In CVPR , 2019.

Xiaoyi Dong, Yinglin Zheng, Jianmin Bao, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, et al. Maskclip: Masked self-distillation advances contrastive language-image pretraining. arXiv preprint arXiv:2208.12262 , 2022.

Sergio Escalera, Junior Fabian, Pablo Pardo, Xavier Baró, Jordi Gonzalez, Hugo J Escalante, Dusan Misevic, Ulrich Steiner, and Isabelle Guyon. Chalearn looking at people 2015: Apparent age and cultural event recognition datasets and results. In ICCVW , 2015.

Lev Faivishevsky and Jacob Goldberger. Ica based on a smooth estimation of the differential entropy. In NeurIPS , 2008.

Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544 , 2021.

Xin Geng, Chao Yin, and Zhi-Hua Zhou. Facial age estimation by learning from label distributions. TPAMI , 2013.

Yu Gong, Greg Mori, and Frederick Tung. Ranksim: Ranking similarity regularization for deep imbalanced regression. In ICML , 2022.

Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921 , 2021.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML , 2021.

Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.09217 , 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Gihyun Kwon and Jong Chul Ye. Clipstyler: Image style transfer with a single text condition. In CVPR , 2022.

Seon-Ho Lee and Chang-Su Kim. Deep repulsive clustering of ordered data based on order-identity decomposition. In ICLR , 2021.

Seon-Ho Lee, Nyeong Ho Shin, and Chang-Su Kim. Geometric order learning for rank estimation. In NeurIPS , 2022.

Gil Levi and Tal Hassncer. Age and gender classification using convolutional neural networks. In CVPRW , 2015.

Peipei Li, Yibo Hu, Ran He, and Zhenan Sun. Global and local consistent wavelet-domain age synthesis. TIFS , 14(11):2943-2957, 2019.

Peipei Li, Yibo Hu, Xiang Wu, Ran He, and Zhenan Sun. Deep label refinement for age estimation. PR , 2020.

Peipei Li, Huaibo Huang, Yibo Hu, Xiang Wu, Ran He, and Zhenan Sun. Hierarchical face aging through disentangled latent characteristics. In ECCV , 2020.

Peipei Li, Rui Wang, Huaibo Huang, Ran He, and Zhaofeng He. Pluralistic aging diffusion autoencoder. In ICCV , 2023.

Qiang Li, Jingjing Wang, Zhaoliang Yao, Yachun Li, Pengju Yang, Jingwei Yan, Chunmao Wang, and Shiliang Pu. Unimodal-concentrated loss: Fully adaptive label distribution learning for ordinal regression. In CVPR , 2022.

Siyuan Li, Li Sun, and Qingli Li. Clip-reid: Exploiting vision-language model for image re-identification without concrete text labels. arXiv preprint arXiv:2211.13977 , 2022.

Wanhua Li, Xiaoke Huang, Jiwen Lu, Jianjiang Feng, and Jie Zhou. Learning probabilistic ordinal embeddings for uncertainty-aware regression. In CVPR , 2021.

Wanhua Li, Xiaoke Huang, Zheng Zhu, Yansong Tang, Xiu Li, Jie Zhou, and Jiwen Lu. Ordinalclip: Learning rank prompts for language-guided ordinal regression. In NeurIPS , 2022.

Wanhua Li, Jiwen Lu, Jianjiang Feng, Chunjing Xu, Jie Zhou, and Qi Tian. Bridgenet: A continuity-aware probabilistic network for age estimation. In CVPR , 2019.

Kyungsun Lim, Nyeong-Ho Shin, Young-Yoon Lee, and Chang-Su Kim. Order learning and its application to age estimation. In ICLR , 2020.

Zhiqiu Lin, Samuel Yu, Zhiyi Kuang, Deepak Pathak, and Deva Ramanan. Multimodality helps unimodality: Cross-modal few-shot learning with multimodal models. In CVPR , 2023.

Liang Liu, Hao Lu, Haipeng Xiong, Ke Xian, Zhiguo Cao, and Chunhua Shen. Counting objects by blockwise classification. TCSVT , 2019.

Yanzhu Liu, Adams Wai Kin Kong, and Chi Keong Goh. A constrained deep neural network for ordinal regression. In CVPR , 2018.

Yanzhu Liu, Fan Wang, and Adams Wai Kin Kong. Probabilistic deep ordinal regression based on gaussian processes. In ICCV , 2019.

Haoyu Ma, Handong Zhao, Zhe Lin, Ajinkya Kale, Zhangyang Wang, Tong Yu, Jiuxiang Gu, Sunav Choudhary, and Xiaohui Xie. Ei-clip: Entity-aware interventional contrastive learning for e-commerce cross-modal retrieval. In CVPR , 2022.

Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. Ordinal regression with multiple output cnn for age estimation. In CVPR , 2016.

Frank Palermo, James Hays, and Alexei A Efros. Dating historical color images. In ECCV , 2012.

Hongyu Pan, Hu Han, Shiguang Shan, and Xilin Chen. Mean-variance loss for deep age estimation from a face. In CVPR , 2018.

Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In ICCV , 2021.

Gorkem Polat, Ilkay Ergenc, Haluk Tarik Kani, Yesim Ozen Alahdab, Ozlen Atug, and Alptekin Temizel. Class distance weighted cross-entropy loss for ulcerative colitis severity estimation. arXiv preprint arXiv:2202.05167 , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompting. In CVPR , 2022.

Karl Ricanek and Tamirat Tesafaye. Morph: A longitudinal image database of normal adult age-progression. In FGR , 2006.

Rasmus Rothe, Radu Timofte, and Luc Van Gool. Dex: Deep expectation of apparent age from a single image. In ICCVW , 2015.

Rossano Schifanella, Miriam Redi, and Luca Maria Aiello. An image is worth more than a thousand favorites: Surfacing the hidden beauty of flickr pictures. In AAAI , 2015.

Wei Shen, Yilu Guo, Yan Wang, Kai Zhao, Bo Wang, and Alan L Yuille. Deep regression forests for age estimation. In CVPR , 2018.

Nyeong-Ho Shin, Seon-Ho Lee, and Chang-Su Kim. Moving window regression: a novel approach to ordinal regression. In CVPR , 2022.

Juan Luis Suárez, Salvador García, and Francisco Herrera. Ordinal regression with explainable distance metric learning based on ordered sequences: Extended abstract. In DSAA , 2021.

Zichang Tan, Jun Wan, Zhen Lei, Ruicong Zhi, Guodong Guo, and Stan Z Li. Efficient group-n encoding and decoding for facial age estimation. TPAMI , 2017.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR , 2008.

Xin Wen, Biying Li, Haiyun Guo, Zhiwei Liu, Guosheng Hu, Ming Tang, and Jinqiao Wang. Adaptive variance based label distribution learning for facial age estimation. In ECCV , 2020.

Shihao Zhang, Linlin Yang, Michael Bi Mi, Xiaoxu Zheng, and Angela Yao. Improving deep regression with ordinal entropy. arXiv preprint arXiv:2301.08915 , 2023.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR , 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV , 2022.

Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Lafite: Towards language-free training for text-to-image generation. In CVPR , 2022.

6 Appendix

This supplementary material begins with a comprehensive visualization of the datasets central to our study. The specifics of our experimental settings are subsequently outlined in Section 6.2. Section 6.1 features an expanded analysis, including results from ablation studies. A key highlight of this section is the visual interpretation of the CLIP image features facilitated by t-SNE [51]. Concurrently, a comparative analysis is conducted, comparing the efficacy of interpolation-based strategies with our learning-based methods(i.e. L2RCLIP).

6.1 More Analysis of L2RCLIP

16

77

(a) Vanilla CLIP

(b)+RankFormer

(c)+RankFormer&Lcop

(d)+ Lcop & Lscop

(e) Ours

Additional Ablation Study. Figure 4 presents the embedding spaces corresponding to various ablation settings, mapped from the original 512D embedding spaces via t-SNE [51]. In Panel (a), the vanilla CLIP implementation reveals a sub-optimal ordering relationship among images of distinct ranks. Feature demarcations across different ranks are ambiguous, displaying considerable overlap. The incorporation of RankFormer with global context prompts [55], illustrated in Panel (b), aids in improving and consolidating the order alignment of these image features. As shown in Panel (c), the implementation of our proposed loss function, L cop , further enhances this order alignment. Conversely, Panel (d) indicates that without the RankFormer module, the order alignment between the lowest and highest ranks falls short of the desired outcome. These findings substantiate the efficacy of the modules proposed in this study, with each component playing a significant role in the overall model performance. This underscores the criticality of their synergistic implementation.

We also explore the role of different batchsize settings. We report the result in Table 10. According to the result, we choose batchsize with 64 to conduct other experiments.

Compared with Interpolation-based method. In this study, we also propose an interpolation technique for our basic rank templates, which we term L2RCLIP-I. Two distinguishing characteristics set L2RCLIP-I apart from OrdinalCLIP [29]. Firstly, we utilize the ViT-B/16 visual backbone of

CLIP for image feature extraction, whereas OrdinalCLIP employs a pre-trained VGG-16 network supplemented by a linear projection layer. Secondly, our method relies on a two-stage training strategy, in contrast to the one-stage approach adopted by OrdinalCLIP. Importantly, both training strategies require a comparable time commitment. The results are reported in Tables 7 and 11.

Local ordinality score of L2RCLIP. To further prove our L2RCLIP have learned better ordering relationship, we follow OrdinalCLIP[29] and use the local ordinality score. The formula is defined as: LOS (%) = ∑ K i =1 ∑ K j = i I { s i,j > s i,j +1 } / ( K × ( K -1) / 2) , where K is the size of local window and s i,j represents the cosine similarity of each pair of templates. We propose that the locally linear manifold can be preserved within a fixed small window size. Therefore, we calculate the local ordinality score using window sizes of 2, 4, 8, 16, and 32. The results of the local ordinality score are shown in Table 12.

More ablation study of L2RCLIP To avoid the effect of token mixing and the type of architecture of RankFormer, we conduct more detailed ablation study. The results are shown in Table13. Our proposed method is complementary to previous prompt tuning methods and our L2RCLIP can achieve comparable performance with OrdinalCLIP even without context prompt. We have also compared L2RCLIP performance with an MLP-based architecture to avoid effects driven by extra computation. Note that both RankFormer and MLP have similar training parameters. The results show that our token-wise RankFormer can enhance the ordinality between input rank templates.

More results on Morph II datasets We have conducted experiments on the other three settings of Morph II. The results are presented in Table 14. The details for each settings are as follows:

Setting A: A total of 5,492 images of Caucasians are sampled and then randomly divided into training and test sets with a ratio of 8 : 2 .

Setting B: Approximately 21,000 images of Caucasians and Africans are randomly selected, ensuring a balanced ratio of 1 : 1 between Caucasians and Africans, as well as a ratio of 1 : 3 between females and males. The dataset is then divided into three subsets (S1, S2, S3). The training and testing process is repeated twice: 1) training on S1 and testing on S2+S3, and 2) training on S2 and testing on S1+S3.

Setting C: The entire dataset is randomly partitioned into five folds, with the constraint that images of the same person belong to only one fold. The 5-fold cross-validation is then performed.

Setting D: The entire dataset is randomly divided into five folds without any restrictions. The 5-fold cross-validation is then performed.

6.2 Experiment settings

Dataset Details. In the scope of this study, we only utilize publicly available data. To provide a comprehensive understanding of the tasks at hand, we illustrate a selection of random samples from the image aesthetics assessment dataset (Figure 5) and the historical image dating dataset (Figure 6). To further enhance our exposition, Figure 7 depicts both the original and adjusted distributions of the MORPH II dataset.

(b)

)Professional.

(d) Flawed.

U.S.N

1940s.

(a)

(a)

1950s.

a)

1960s.

Original Dist.

Recls_10_p_0.8Dist.

Recls_10_p_0.9 Dist.

0.08

0.08

0.08

test

test

test

0.06

train

0.06

train

0.06

train

Frequency

Frequency

Frequency

0.04

0.04

0.04-

0.02

0.02

0.02

0.00

0.00

0.00

20

30

40

50

60

70

80

20

30

40

50

60

70

80

20

30

40

50

60

70

80

Age

Age

Age

Recls_20_p_0.8 Dist.

Recls_30_p_0.8 Dist.

0.08

0.08

0.08

test

test

test

0.06

train

0.06

train

0.06

train

Frequency

Frequency

Frequency

0.04

0.04

0.04

0.02

0.02

0.02

0.00

0.00

0.00

20

30

40

50

60

70

80

20

30

40

50

60

70

80

20

30

40

50

60

70

80

Age

Age

Age

Recls_30_p_0.9Dist.

Recls_40_p_0.8Dist.

0.08

0.08

0.08

test

test

test

90°0

train

90:0

train

0.06

train

Frequency

Frequency

Frequency

0.04-

0.04

0.04

0.02

0.02

0.02

0.00

0.00

0.00

20

30

40

50

60

70

80

20

30

40

50

60

70

80

20

30

40

50

60

70

80

Age

Age

Age

Peer reviews:
comments: SUMMARY:
The paper presents a novel language-driven ordering alignment method called L2RCLIP for ordinal classification. The authors leverage pre-trained vision-language models to incorporate rich ordinal priors from human language. They propose RankFormer, a prompt tuning technique that enhances the ordering relation of rank prompts using token-level attention and residual-style prompt blending. Additionally, they introduce a cross-modal ordinal pairwise loss to refine the CLIP feature space, ensuring semantic and ordering alignment between texts and images. The proposed method is evaluated on facial age estimation, historical color image classification, and aesthetic assessment tasks, showing promising performance.

STRENGTHS:
1. The paper introduces a novel method that leverages language priors to address the overfitting issue in ordinal classification.
2. The experimental results indicate that the proposed method achieves promising performance on the evaluated tasks, suggesting its effectiveness in addressing the overfitting problem.

WEAKNESSES:
N/A

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents L2RCLIP, a novel language-driven ordering alignment method for ordinal classification. The authors propose to leverage the rich ordinal priors in human language by converting the original task into a vision-language alignment task. The method introduces a complementary prompt tuning technique called RankFormer, designed to enhance the ordering relation of original rank prompts. In addition, the authors propose a cross-modal ordinal pairwise loss to refine the CLIP feature space, where texts and images maintain both semantic alignment and ordering alignment. The method is evaluated on three ordinal classification tasks, including facial age estimation, historical color image (HCI) classification, and aesthetic assessment, showing promising performance.\n\nSTRENGTHS:\nThe proposed loss function is an interesting contribution to the field, as it provides a new perspective on viewing cross-entropy loss within the context of ordinal regression. The authors' approach to incorporating language priors and restructuring the cross-modal embedding space using cross-modal ordinal pairwise loss is innovative and well-presented.\nThe method demonstrates impressive performance on various ordinal classification tasks, outperforming previous state-of-the-art methods. This indicates the potential of L2RCLIP in addressing real-world problems related to ordinal classification, including facial age estimation, historical image dating, and image aesthetics assessment.\n\nWEAKNESSES:\nThe comparison with previous methods seems unfair, as the authors use a much stronger image backbone from CLIP, while previous papers use VGG16 as their backbone. This could be a significant factor contributing to the improved performance of L2RCLIP. It would be beneficial for the authors to provide a fair comparison by also evaluating their method using a similar backbone to previous works.\nIn Figure 3, some unexpected spikes around age 10 and some cube-like patterns are observed, indicating that the ordinality score is not as smooth as expected, despite being higher than previous methods. The authors could consider reducing the window of comparisons and evaluating the ordinality score more comprehensively. This would provide a better understanding of the method's performance in terms of ordinality.\nThe claim of Rank-specific prompts being effective in enhancing the ordering relation is too strong, and there is no experimental evidence to support it. While token mix may increase computation and information flow, it does not necessarily guarantee the ordinal property of rank features or the final textual features. The authors should provide experimental evidence or a more detailed explanation to support this claim.\nThe paper could be improved by describing the final loss composition in the main text, which is currently missing. Providing a clear explanation of how the different components of the loss function are combined would help the reader better understand the proposed method.\n\nQUESTIONS:\nCan the authors provide a fair comparison by evaluating L2RCLIP using a similar backbone to previous works?\nHow can the ordinality score be better evaluated to provide a more comprehensive understanding of the method's performance?\nCould the authors provide experimental evidence or a more detailed explanation to support the claim of Rank-specific prompts being effective in enhancing the ordering relation?\nCan the authors describe the final loss composition in the main text, providing a clear explanation of how the different components of the loss function are combined?\n\nLIMITATIONS:\nThe authors need to analysis the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, a novel ordinal regression framework based on CLIP is proposed. The proposed algorithm, which is called L2RCLIP, exploits language priors together with image features. \nIt encourages that image features at each class locate around the text feature of that class in the embedding space. To this end, L2RCLIP uses RankFormer to obtain text features and CLIP image encoders to obtain image features.\nThe network is optimized via cross-modal ordinal pairwise loss. Extensive experiments on various ordinal regression tasks show that the proposed algorithm outperforms the previous SOTA, ordinalCLIP.\n\nSTRENGTHS:\n1. The paper is easy to follow. It describes the proposed algorithm clearly and seems like to be reproducible.\n\n2. The proposed algorithm is simple but technically sound. It also achieves the best scores in most tests.\n\n3. Experiments are diverse and solid enough to evaluate the performances properly. It also provides extensive ablation studies for better understanding on each part of the proposed algorithm\n\nWEAKNESSES:\n* Major\n1. It would be better to discuss about the many loss functions in Eq(2)~Eq(6) more deeply. It lacks the how the loss function operates to encourage network training to be performed into the desirable direction.\n\n2. Are global context prompts learnable parameters as well?\n\n3. If so, it would be interesting to see the ablation result for global context prompts. It may be used as the auxiliary role for the language priors, but it may reduce the impact of the rank template encoding. \nIn such a case, without the global context prompts, the results in Table 6 may be changed meaningfully. \n\n4. MORPH II has 4 widely used evaluation settings. In the paper, the evaluation on the most simple setting is provided only. It would be helpful to compare the performances on the other challenging settings.\n\n* Minor\n\n1. L135, Fig. 3.2 -> Figure 2\n\n2. In Eq (1):  z_j -> z_i?\n\nQUESTIONS:\nPlease see the weakness section for my questions on the paper. In overall, I'm leaning to accept because the proposed algorithm is clearly described, and technically sound. Also, it achieves the good scores on various benchmark tests. However, I will see the other reviewer's opinion and the author response too.\n\nLIMITATIONS:\nThe authors have addressed the potential negative social impact in the main paper but I was not able to find specific discussion about the limitation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed to leverage vision-and-language models to improve ordinal classification. This is a follow-up work on the previous OrdinalCLIP paper. The major contribution is RankFormer, which is designed to enhance the ordering of the original rank prompts. Also a cross-modal ordinal pairwise loss is proposed to refine the CLIP feature space. Experimental results are presented on three ordinal classification tasks, including facial age estimation, historical color image classification, and aesthetic  assessment.\n\nSTRENGTHS:\nThe intuition behind the proposed method makes sense to me. The overall idea is simple and straightforward, and should be easy to reproduce. The experimental results seem to be extensive and can demonstrate the effectiveness of the proposed method.\n\nWEAKNESSES:\nThe following are more detailed comments and suggestions about the paper.\n\n1, In Line 10-11, if the goal is to incorporate language priors, why use the CLIP model? The text encoder in CLIP is not very strong. Existing LLMs can provide much better language priors.\n\n2, The paper claims that the proposed model is designed to learn both semantic and ordering based on Figure 1. It is better to provide some examples or analysis about how the semantic and ordering are learned simultaneously. The visualization in Figure 3 is not very helpful.\n\n3, The paper may want to provide more details about RankFormer in Line 145-152. Why call it token-wise attention? It seems to be the basic attention mechanism. Maybe illustrate the architecture of RankFormer as well.\n\n4, In Figure 2, why fix the CLIP_{Text} encoder but fine tuning the CLIP_{Image} encoder? The “C” in the left part means “concatenation”?\n\n5, In Line 147-149, “k is the length of rank templates”, so k is smaller than M? How to pick k in this formulation?\n\n6, The writing of the paper can be much better:\nIn Line 149, is the the length of rank templates…\nWhere is Fig. 3.2 in Line 135?\n“k” is used everywhere in the paper: in Line 182, for the number of global context prompts; in Figure 2 for the index of rank templates; in Line 149, for the length of rank templates.\n\n7, In Line 184, the paper proposed to use asymmetrical contrastive loss to handle many-to-many image-text mapping within the batch. It is unclear to me why this asymmetrical loss can better handle many-to-many mapping? Please elaborate more.\n\n8, As a follow-up paper of OrdinalCLIP, the paper is an incremental improvement over OrdinalCLIP, and the novelty of the paper seems to be limited.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes L2RCLIP, which features two modules for ordinal classification with vision-language models, i.e., CLIP. The first is a token-wise attention module called RankFormer to tune the rank prompts. And the second is a pairwise ordinal loss to inject rank information into the supervision. Synergically, the two modules achieves competitive performance across age estimation, aesthetics assessment and historical image dating benchmarks, as well as improvements in few-shot and distribution shift experiments.\n\nSTRENGTHS:\n1.\tThe proposed two modules are simple and effective in improving ordinal classification ability of CLIP models.\n2.\tThe token-wise attention in prompt tuning is interesting.\n3.\tThe writing of this paper is of good quality.\n\nWEAKNESSES:\n1.\tConsidering rank information in the loss has been a common practice in ordinal classification methods as depicted in the related works (line80-line92). It is unclear how language priors are applied in the proposed pairwise loss (Eq.6), and thus distinguish this loss from existing methods.\n2.\tAlthough the token-wise attention is new and intuitively incorporates the information of different rank prompts, the ordinal properties with strictly ordered ranks are not assured in the attention process.\n3.\tSome definition of variables may cause confusion, e.g., The ‘T’s in Eq.3,4,5 represent the output embeddings of text encoder while the ‘T’s in Figure2(a) seems representing the prompt embeddings. And it’s confusing that whether ‘k’s in line147 and line149 are for the same thing.\n\nQUESTIONS:\n1.\tThe authors might give qualitative or quantitative analysis of the rank prompts to support that rank information is learned and preserved by RankFormer.\n2.\tThe authors might give more elaboration on the synergy of the two proposed modules in Table 5, e.g., explain what necessitates the use of a pairwise loss to unleash the power of RankFormer and how / whether the pairwise loss can be regarded as an indispensable part of the RankFormer? \n3.\tSome details need the authors’ clarification:\n1)\tWhy OrdinalCLIP does not preserve semantic alignment as it has also learned a set of context prompts other than the rank prompts?\n2)\tAs the rand templates in an Mxkxc tensor in the token-wise attention (line147), does the attention operate on the 1st dimension (M) or on the 2nd dimension (k)?\n3)\tWhat does “language-related parameters frozen” mean in line 178? Does it mean the prompting part (including context and rank prompts) is frozen or only the text encoder is frozen.\n4)\tAs Lscop is a simplified special case of Lcop, how would the two co-exist in Table 5.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a language-driven ordering alignment method for ordinal classification. For the language prompt, this paper introduces the RankFormer, which uses Transformer to learn token-wise attention over a set of rank templates. For the loss function, this paper presents a cross-modal ordinal pairwise loss under the pairwise cross-entropy loss formulation. Experimental results show the effectiveness of the proposed method.\n\nSTRENGTHS:\n1.\tConsidering language prior to ordinal regression is a promising direction.\n\n2.\tNew SOTA is achieved as shown in the experiments.\n\nWEAKNESSES:\n1.\tThe design of RankFormer doesn’t make any sense. RankFormer is essentially learning R’ = f_W(R), where W and R are learnable parameters. As R’ is only conditional on W and R, learning R and R’ are completely equivalent mathematically. Any R’ learned by the proposed method is in the solution space of R. They are not fundamentally different, and any difference in performance between the two could be due to the randomness of the network.\n\n2.\tIt’s unclear what is the total loss used in this paper. \n\n3.\tIn Line 178, the meaning of “To further refine the CLIP feature space, we also propose a simplified cross-modal ordinal pairwise loss L_scop with language-related parameters frozen.” is unclear. What’s the purpose of L_scop? There is no significant difference between L_cop and L_scop. Why can’t merge them and simply use one Loss? Mathematically, you can achieve the same effect by changing the value of \\lambda.\n\n4.\tLine 187: “Many to many image-text mappings within a batch”. No, it’s one-to-many mapping. For each image, it only has one label. For each category, there may exist multiple hits. \n\n5.\tSome results are confusing. In Table 2, the variance of L2RCLIP in terms of Accuracy is the highest (7.2) while the MAE variance is relatively low (0.05). \n\n6.\tThere are many inconsistent in the formulation and equations. For example, in line 147, M represents the class numbers and k is the length of rank templates. But in Eq 2, K is the class number and k is the class index. Again, in Line 182, k becomes the length of global context prompts. \n\n7.\tThe equations from (3)-(6) are pretty messy. The sign in Eq 4 is wrong.\n\n8.\tTypos: Line 135, there is no Fig. 3.2.\n\nQUESTIONS:\nThe key design has flaws. There are many typos and errors. This paper is highly unpolished.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}