SlowGuess/NeurIPS_2023_OCR huggingface dataset:
is set as the ground truth. Following [48], we split it into 2,476 for training, 1,136 for validation, and 1,079 for testing. Adience [21] contains discrete labels annotated with eight age groups. We adopt the same five-fold cross-validation protocol used in [21]. For evaluation metric, we use the mean average error (MAE) to measure the absolute differences between the ground truth labels and the predicted ones. Classification accuracy is additionally adopted for Adience. For detailed experimental settings, please refer to the supplementary material.
Comparison with State-of-the-art Methods. In Table 1, our L2RCLIP outperforms both conventional algorithms and language-powered algorithms in all tests. For results on the Morph II, compared with conventional algorithms, L2RCLIP achieves state-of-the-art performance on MAE at 2.13, verifying the significance of leveraging rich priors in human language. Furthermore, compared to methods utilizing language priors, our L2RCLIP exhibits substantial performance improvements. CoOp significantly enhances the vanilla CLIP's performance by learning global context prompts for semantic alignment. Subsequently, OrdinalCLIP further reduces MAE by 0.07 through fixed interpolation for ordering alignment. Nonetheless, a considerable margin remains compared to our methods, which validates the effectiveness of our approach.
Table 1 also compares the results on CLAP2015. Due to the great challenge, many previous methods adopt additional boosting schemes [30, 50]. However, without using such schemes, L2RCLIP outperforms all conventional algorithms. Compared with language-guided models, our L2RCLIP also shows much better results on MAE, specifically a significant MAE margin of 0.15 evaluated on the test set. Table 2 shows the comparison results on Adience [21] using the metrics of MAE and Accuracy. Compared with Morph II and CLAP2015, Adience is used for age group estimation. The underline indicates the mean value reported in the original paper. Our method outperforms the state-of-the-art algorithms by significant gaps of  $5.7\%$  in accuracy and 0.07 in MAE.
Figure 3: The similarity matrices of rank templates in different methods. The redder, the more similar the pair of rank templates. The percentages of templates pairs that obey the ordinality are:  $55.36\%$ ,  $59.92\%$ ,  $65.94\%$ , and  $71.87\%$ , respectively.
**Ordinality of Learned Rank Templates.** Following [29], we report the ordinality score by measuring the cosine similarity between rank templates. The ordinality score is the percentage of rank template pairs that obey the ordinal property. Figure 3 reports the ordinality scores in different methods. Vanilla CLIP in Figure 3(a) contains a certain degree of ordering relations, where more than half of the rank template pairs show correct ordering relation. Moreover, CoOp in Figure 3(b) introduces additional global context prompts and improves the ordinality score by  $4.56\%$ . OrdinalCLIP in Figure 3(c) adopts an explicit interpolation strategy and further improves the ordinality by  $6.02\%$ . However, there is a noticeable "red stripe" in the upper right corner, indicating that this section of rank template pairs significantly violates ordinality. We believe that explicit interpolation effectively ensures local ordinal properties, but may fail to preserve global ordinal properties. Our methods greatly alleviate this problem and achieve a higher ordinality score. The "red blob" in the lower right corner results from insufficient samples for old individuals. Therefore, to avoid the model's overconfidence towards incorrect predictions, this part of rank templates should exhibit higher similarity while ensuring ordering alignment. By comparison, our method outperforms previous methods.
Few-shot Learning. Table 4 demonstrates the generalization ability of L2RCLIP for few-shot learning. Following [29], the full dataset is split into  $80\%$  for training and  $20\%$  for testing. The entire test set is used for validation, and only 1/2/4/8/16/32/64 samples in the training set from each class of labels are chosen for training. We observe that introducing rank information in CLIP
i.e.,  $OS(\%) = \sum_{i=1}^{M} \sum_{j=i}^{M-1} \mathbb{I}\{s_{i,j} > s_{i,j+1}\} / (M \times (M-1)/2)$ , where  $M$  is the number of templates and  $s_{i,j}$  represents the cosine similarity of each pair of templates.
Table 3: We report the MAE results under few-shot settings on the MOPRH II dataset.
# Shots	1	2	4	8	16	32	64
CoOp [55]	5.09	4.50	3.81	3.57	3.23	2.87	2.61
OrdinalCLIP [29]	4.94	4.36	3.55	3.31	3.07	2.76	2.57
L2RCLIP(Ours)	4.54	3.92	3.40	3.28	2.81	2.55	2.38
actually benefits few-shot learning tasks. Compared with CoOp, both OrdinalCLIP and our L2RCLIP achieve significant improvements across all settings, which verifies the effectiveness of ordering alignment. Moreover, our L2RCLIP further reduces the average MAE by 0.24, which demonstrates the effectiveness of the proposed learning-to-rank method in CLIP.
Table 4: The MAE results under the distribution shift setting on the MOPRH II. "re cls" denotes the number of reduced classes, and "re tmp" means the percentage of reduced sampled in one class.
recls - reemp	10-80	10-90	20-80	20-90	30-80	30-90	40-80	40-90
CoOp [55]	2.71	2.85	2.98	3.51	3.06	3.36	2.99	3.30
OrdinalCLIP [29]	2.61	2.67	2.77	3.06	2.86	3.21	2.84	3.12
L2RCLIP(Ours)	2.28	2.30	2.37	2.43	2.51	2.61	2.68	2.79
Distribution Shift. Following [29], we conduct data distribution shift experiments on the MORPH II dataset for generalization. We use the same setting as the general regression setting. For the training set, we randomly choose several rank labels, e.g., 10, 20, 30, and 40. Then, in those classes, we randomly discard some portion of training data, e.g., 80 and 90. We report our experiments in Table 4. For the most severe settings, CoOp, OrdinalCLIP, and our L2RCLIP methods incur performance losses of  $42.24\%$ ,  $34.49\%$ , and  $30.98\%$ , respectively, which demonstrates that our approach exhibits superior robustness when faced with data distribution shift problems. For all shift settings, our method shows better performance than OrdinalCLIP, which highlights the effectiveness of the proposed learning-to-rank method in CLIP compared with interpolation.
Table 5: Ablation Study of L2RCLIP on the MORPH II and CLAP2015.
Ablation	Choices
RankFormer	✓	✓	✓	✓
\(L_{cop}\)	✓	✓	✓	✓
\(L_{scop}\)	✓	✓	✓	✓
Morph II	MAE(↓)OS(%,↑)	2.5055.89	2.4857.58	2.3766.15	2.3855.89	2.2166.15	2.2971.87	2.3157.58	2.1371.87
CLAP2015	MAE(↓)OS(%,↑)	2.7553.33	2.6854.77	2.7157.33	2.7153.33	2.7057.33	2.6567.55	2.6655.25	2.6267.55
Ablation Study. In this section, we conduct an ablation study to examine the respective roles of each component in L2RCLIP, and the results are reported in Table 5. Three key observations can be drawn from Table 5. First, each proposed component demonstrates improvements over the baseline model and complements one another. Second, although RankFormer marginally enhances the ordinal information of vanilla rank templates, its performance remains unsatisfactory compared to other methods when only contrastive loss is utilized for training. This result further highlights the importance of our cross-modal pairwise loss in refining the CLIP feature space. Third, fine-tuning rank templates without RankFormer significantly impairs performance on CLAP2015, potentially due to imbalanced training for certain rank templates. This finding underscores the effectiveness of RankFormer in modeling the ordering relation derived from the complete set of rank templates.
Initialization Impact. As we aim to enhance the ordering relation vanilla rank templates, it is essential to examine the effects of initialization. We report our results in Table 6. Although our
Table 6: Initialization Impact Analysis.
Initialization	MAE(↓)	OS(%, ↑)
A photo of {age} years old face.	2.13	71.87
Age estimation: a person at the age of {age}.	2.14	72.07
The age of the person in the portrait is {age}.	2.12	71.32
The age of the person is {age}.	2.16	71.52
The age of the face is {age}.	2.14	71.08
Mean/Std	2.14/0.01	71.57/0.40
methods fix the original rank templates, different initializations lead to similar convergence and performance with a low standard deviation value of 0.014. This observation further substantiates the robustness of our methods against diverse initializations.
Table 7: Results on Morph, CLAP2015 and Adience datasets.
Methods	Morph (MAE, ↓)	CLAP2015 (MAE, ↓)	Adience (Accuracy, ↑)	(MAE, ↓)
L2RCLIP-I	2.19	2.78	62.9 ± 5.5	0.42 ± 0.06
L2RCLIP (Ours)	2.13	2.62	68.2 ± 7.2	0.36 ± 0.05
Compared with interpolation-based method. To further prove the effectiveness of our proposed method, we compare our L2RCLIP with previous interpolation-based method (e.g. OrdinalCLIP[29]). We adopt the same setting except the process of ordinality learning and we term it as L2RCLIP-I. We report the results on the aging dataset. More experiments for few-shot learning and distribution shift and the implementation details setting for L2RCLIP-I can be found in appendix.
As illustrated in Table 7, our method outperforms interpolation-based methods with a significant margin in experiments involving a large number of rank categories. This outcome is attributable to the challenge posed by direct interpolation methods in modelling complex ordering relationships. Our approach continues to surpass interpolation-based methods even in experiments with a smaller number of rank categories. Collectively, these experiments corroborate the effectiveness of the methods proposed in this study.
4.4 Image Aesthetics Assessment
Datasets. CrowdBeauty [46] consists of 13,929 available Flickr photos across four categories: nature, animal, urban, and people. The aesthetic quality of each image is evaluated using five absolute rating scales: "unacceptable", "flawed", "ordinary", "professional", and "exceptional". Following previous methods [29, 20], we select  $80\%$  of the images for training and the rest for testing. Five-fold cross-validation is employed for fair comparisons. Both the mean MAE and accuracy are reported.
Table 8: Results on Image Aesthetics dataset.
Methods	Accuracy(%,↑)	MAE(↓)
Nature	Animal	Urban	People	Overall	Nature	Animal	Urban	People	Overall
CNNPOR [34]	71.86	69.32	69.09	69.94	70.05	0.294	0.322	0.325	0.321	0.316
SORD [7]	73.59	70.29	73.25	70.59	72.03	0.271	0.308	0.276	0.309	0.290
POE [28]	73.62	71.14	72.78	72.22	72.44	0.273	0.299	0.281	0.293	0.287
GOL [20]	73.8	72.4	74.2	69.6	72.7	0.27	0.28	0.26	0.31	0.28
Vanilla CLIP [42]	65.24	45.67	58.78	53.06	55.68	0.461	0.557	0.468	0.524	0.502
CoOp [55]	72.74	71.46	72.14	69.34	71.42	0.285	0.298	0.294	0.330	0.302
OridinalCLIP [29]	73.65	72.85	73.20	72.50	73.05	0.273	0.279	0.277	0.291	0.280
L2RCLIP(Ours)	73.51	75.26	77.76	78.69	76.07	0.267	0.253	0.216	0.246	0.245
Results. Table 8 presents our results on the CrowdBeauty dataset. Aesthetic score prediction is challenging due to the subjectivity and ambiguity of aesthetic criteria; however, our L2RCLIP demonstrates state-of-the-art performance on most experimental settings by fully exploring learning-to-rank with language prior. Compared with OrdinalCLIP, L2RCLIP improves accuracy by  $3.02\%$  and
reduces MAE by 0.35 overall, which further verifies the effectiveness of our proposed learning-to-rank method. When compared with the best approach without language prior, L2RCLIP improves overall MAE by 0.35 and overall accuracy by  $3.37\%$ . Consistent improvements are observed across all categories compared to previous methods, showcasing the effectiveness of our proposal in exploiting language ordinal information.
4.5 Historical Image Dating
Datasets. The historical image dating dataset [38] serves as a benchmark for automatically predicting the decade of historical colored images. The dataset comprises five-decade categories, ranging from the 1930s to the 1970s. Following [34, 38], we adopt the same train-test split and ten-fold cross-validation. Both the mean and standard deviation for MAE and accuracy metrics are reported.
Results. Table 9 showcases improvements compared to other state-of-the-art models using the full dataset. Initially, zero-shot CLIP exhibits suboptimal performance resulting from inadequate semantic and ordinal alignment. CoOp enhances the results by incorporating global context prompts to facilitate semantic alignment. Furthermore, Ordinal-CLIP exploits interpolation to improve the average MAE by 0.09. Our L2RCLIP further advances the average MAE by 0.25 with a lower standard deviation. In comparison with the previous conventional model, L2RCLIP achieves a new state-of-the-art performance with an MAE of 0.43 and an accuracy of  $67.22\%$ , thereby validating the effectiveness of our proposed method.
Table 9: Results on HCI dataset.
Methods	HCI
MAE(↓)	Accuracy(%, ↑)
CNNPOR [34]	0.82 ± 0.05	50.12 ± 2.65
GP-DNNOR [35]	0.76 ± 0.05	46.60 ± 2.98
POE [28]	0.76 ± 0.04	54.68 ± 3.21
MWR [48]	0.58	57.8
GOL [20]	0.55	56.2
Vanilla CLIP [42]	1.01 ± 0.03	30.41 ± 3.32
CoOp [55]	0.76 ± 0.06	51.94 ± 2.60
OridinalCLIP [29]	0.67 ± 0.03	56.44 ± 1.66
L2RCLIP(Ours)	0.43 ± 0.02	67.22 ± 1.59
5 Discussions and Conclusions
In this paper, we propose L2RCLIP to boost learning-to-rank of CLIP for ordinal classification. Specifically, we introduce a complementary prompt tuning method, termed RankFormer, to enhance the ordering relation of the original rank prompts. It performs token-level attention and residual-style prompt blending for prompt tuning. Additionally, we revisit the approximate bound optimization of cross-entropy and reformulate it in the cross-modal embedding space by incorporating the language knowledge. To additionally recover the ordinal information, we further introduce cross-modal ordinal pairwise loss to refine the ordering alignment of CLIP feature space. Extensive experiments demonstrate the effectiveness of our approach on various ordinal classification tasks, including facial age estimation, historical image dating, and image aesthetics assessment. Furthermore, L2RCLIP outperforms in the challenging few-shot learning and data distribution shift learning scenarios. Lastly, we conduct a comprehensive ablation study to verify the effectiveness of each component of our proposed method.
Broader Impacts. As a versatile approach, L2RCLIP can be applied to any ordinal classification tasks such as image aesthetics assessment or other rank assessment. However, these tasks may pose a risk of unlawful surveillance or invasion of privacy if abused. Meanwhile, as L2RCLIP is based on a large-scale vision-language model, addressing demographic biases in pre-trained vision-language models is of significant importance. Therefore, we emphasize that L2RCLIP represents a research proof of language-driven learning and is not appropriate for real-world usage without strict technical controls.
Acknowledgement. This research is sponsored by National Natural Science Foundation of China (Grant No. 62306041, 62006228), Beijing Nova Program (Grant No. Z211100002121106, 20230484488, 20230484276), and Youth Innovation Promotion Association CAS (Grant No.2022132).
This supplementary material begins with a comprehensive visualization of the datasets central to our study. The specifics of our experimental settings are subsequently outlined in Section 6.2. Section 6.1 features an expanded analysis, including results from ablation studies. A key highlight of this section is the visual interpretation of the CLIP image features facilitated by t-SNE [51]. Concurrently, a comparative analysis is conducted, comparing the efficacy of interpolation-based strategies with our learning-based methods (i.e. L2RCLIP).
6.1 More Analysis of L2RCLIP
(a) Vanilla CLIP
(b) +RankFormer
(c) +RankFormer &  $L_{\text{cop}}$
Figure 4: Visualizing the ablation effects on the MORPH II dataset: t-SNE visualizations of 512D spaces in CLIP latent space.
(d)  $+L_{cop}$  &  $L_{scop}$
(e) Ours
Additional Ablation Study. Figure 4 presents the embedding spaces corresponding to various ablation settings, mapped from the original 512D embedding spaces via t-SNE [51]. In Panel (a), the vanilla CLIP implementation reveals a sub-optimal ordering relationship among images of distinct ranks. Feature demarcations across different ranks are ambiguous, displaying considerable overlap. The incorporation of RankFormer with global context prompts [55], illustrated in Panel (b), aids in improving and consolidating the order alignment of these image features. As shown in Panel (c), the implementation of our proposed loss function,  $L_{cop}$ , further enhances this order alignment. Conversely, Panel (d) indicates that without the RankFormer module, the order alignment between the lowest and highest ranks falls short of the desired outcome. These findings substantiate the efficacy of the modules proposed in this study, with each component playing a significant role in the overall model performance. This underscores the criticality of their synergistic implementation.
Table 10: Batchsize Analysis.
Batchsize	16	32	64	96	128
MAE(↓)	2.15	2.13	2.13	2.17	2.17
OS(%,↑)	67.21	68.55	71.87	71.71	75.42
We also explore the role of different batchsize settings. We report the result in Table 10. According to the result, we choose batchsize with 64 to conduct other experiments.
Table 11: The MAE results under the distribution shift setting and few shot setting on the MOPRH II.
re cls - re tmp	10-90	20-80	20-90	30-80	30-90	40-80	40-90
L2RCLIP-I	2.39	2.45	2.50	2.57	2.70	2.73	2.93
L2RCLIP(Ours)	2.30	2.37	2.43	2.51	2.61	2.68	2.79
#Shots	#1	#2	#4	#8	#16	#32	#64
L2RCLIP-I	4.31	4.02	3.63	3.48	3.13	2.80	2.62
L2RCLIP(Ours)	4.54	3.92	3.40	3.28	2.81	2.55	2.38
Compared with Interpolation-based method. In this study, we also propose an interpolation technique for our basic rank templates, which we term L2RCLIP-I. Two distinguishing characteristics set L2RCLIP-I apart from OrdinalCLIP [29]. Firstly, we utilize the ViT-B/16 visual backbone of
CLIP for image feature extraction, whereas OrdinalCLIP employs a pre-trained VGG-16 network supplemented by a linear projection layer. Secondly, our method relies on a two-stage training strategy, in contrast to the one-stage approach adopted by OrdinalCLIP. Importantly, both training strategies require a comparable time commitment. The results are reported in Tables 7 and 11.
Local ordinality score of L2RCLIP. To further prove our L2RCLIP have learned better ordering relationship, we follow OrdinalCLIP[29] and use the local ordinality score. The formula is defined as:  $LOS(\%) = \sum_{i=1}^{K} \sum_{j=i}^{K} \mathbb{I}\{s_{i,j} > s_{i,j+1}\} / (K \times (K-1)/2)$ , where  $K$  is the size of local window and  $s_{i,j}$  represents the cosine similarity of each pair of templates. We propose that the locally linear manifold can be preserved within a fixed small window size. Therefore, we calculate the local ordinality score using window sizes of 2, 4, 8, 16, and 32. The results of the local ordinality score are shown in Table 12.
Table 12: The local ordinality score results on the MORPH II dataset.
# window size	2	4	8	16	32
Vanilla CLIP	100.00	83.33	78.57	70.83	60.08
OrdinalCLIP[29]	100.00	100.00	100.00	96.19	-
L2RCLIP(Ours)	100.00	100.00	100.00	100.00	97.78
Table 13: Ablation study of global context prompts and architecture.
Method	Morph(MAE)	Morph(OS%)	CLAP2015(MAE)	CLAP2015(OS%)
Vanilla CLIP	6.91	55.36	4.66	52.51
w/o context prompt	2.23	65.46	2.76	67.17
RankFormer→MLP	2.27	67.48	-	-
L2RCLIP(Ours)	2.13	71.87	2.62	67.55
More ablation study of L2RCLIP To avoid the effect of token mixing and the type of architecture of RankFormer, we conduct more detailed ablation study. The results are shown in Table13. Our proposed method is complementary to previous prompt tuning methods and our L2RCLIP can achieve comparable performance with OrdinalCLIP even without context prompt. We have also compared L2RCLIP performance with an MLP-based architecture to avoid effects driven by extra computation. Note that both RankFormer and MLP have similar training parameters. The results show that our token-wise RankFormer can enhance the ordinality between input rank templates.
Table 14: Additional results on Morph II.
Methods	Setting A	Setting B	Setting C	Setting D
DRC-ORID[19]	2.26	2.51	2.58	2.16
OL[31]	2.41	2.75	2.68	2.22
MWR-G[48]	2.24	2.55	2.61	2.16
GOL[20]	2.17	2.60	2.51	2.09
L2RCLIP(Ours)	2.13	2.53	2.56	1.95
More results on Morph II datasets We have conducted experiments on the other three settings of Morph II. The results are presented in Table 14. The details for each settings are as follows:
- Setting A: A total of 5,492 images of Caucasians are sampled and then randomly divided into training and test sets with a ratio of  $8:2$ .
- Setting B: Approximately 21,000 images of Caucasians and Africans are randomly selected, ensuring a balanced ratio of  $1:1$  between Caucasians and Africans, as well as a ratio of  $1:3$  between females and males. The dataset is then divided into three subsets (S1, S2, S3). The training and testing process is repeated twice: 1) training on S1 and testing on S2+S3, and 2) training on S2 and testing on S1+S3.
- Setting C: The entire dataset is randomly partitioned into five folds, with the constraint that images of the same person belong to only one fold. The 5-fold cross-validation is then performed.
- Setting D: The entire dataset is randomly divided into five folds without any restrictions. The 5-fold cross-validation is then performed.
6.2 Experiment settings
Dataset Details. In the scope of this study, we only utilize publicly available data. To provide a comprehensive understanding of the tasks at hand, we illustrate a selection of random samples from the image aesthetics assessment dataset (Figure 5) and the historical image dating dataset (Figure 6). To further enhance our exposition, Figure 7 depicts both the original and adjusted distributions of the MORPH II dataset.
(a) Exceptional.
Figure 5: Samples from the urban collections of the aesthetics dataset.
(b) Professional.
(c) Ordinary.
(d) Flawed.
(e) Unacceptable.
(a) 1930s.
Figure 6: Samples from the historical image dating dataset.
(a) 1940s.
(a) 1950s.
(a) 1960s.
(a) 1970s.
Figure 7: Original and shifted distributions of the MORPH II dataset.