SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rui Wang $^{1}$ , Peipei Li $^{1*}$ , Huaibo Huang $^{2}$ , Chunshui Cao $^{3}$ , Ran He $^{2}$ , Zhaofeng He $^{1}$
$^{2}$ CRIPAC&MAIS, Institute of Automation, Chinese Academy of Sciences
3 WATRIX.AI
{wr_bupt, lipeipei, zhaofenghe} @bupt.edu.cn
huaibo.huang@cripac.ia.ac.cn, chunshui.cao@watrix.ai, rhe@nlpr.ia.ac.cn
We present a novel language-driven ordering alignment method for ordinal classification. The labels in ordinal classification contain additional ordering relations, making them prone to overfitting when relying solely on training data. Recent developments in pre-trained vision-language models inspire us to leverage the rich ordinal priors in human language by converting the original task into a vision-language alignment task. Consequently, we propose L2RCLIP, which fully utilizes the language priors from two perspectives. First, we introduce a complementary prompt tuning technique called RankFormer, designed to enhance the ordering relation of original rank prompts. It employs token-level attention with residual-style prompt blending in the word embedding space. Second, to further incorporate language priors, we revisit the approximate bound optimization of vanilla cross-entropy loss and restructure it within the cross-modal embedding space. Consequently, we propose a cross-modal ordinal pairwise loss to refine the CLIP feature space, where texts and images maintain both semantic alignment and ordering alignment. Extensive experiments on three ordinal classification tasks, including facial age estimation, historical color image (HCI) classification, and aesthetic assessment demonstrate its promising performance. The code is available at https://github.com/raywang335/L2RCLIP.
Ordinal classification aims to predict labels that are related in a natural or implied order, which can be considered as a special case of ordinal regression after label discretization, i.e. discretize the continuous labels and each bin is then treated as a class. Common examples of such tasks are facial age estimation (e.g., estimating the facial age from 1 to 100), historical color image classification (e.g., assigning a time period to color photographs, ranging from the 1930s to the 1970s), and aesthetics assessment (e.g., rating image quality on a scale from "unacceptable" to "exceptional").
Compared with common classification, ordinal property of labels need to be additionally considered in ordinal classification. Many algorithms [4, 35, 12] employ the ordinal classification framework, which trains a set of classifiers or integrates probabilistic priors to directly predict rank labels. However, these methods exhibit suboptimal performance as a result of insufficiently harnessing the ordering properties. Order learning algorithms [48, 39, 26, 12] demonstrate competitive performance by effectively capturing relative ordering relationships. These approaches determine the target order of a
* Corresponding author
novel instance by contrasting it with well-defined reference instances. Nonetheless, the algorithm's performance can be substantially compromised by inadequate indexing quality of the reference instances.
Furthermore, many metric learning techniques [3, 41, 13, 49] have been developed to construct an ordinal embedding space in which the distances between diverse features effectively represent the differences in their respective ranks. However, all these methods learn ranking concepts depending solely on training data, which renders them vulnerable to overfitting[29].
(a) CoOp
(b) OrdinalCLIP
Figure 1: Comparison with CoOp(a) and OrdinalCLIP(b), where  $v$  represents the image features and  $R, r$  represents the rank templates in word embedding space and CLIP feature space, respectively. (a) CoOp aligns each rank template with its corresponding images via contrastive loss but vanilla CLIP fails to ensure ordering alignment. (b) OrdinalCLIP considers additional interpolation to explicitly maintain ordering alignment. However, the interpolation term can not preserve semantic alignment in CLIP feature space. (c) Our method enhances the ordering relation of vanilla rank templates while ensuring the semantic alignment of CLIP space.
(c) Ours
Fortunately, recent developments in large pre-trained vision-language models offer new insights for various visual tasks [32, 27, 29, 25, 56]. Compared with visual content, human language contains highly abstract concepts and rich semantic knowledge [29, 32]. Motivated by it, we attempt to borrow knowledge from language domain by two major observations. Firstly, rank templates inherently contains ordinal information, e.g. "a sixty years old face"  $\succ$  "a ten years old face", which is also demonstrated in Figure 3(a). Secondly, inspired by [25, 56], rank features encapsulate the average information of numerous image features within a well-aligned cross-modal feature space, which can be considered as a robust prior for cross-modal metric learning.
Hence, we propose L2RCLIP to boost learning-to-rank of CLIP-based models for ordinal classification. Specifically, we first introduce a complementary prompt tuning method, termed RankFormer, to enhance the ordering relation of original rank templates. Specifically, RankFormer employs a token-wise attention layer and performs residual-style prompt blending with the original templates for prompt tuning. Moreover, inspired by pairwise metric learning [2], we propose cross-modal ordinal pairwise loss to ensure both semantic and ordering alignment in the CLIP feature space. Concretely, we revisit the approximate bound optimization of conventional cross-entropy loss and reformulate it in the cross-modal embedding space with ordinal language priors. OrdinalCLIP [29] also incorporates language priors to model ordering alignment, demonstrating impressive performance. However, our method distinguishes itself from previous works, as depicted in Figure 1. CoOp achieves semantic alignment through contrastive loss but fails to maintain ordering alignment. OrdinalCLIP addresses ordering alignment at the cost of weakened semantic alignment. Conversely, by leveraging Rank-Former and cross-modal ordinal pairwise loss, our approach simultaneously considers both semantic alignment and ordering alignment.
The contributions of this paper can be summarized as follows: (1) We incorporate learning-to-rank into vision-language pre-training model for ordinal classification, in which we present RankFormer to enhance the ordering relation of vanilla language prompts. (2) We explicitly utilize the language priors and further propose cross-modal ordinal pairwise loss to refine CLIP embedding space, in which image features and text features maintain both semantic and ordering alignment. (3) Extensive experiments demonstrate the competitive performance of L2RCLIP on age estimation, aesthetics assessment and historical image dating, as well as improvements in few-shot and distribution shift experiments.
Ordinal Classification Ordinal classification attempts to solve classification problems in which not all wrong classes are equally wrong. Early techniques [4, 45] adopted the classification framework and train a set of classifiers to directly estimate the rank labels. These methods got degraded performance due to ignoring the ordering relation. By incorporating probabilistic priors, Geng et al. [12] firstly proposed label distribution learning and assigned a Gaussian or Triangle distribution for an instance. The mean-variance loss was introduced in [39] for learnable label distribution and penalizes the learned variance of estimated distribution to ensure a sharp distribution. Probabilistic embedding [28] was developed to model the data uncertainty for ordinal regression. Liu et al. [33] proposed predicting the ordinal targets that fall within a certain interval with high confidence. These methods learn better rank concepts and significantly reduce the model's overconfidence toward incorrect predictions. In contrast to them, our L2RCLIP only focuses on enhancing vision-language alignment without complicated probabilistic distribution assumption.
Furthermore, many techniques [3, 41, 13, 49, 23, 24, 22] solve the ordinal classification task from the perspective of metric learning. These methods exploit the pairwise ordering relation in embedding space, where the distance between different features reflects ordinal information. For example, Ordinal log-loss (OLL) was presented in [3] with an additional distance-aware weighting term for ordinal classification. RankSim [13] proposed a regularization loss to ensure the ordinal consistency between label and feature. Suárez et al. [49] ranked the embedding distances between pairwise rank differences of features. Another way for ordinal classification is order learning [31, 48, 19], which learns ordering relation by comparison between instances. It usually show more promising results as learning relative ordering relation is much easier than learning absolute ordering relation. Lim et al. [31] firstly proposed this concept and determined the ordinal information of an unseen instance by compared to some known reference instances. Lee&Kim et al. [19] improved the quality of indexing to boost the performance. MWR [48] further proposed to learn a continuous regression score from reference instance. However, these methods often depend solely on training data to learn rank concepts, potentially leading to overfitting. To mitigate these issues, we leverage rich priors in human language to learn language-driven rank concepts.
Vision-language Learning Vision-language pre-training(VLP) has significantly improved the performance on many downstream tasks by text and image matching, including segmentation [43, 8], object detection [14], image retrieval [36, 1], generation tasks [18, 40, 25, 5] and ordinal regression [29]. CLIP [42] and ALIGN [15] proposed to embed images and texts into the same representation space with two separate encoders in a contrastive-based approach. The experimental results show that the impressive "zero-shot" performance on downstream tasks, which show the power of language prior. Inspired by the recent advances in NLP, prompts and adapter-based tuning becomes prevalent in improving ability of CLIP. CLIP-Adapter [11] adds a light-weight module on top of image and text encoder. CoOp [55] proposes to learn context prompts for image classification. Due to lack of ordinal property, these methods lead to degraded performance in ordinal classification. Considering the great potential of language prior, we propose to incorporate learning-to-rank into CLIP for ordinal classification.
Ordinal classification is a unique case of image classification where labels possess ordinal properties. Mathematically, let  $x_{i} \in \mathcal{X}$  denote the  $i$ -th input instance with  $i = 1,2,\dots,N$ ,  $y_{i} \in \{r_{1},r_{2},\ldots,r_{M}\}$  with ordered ranks  $r_{M} \succ r_{M-1} \succ \dots \succ r_{1}$  denote the ground truth value and  $\hat{y}_{i}$  represent the predicted rank by the network, where  $N$  represents the total number of instances,  $M$  represents the number of ranks, and  $\succ$  indicates the ordering between different ranks. Analogous to normal classification, ordinal classification aims to recover  $y_{i}$  by encoding the image to feature  $z_{i} = \Phi(x_{i})$  with encoder  $\Phi$  and then using a classifier  $f_{\omega}(\cdot)$  to compute probability  $p_{i}$ . The predicted label  $\hat{y}_{i}$  is the result with the highest probability  $p_{y_i}$ . The classification probability can be calculated by:
Figure 2: An overview of the proposed L2RCLIP. We incorporate learning-to-rank into CLIP from two perspectives. First, RankFormer performs a token-level attention mechanism to enhance the ordering relation of vanilla rank prompts. Then, refined rank-specific prompts and randomly initialized context prompts are concatenated in the word embedding space and are sent to a text encoder to extract the corresponding text features. Moreover, we present two types of losses to refine CLIP feature space by attraction and repulsion, respectively. Attraction refers to an asymmetrical contrastive loss and a tightness term to attract paired images and text features while repulsion refers to a reweighting diversity term to ensure the ordering alignment.
$$
p _ {i} = \frac {\exp \left(\omega_ {i} ^ {\top} z _ {i}\right)}{\sum_ {j = 1} ^ {M} \exp \left(\omega_ {j} ^ {\top} z _ {i}\right)}. \tag {1}
$$
To exploit ordinal information in language, ordinal classification can be transformed into a vision-language alignment task. Specifically, we use a pre-trained CLIP image feature extractor  $Image(\cdot)$  to extract features from input images:  $v_{i} = z_{i} = Image(x_{i})$ . For text features, we first construct hard rank templates  $R = \{R_{1}, R_{2}, \dots, R_{M}\}$  for a given ordinal classification task. For each template, we convert it into fixed-length tokens and then map them into 512-dimensional word embeddings. The language feature extractor  $Text(\cdot)$  encodes the embeddings as a classifier weight  $r_{i}$ . The process can be formulated as:  $r_{i} = w_{i} = Text(Tokenizer(R_{i}))$ . Finally, we can calculate the prediction probability  $p_{i}$  for rank  $i$  with Eq.(1).
Our objective is to integrate learning-to-rank into CLIP and learn language-driven rank concepts for ordering alignment while preserving original semantic alignment. Inspired by [16, 27], we firstly learn rank concepts in the word embedding space and subsequently refine the CLIP feature space to maintain both semantic and ordering alignment. Specifically, we introduce RankFormer to enhance the ordering relation of the original language prompts. RankFormer employs a token-wise attention layer for rank prompt tuning. To better utilize language prior, we reformulate the approximate bound optimization within cross-modal embedding space. Furthermore, we propose a cross-modal ordinal pairwise loss to ensure ordering alignment. Moreover, randomly initialized global context prompts and an asymmetrical contrastive loss are adopted to ensure semantic alignment. The overall framework is illustrated in Figure 2.
Rank-specific Prompts. As shown in Figure 3(a), rank templates in the vanilla CLIP contain some degree of ordinal information. However, a substantial proportion, nearly half of the pairs, lack clear ordinality. One intuitive approach to enhance this ordinal information is to further fine-tune these rank templates. However, implementing such a strategy introduces two primary challenges. First, the performance of different ranks for ordinal classification varies significantly driven by the imbalanced training data, and certain ranks cannot even be trained due to insufficient training data in extreme cases. Second, the contrastive loss introduces difficulties in enhancing the ordering relation during the training [53]. To address the first challenge, OrdinalCLIP [29] trains only a small number of base rank templates and generates other rank templates through explicit interpolation, which confines the ordinal information of vanilla rank templates. In this paper, we train a RankFormer with token-wise attention to enhance the ordinal information of the fixed rank templates. Specifically, we denote tokenized rank templates as  $R \in \mathbf{R}^{\mathbf{M} \times \mathbf{n} \times \mathbf{D}}$ , where  $M$  and  $D$  represent the number of templates and word embedding channels, respectively. Note that we only consider the ranking tokens with length of  $n$ . Subsequently, we perform token-wise attention for prompt tuning and apply residual-style prompt blending with the original prompts. The process can be formulated as:  $R' = (1 - \alpha) \cdot R + \alpha \cdot f_{FFN}(f_{MSA}(f_{LN}(R)))$ , where  $\alpha$  represents the residual ratio.
Cross-modal Ordinal Pairwise Loss. Since cross-entropy loss ignored the ordinal information during training [53], we turn to a solution that recovers the ordering relation while maintaining semantic alignment. Inspired by the pairwise metric learning, we firstly revisit the vanilla cross-entropy loss from the perspective of approximate bound optimization. Boudiaf et al. [2] proved that minimizing the cross-entropy loss accomplishes by approximating a lower bound pairwise cross-entropy loss  $L_{PCE}$ .  $L_{PCE}$  contains a tightness term and a diversity term, as follows:
$$
L _ {P C E} = \underbrace {- \frac {1}{2 \lambda N ^ {2}} \sum_ {i = 1} ^ {N} \sum_ {j : y _ {j} = y _ {i}} z _ {i} ^ {\top} z _ {j}} _ {\text {T I G H T N E S S}} + \underbrace {\frac {1}{N} \sum_ {i} ^ {N} \log \sum_ {k = 1} ^ {K} \exp \left(\frac {\sum_ {j = 1} ^ {N} p _ {j k} z _ {i} ^ {\top} z _ {j}}{\lambda N}\right) - \frac {1}{2 \lambda} \sum_ {k = 1} ^ {K} \| c _ {k} \|} _ {\text {D I V E R S I T Y}}, \tag {2}
$$
where  $z_{i}$  represents the image feature in the embedding space,  $p_{ij}$  represents the softmax probability of point  $z_{i}$  belonging to class  $j$ ,  $c_{k} = \sum_{i=1}^{N} p_{ik} z_{i}$  represents the soft mean of class  $k$  and  $\lambda \in \mathcal{R}$  is to make sure  $L_{CE}$  is a convex function with respect to encoder  $\Phi_{w}$ .
By incorporating the language priors, we turn Eq.(2) to a cross-modal pairwise cross-entropy loss. Specifically, as human language contains rich prior knowledge, text features can be considered as both hard mean  $r_k = \frac{1}{N_k} \sum_{i=1}^{N_k} v_k$  and soft mean  $r_k = \frac{1}{N} \sum_{i=1}^{N} p_{ik} v_i$  of image features at class  $k$ , where  $N_k$  represents the sample number of class  $k$  and  $N$  represents the total sample number. Then, we reformulate the original pairwise cross-entropy loss  $L_{PCE}$  as cross-modal pairwise cross-entropy loss  $L_{CPCE}$ :
$$
L _ {C P C E} = \underbrace {- \frac {1}{2 \lambda N} \sum_ {i = 1} ^ {N} v _ {i} ^ {\top} r _ {y _ {i}}} _ {\text {T I G H T N E S S}} + \underbrace {\frac {1}{N} \sum_ {i} ^ {N} \log \sum_ {k = 1} ^ {K} \exp \left(\frac {v _ {i} ^ {\top} r _ {k}}{\lambda}\right) - \frac {1}{2 \lambda} \sum_ {k = 1} ^ {K} \left\| r _ {k} \right\|} _ {\text {D I V E R S I T Y}}, \tag {3}
$$
Inspired by [53], we use meanNN entropy estimator [10] to estimate the diversity term in Eq.(3). The process is formulated as:
$$
L _ {C P C E} ^ {\text {d i v e r s i t y}} \propto \frac {D}{N (N - 1)} \sum_ {i = 1} ^ {N} \sum_ {j \neq i} ^ {N} \log \left(v _ {i} ^ {\top} r _ {j} + r _ {i} ^ {\top} r _ {j}\right), \tag {4}
$$
$$
L _ {C P C E} ^ {\text {t i g h t n e s s}} \propto - \frac {1}{N} \sum_ {i = 1} ^ {N} v _ {i} ^ {\top} r _ {y _ {i}}, \tag {5}
$$
where  $D$  represents the feature dimensions. To recover ordering relation, we propose an additional weighting term. Intuitively, each rank template will have a high similarity score with images of a close rank and a low similarity score with images of a distant rank. As such, we opt to weight the cross-modal features with  $w_{ij}$ , where  $w_{ij}$  are the distances in the label space. The final cross-modal ordinal pairwise loss is defined as follows:
$$
L _ {c o p} \left(y _ {i}\right) = \frac {1}{(B - 1)} \sum_ {j \neq i} ^ {B} w _ {y _ {i} j} \cdot \left(v _ {y _ {i}} + \gamma r _ {y _ {i}}\right) ^ {\top} r _ {j} - v _ {y _ {i}} ^ {\top} r _ {y _ {i}}, \tag {6}
$$
where  $\gamma$  controls the strength of the rank within rank templates and  $B$  represents the batchsize. To further refining the CLIP feature space, we also propose a simplified cross-modal ordinal pairwise loss  $L_{scop}$  with language-related parameters frozen(i.e.  $\gamma = 0$ ).
Global Context Prompts. Given that global context prompts significantly surpass manually designed discrete prompts in vision tasks [55, 27, 54], we integrate them with our complementary rank-specific prompts in RankFormer to enhance semantic alignment. Specifically, we randomly initialize  $L$  global context prompts, denoted as  $G = \{G_1, G_2, \dots, G_L\}$ , and concatenate them with rank-specific prompts in the word embedding space of CLIP.
Asymmetrical Contrastive Loss. In vanilla CLIP [42], models are optimized using the standard contrastive loss, including a text-image contrastive loss  $L_{t2i}$  and an image-text contrastive loss  $L_{i2t}$ . In this work, we replace the original contrastive loss with an asymmetrical contrastive loss due to
many-to-many image-text mappings within a batch. In other words, different images in a batch may have the same rank, and the same rank may have the same description. Therefore, we replace  $t_i$  with  $t_{y_i}$  in both text-image contrastive loss  $L_{t2i}$  and image-text contrastive loss  $L_{i2t}$ . Specifically, our improved asymmetrical contrastive loss is defined as follows:
$$
L _ {t 2 i} \left(y _ {i}\right) = \frac {1}{\left| Z \left(y _ {i}\right) \right|} \sum_ {z \in Z \left(y _ {i}\right)} \log \frac {\exp \left(v _ {z} ^ {\top} r _ {y _ {i}} / \tau\right)}{\sum_ {j = 1} ^ {B} \exp \left(v _ {j} ^ {\top} r _ {y _ {i}} / \tau\right)}, \tag {7}
$$
where  $\tau$  is the temperature parameter and  $Z(y_{i}) = \{z\in 1\dots B:y_{z} = y_{i}\}$
3.3 Loss functions
We employ a two-stage training scheme to refine our approach following [27]. In the first stage, our focus is to enhance the ordinal information of the vanilla rank templates. We utilize cross-modal ordinal pairwise loss  $L_{cop}$  and the asymmetrical contrastive losses  $L_{t2i}$  and  $L_{i2t}$ . In the second stage, our objective is to simultaneously improve both semantic alignment and ordering alignment within the CLIP latent space. We use the simplified cross-modal ordinal pairwise loss  $L_{scop}$  and the cross-entropy loss  $L_{ce}$ . More detail can be found in Sec. 4.1.
We adopt the ViT-B/16 visual encoder and the text encoder from CLIP [42] as the backbone for our image and text feature extractor. All training data is resized to  $224 \times 224$  and subjected to random horizontal flipping. For all experiments, we employ the Adam [17] optimizer with default settings. The learning rates for the RankFormer and the visual backbone are  $3.5 \times 10^{-4}$  and  $1 \times 10^{-5}$ , respectively. The model is trained for 20 epochs in the first stage and 40 epochs in the second stage, with the learning rate decayed by a factor of 0.1 in epoch 30. We set  $L = 5$  in most of our experiments. For MORPH and CLAP2015, we set  $L_{t2i}$ ,  $L_{i2t}$ , and  $L_{cop}$  with weights of 0.03, 0.03, and 3 for the first stage, and  $L_{ce}$  and  $L_{scop}$  with weights of 1 and 1 for the second stage. For other datasets, we set  $L_{t2i}$ ,  $L_{i2t}$ , and  $L_{cop}$  with weights of 0.1, 0.1, and 3 for the first stage, and  $L_{ce}$  and  $L_{scop}$  with weights of 1 and 1 for the second stage. All experiments are conducted on a single NVIDIA 3090 GPU.
4.2 Age Estimation
Table 1: Results on MORPH II and CLAP2015.
Methods	Morph	CLAP2015
MAE(↓)	MAE(↓)
AGEn [50]	2.52	2.94
BridgeNet [30]	2.38	2.87
AVDL [52]	2.37	-
POE [28]	2.35	-
DRC-ORID [19]	2.26	-
PML [6]	2.15	2.91
MWR [48]	2.13	2.77
Vanilla CLIP [42]	6.91	4.66
CoOp [55]	2.39	2.75
OridinalCLIP [29]	2.32	-
L2RCLIP(Ours)	2.13	2.62
Table 2: Results on Adience dataset.
Methods	Adience
Accuracy(%,↑)	MAE(↓)
OR-CNN [37]	56.7	0.54
CNNPOR [34]	57.4 ± 5.8	0.55 ± 0.08
GP-DNNOR [35]	57.4 ± 5.5	0.54 ± 0.07
SORD [7]	59.6 ± 3.6	0.49 ± 0.05
POE [28]	60.5 ± 4.4	0.47 ± 0.06
MWR [48]	62.6	0.45
GOL [20]	62.5	0.43
Vanilla CLIP [42]	43.3 ± 3.6	0.80 ± 0.02
CoOp [55]	60.6 ± 5.5	0.50 ± 0.08
OridinalCLIP [29]	61.2 ± 4.2	0.47 ± 0.06
L2RCLIP(Ours)	66.2 ± 4.4	0.36 ± 0.05
Datasets. Age estimation aims to predict the age of a given facial image. We train and evaluate our method on the widely-used MORPH II [44] dataset, CLAP2015 [9] dataset, and Adience [21] dataset. MORPH II [44] is one of the largest and most commonly used longitudinal face databases and we follow the widely adopted training and evaluation following [45, 28, 48, 47]. CLAP2015 [9] is used for apparent age estimation, where each image is rated by at least 10 annotators, and the mean rating