SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Comparison and compatibility to end-to-end regression methods. RNC is compatible to end-to-end regression methods, and consistently improves the performance over all datasets.
Metrics	AgeDB	TUAB	MPIIFaceGaze	SkyFinder
MAE↓	R2↑	MAE↓	R2↑	Angular↓	R2↑	MAE↓	R2↑
L1	6.63	0.828	7.46	0.655	5.97	0.744	2.95	0.860
RNC(L1)	6.14 (+0.49)	0.850 (+0.022)	6.97 (+0.49)	0.697 (+0.042)	5.27 (+0.70)	0.815 (+0.071)	2.86 (+0.09)	0.869 (+0.009)
MSE	6.57	0.828	8.06	0.585	6.02	0.747	3.08	0.851
RNC(MSE)	6.19 (+0.38)	0.849 (+0.021)	7.05 (+1.01)	0.692 (+0.107)	5.35 (+0.67)	0.802 (+0.055)	2.86 (+0.22)	0.869 (+0.018)
HUBER	6.54	0.828	7.59	0.637	6.34	0.709	2.92	0.860
RNC(HUBER)	6.15 (+0.39)	0.850 (+0.022)	6.99 (+0.60)	0.696 (+0.059)	5.15 (+1.19)	0.830 (+0.121)	2.86 (+0.06)	0.869 (+0.009)
DEX [36]	7.29	0.787	8.01	0.537	5.72	0.776	3.58	0.778
RNC(DEX)	6.43 (+0.86)	0.836 (+0.049)	7.23 (+0.78)	0.646 (+0.109)	5.14 (+0.58)	0.805 (+0.029)	2.88 (+0.70)	0.865 (+0.087)
DLDL-v2 [14]	6.60	0.827	7.91	0.560	5.47	0.799	2.99	0.856
RNC(DLDL-v2)	6.32 (+0.28)	0.844 (+0.017)	6.91 (+1.00)	0.697 (+0.137)	5.16 (+0.31)	0.802 (+0.003)	2.85 (+0.14)	0.869 (+0.013)
OR [33]	6.40	0.830	7.36	0.646	5.86	0.770	2.92	0.861
RNC(OR)	6.34 (+0.06)	0.843 (+0.013)	7.01 (+0.35)	0.688 (+0.042)	5.13 (+0.73)	0.825 (+0.055)	2.86 (+0.06)	0.867 (+0.006)
CORN [40]	6.72	0.811	8.11	0.597	5.88	0.762	3.24	0.819
RNC(CORN)	6.44 (+0.28)	0.838 (+0.027)	7.22 (+0.89)	0.663 (+0.066)	5.18 (+0.70)	0.820 (+0.058)	2.89 (+0.35)	0.862 (+0.043)
Datasets. We perform extensive experiments on five regression datasets that span different tasks and domains, including computer vision, human-computer interaction, and healthcare. Complete descriptions of each dataset and example inputs are in Table 7, Appendix C and D.
- AgeDB (Age) [32, 44] is a dataset for predicting age from face images, containing 16,488 in-the-wild images of celebrities and the corresponding age labels.
- IMDB-WIKI (Age) [36, 44] is a dataset for age prediction from face images, which contains 523,051 celebrity images and the corresponding age labels. We use this dataset only for the analysis.
- TUAB (Brain-Age) [34, 11] aims for brain-age estimation from EEG resting-state signals, with 1,385 21-channel EEG signals sampled at  $200\mathrm{Hz}$  from individuals with age from 0 to 95.
- MPIIFaceGaze (Gaze Direction) [51, 52] contains 213,659 face images collected from 15 participants during natural everyday laptop use. The task is to predict the gaze direction.
- SkyFinder (Temperature) [31, 7] contains 35,417 images captured by 44 outdoor webcam cameras for in-the-wild temperature prediction.
Metrics. We report two distinct metrics for model evaluation: ① Prediction error, and ② Coefficient of determination  $(\mathbb{R}^2)$ . Prediction error provides practical insight for interpretation, while  $\mathbb{R}^2$  quantifies the difficulty of the task by measuring the extent to which the model outperforms a dummy regressor that always predicts the mean value of the training labels. We report the mean absolute error (MAE) for age, brain-age, and temperature prediction, and the angular error for gaze direction estimation.
Experiment Settings. We employ ResNet-18 [19] as the main backbone for AgeDB, IMDB-WIKI, MPIIFaceGaze, and SkyFinder. In addition, we confirm that the results are consistent with other backbones (e.g., ResNet-50), and report related results in Appendix G.1. For TUAB, a 24-layer 1D ResNet [19] is used as the backbone model to process the EEG signals, with a linear regressor acting as the predictor. We fix data augmentations to be the same for all methods, where the details are reported in Appendix D. Negative  $L_{2}$  norm (i.e.,  $-\| \pmb{v}_i - \pmb{v}_j\|_2$ ) is used as the feature similarity measure in  $\mathcal{L}_{\mathrm{RNC}}$ , with  $L_{1}$  distance as the label distance measure for AgeDB, IMDB-WIKI, SkyFinder, and TUAB, and angular distance as the label distance measure for MPIIFaceGaze. We provide complete experimental settings and hyper-parameter choices in Appendix F.
Comparison and compatibility to end-to-end regression methods. As explained earlier, our model learns a regression-suitable representation that can be used by any end-to-end regression methods. Thus, we implement seven generic regression methods:  $L_{1}$ , MSE, and HUBER [44] ask the model to directly predict the target and utilize an error-based loss function; DEX [36] and DLDL-v2 [14] divide the regression range of each label dimension into pre-defined bins and learn the probability
Table 3: Comparisons to state-of-the-art representation & regression learning methods. MAE $^{\downarrow}$  is used as the metric for AgeDB, TUAB, and SkyFinder, and Angular Error $^{\downarrow}$  is used for MPIIFaceGaze.  $L_{1}$  loss is employed as the default regression loss if not specified. RNC surpasses state-of-the-art methods on all datasets.
Method	AgeDB	TUAB	MPIIFaceGaze	SkyFinder
Representation learning methods (Linear Probing):
SIMCLR [4]	9.59	11.01	9.43	4.70
DINO [3]	10.26	11.62	11.92	5.63
SUPCON [25]	8.13	8.47	9.27	3.97
Representation learning methods (Fine-tuning):
SIMCLR [4]	6.57	7.57	5.50	2.93
DINO [3]	6.61	7.58	5.80	2.98
SUPCON [25]	6.55	7.41	5.54	2.95
Regression learning methods:
L1	6.63	7.46	5.97	2.95
LDS+FDS [44]	6.45	-	-	-
L2CS-NET [1]	-	-	5.45	-
LDE [7]	-	-	-	2.92
RANKSIM [17]	6.51	7.33	5.70	2.94
ORDINAL ENTROPY [50]	6.47	7.28	-	2.94
RNC(L1)	6.14	6.97	5.27	2.86
GAINS	+0.31	+0.31	+0.18	+0.06
distribution over them; OR [33] and CORN [40] design multiple ordered thresholds for each label dimension and learn a binary classifier for each threshold. Further details of these baselines are included in Appendix E.1. In our comparison, we first train the encoder with the proposed  $\mathcal{L}_{\mathrm{RNC}}$ . We then freeze the encoder and train a predictor on top of it using each of the baseline methods. The original baseline without the RNC representation is then compared to that with RNC. For instance, we denote the end-to-end training of the encoder and predictor using  $L_{1}$  loss as  $L_{1}$ , while  $\mathrm{RNC}(L_1)$  denotes training the encoder with  $\mathcal{L}_{\mathrm{RNC}}$  and subsequently training the predictor using  $L_{1}$  loss.
Table 2 summarizes the evaluation results on AgeDB, TUAB, MPIIFaceGaze and SkyFinder. Green numbers highlight the performance gains by using RNC representation. The standard deviations of the best results on each dataset are reported in Appendix G.2. As the table indicates, RNC consistently achieves the best performance on both metrics across all datasets. Moreover, incorporating RNC to learn the representation consistently reduces the prediction error of all baselines by  $5.8\%$ ,  $9.3\%$ ,  $11.7\%$ , and  $7.0\%$  on average on AgeDB, TUAB, MPIIFaceGaze, and SkyFinder, respectively.
Comparison to state-of-the-art representation learning methods. We further compare RNC with state-of-the-art representation learning methods, SIMCLR [4], DINO [3], and SUPCON [25], under both linear probing and fine-tuning schemes to evaluate their learned representations for regression tasks. Full details are in Appendix E.2. Note that SIMCLR and DINO do not use label information while SUPCON uses label information for training the encoder. The predictor is trained with  $L_{1}$  loss. Table 3 demonstrates that RNC outperforms all other methods across all datasets. Note that some of the representation learning schemes even underperform the vanilla  $L_{1}$  method, which further verifies that the performance gain of RNC stems from our proposed loss rather than the pre-training scheme.
Comparison to state-of-the-art regression learning methods. We also compare RNC with state-of-the-art regression learning schemes, including methods that perform regularization in the feature space [44, 17, 50] and methods that adopt dataset-specific techniques [1, 7]. We provide the details in Appendix E.3.  $L_{1}$  loss is employed as the default regression loss for all methods if not specified. The results are presented in Table 3, with a dash  $(-)$  indicating that the method is not applicable to the dataset. We observe that RNC achieves state-of-the-art performance across all datasets.
Robustness to Data Corruptions. Deep neural networks are widely acknowledged to be vulnerable to out-of-distribution data and various forms of corruption, such as noise, blur, and color distortions [21]. To analyze the robustness of RNC, we generate corruptions on the AgeDB test set using the corruption process from the ImageNet-C benchmark [21], incorporating 19 distinct types of corruption at varying severity levels. We compare  $\mathrm{RNC}(L_1)$  with  $L_{1}$  by training both models on the original AgeDB training set, but directly testing them on the corrupted test set across all severity levels. The
Figure 4: RNC is more robust to data corruptions. We show MAE $^{\ddagger}$  as a function of corruption severity on AgeDB test set.
Figure 5: RNC is more resilient to reduced training data. We show MAE $^{\downarrow}$  as a function of the number of training samples on IMDB-WIKI.
Table 4: Transfer learning results. RNC delivers better performance than  $L_{1}$  in all scenarios.
Metrics	AgeDB → IMDB-WIKI (subsampled, 2k)	IMDB-WIKI (subsampled, 32k) → AgeDB
Linear Probing	Fine-tuning	Linear Probing	Fine-tuning
MAE↓	R2↑	MAE↓	R2↑	MAE↓	R2↑	MAE↓	R2↑
L1	12.25	0.496	11.57	0.528	7.36	0.801	6.36	0.848
RNC(L1)	11.12 (+1.13)	0.556 (+0.060)	11.09 (+0.48)	0.546 (+0.018)	7.06 (+0.30)	0.812 (+0.011)	6.13 (+0.23)	0.850 (+0.002)
reported results are averaged over all types of corruptions. Fig. 4 illustrates the results, where the representation learned by RNC is more robust to unforeseen data corruptions, with consistently less performance degradation when corruption severity increases.
Resilience to Reduced Training Data. The availability of massive training datasets has played an important role in the success of modern deep learning. However, in many real-world scenarios, the cost and time involved in labeling large training sets make it infeasible to do so. As a result, there is a need to enhance model resilience to limited training data. To investigate this, we subsample IMDB-WIKI to generate training sets of varying sizes and compare the performance of RNC( $L_{1}$ ) and  $L_{1}$ . As Fig. 5 confirms, RNC is more robust to reduced training data and displays less performance degradation as the number of training samples decreases.
Transfer Learning. We evaluate whether the learned representations by RNC are transferable across datasets. To do so, we first pre-train the feature encoder on a large dataset, and subsequently utilize either linear probing (fixed encoder) or fine-tuning to learn a predictor on a small dataset (which shares the same prediction task). We investigate two scenarios: 1 Transferring from AgeDB which contains  $\sim 12\mathrm{K}$  samples to a subsampled IMDB-WIKI of 2K samples, and 2 Transferring from another subsampled IMDB-WIKI of 32K samples to AgeDB. As shown in Table 4,  $\mathrm{RNC}(L_1)$  exhibits consistent and superior performance compared to  $L_{1}$  in both linear probing and fine-tuning settings for both of the aforementioned scenarios.
Robustness to Spurious Targets. We show that RNC is able to deal with spurious targets that arise in data [46], while existing regression learning methods often fail to learn generalizable features. Specifically, the SkyFinder dataset naturally has a spurious target: different webcams (with distinct locations). Therefore, we show in Fig. 6 the UMAP [30] visualization of the learned features obtained from both  $L_{1}$  and RNC, using data from 10 webcams in SkyFinder. The first column of the figure is colored by the ground-truth target (temperature), while the second column is colored by the 10 different webcams. Our results demonstrate that the representation learned by  $L_{1}$  is clustered by webcam, indicating that it is prone to capturing easy-to-learn features. In contrast, RNC learns the underlying continuous temperature information even in the presence of strong spurious tar
Figure 6: Robustness to spurious targets. RNC is able to capture underlying continuous targets & learn robust representations that generalize.
gets, confirming its ability to learn robust representations that generalize.
Table 6: Ablation studies for RNC. All experiments are performed on the AgeDB dataset and  $L_{1}$  is used as the default loss for training the predictor. Default settings used in the main experiments are marked in gray.
(a) Number of Positives  $K$
MAE↓	R²↑
128	6.46	0.828
256	6.43	0.833
384	6.29	0.845
511	6.14	0.850
(b) Feature Similarity Measure
MAE↓	R2↑
cosine	6.51	0.836
negative L1 norm	6.25	0.842
negative L2 norm	6.14	0.850
(c) Training Scheme
MAE↓	R2↑
linear probing	6.14	0.850
fine-tuning	6.36	0.844
regularization	6.42	0.833
Generalization to Unseen Targets. In practical regression tasks, it is frequently encountered that some targets are unseen during training. To investigate zero-shot generalization to unseen targets, following [44], we curate two subsets of IMDB-WIKI that contain unseen targets during training, while keeping the test set uniformly distributed across the target range. Table 5 shows the label distributions, where regions of unseen targets are marked with pink shading and those of seen targets are marked with blue shading. The first training set has a bi-modal Gaussian
Table 5: Zero-shot generalization to unseen targets. We create two IMDB-WIKI training sets with missing targets, and keep test sets uniformly distributed across the target range.  $\mathrm{MAE}^{\downarrow}$  is used as the metric.
Label Distribution	Method	All	Seen	Unseen
L1	12.53	10.82	18.40
RNC(L1)	11.69	10.46	15.92
(+0.84)	(+0.36)	(+2.48)
L1	11.94	10.43	14.98
RNC(L1)	10.88	9.78	13.08
(+1.06)	(+0.64)	(+1.90)
distribution, while the second one exhibits a tri-modal Gaussian distribution over the target space. The results confirm that  $\mathrm{RNC}(L_1)$  outperforms  $L_{1}$  by a larger margin on the unseen targets without sacrificing the performance on the seen targets.
Ablation on Number of Positives. Recall that in  $\mathcal{L}_{\mathrm{RNC}}$ , all samples in the batch will be treated as the positive for each anchor. Here, we conduct an ablation study on only considering the first  $K$  closest samples to the anchor as positive. Table 6a shows the results on AgeDB for different  $K$ , where  $K = 511$  represents the scenario where all samples are considered as positive (default batch size  $N = 256$ ). The experiments reveal that larger values of  $K$  lead to better performance, which aligns with the intuition behind the design of  $\mathcal{L}_{\mathrm{RNC}}$ : each contrastive term ensures that a group of orders related to the positive sample is maintained. Specifically, it guarantees that all samples that have a larger label distance from the anchor than the positive sample are farther from the anchor than the positive sample in the feature space. Only when all samples are treated as positive and their corresponding groups of orders are preserved can the order in the feature space be fully guaranteed.
Ablation on Similarity Measure. Table 6b shows the performance of using different feature similarity measures  $\mathrm{sim}(\cdot, \cdot)$ . Compared to cosine similarity, both the negative  $L_{1}$  norm and  $L_{2}$  norm produce significantly better results. This improvement can be attributed to the fact that cosine similarity only captures the directions of feature vectors, while the negative  $L_{1}$  or  $L_{2}$  norm takes both the direction and magnitude of the vectors into account. This finding highlights the potential differences between representation learning for classification and regression tasks, where the standard practice for classification is to use cosine similarity [4, 25], while our findings suggest the superiority of  $L_{1}$  and  $L_{2}$  norm for regression-based representation learning.
Ablation on Training Scheme. There are typically three schemes to train the encoder: ① Linear probing, which first trains the feature encoder using the representation learning loss, then fixes the encoder and trains a linear regressor on top of it using a regression loss; ② Fine-tuning, which first trains the feature encoder using the representation learning loss, then fine-tunes the entire model using a regression loss; and ③ Regularization, which trains the entire model while jointly optimizing the representation learning & the regression loss. Table 6c shows the results for the three schemes using  $\mathcal{L}_{\mathrm{RNC}}$  as the representation learning loss and  $L_{1}$  as the regression loss. All three schemes can improve performance over using  $L_{1}$  loss alone. Further, unlike classification problems where fine-tuning often delivers the best performance, freezing the feature encoder yields the best outcome in regression. This is because, in the case of regression, back-propagating the  $L_{1}$  loss to the representation can disrupt the order in the embedding space learned by  $\mathcal{L}_{\mathrm{RNC}}$ , leading to inferior performance.
5.4 Further Discussions
Does RNC rely on data augmentation for superior performance (Appendix G.3)? Table 10 confirms that RNC surpasses the baseline, with or without data augmentation. In fact, unlike typical (unsupervised) contrastive learning techniques that rely on data augmentation for distinguishing the augmented views, data augmentation is not essential for RNC. This is because RNC contrasts samples according to label distance rather than the identity. The role of data augmentation for RNC is similar to its role in typical regression learning, which is to enhance model generalization.
Does the benefit of RNC come from the two-stage training scheme (Appendix G.4)? We confirm in Table 11 that training a predictor on top of the representation learned by the competing regression baselines does not improve performance. In fact, it can even be detrimental to performance. This finding validates that the benefit of RNC is due to the proposed  $\mathcal{L}_{\mathrm{RNC}}$  and not the training scheme. The generic regression losses are not designed for representation learning and are computed based on the final model predictions, which fails to guarantee the final learned representation.
Computational cost of RNC (Appendix G.5)? We verify in Table 12 that the training time of RNC is comparable to standard contrastive learning methods (e.g., SUPCON [25]), indicating that RNC offers similar training efficiency, while achieving notably better performance for regression tasks.
6 Broader Impacts and Limitations
Broader Impacts. We introduce a novel framework designed to enhance the performance of generic deep regression learning. We believe this will significantly benefit regression tasks across various real-world applications. Nonetheless, several potential risks warrant discussion. First, when the framework is employed to regress sensitive personal attributes such as intellectual capabilities, health status, or financial standing from human data, there's a danger it might reinforce or even introduce new forms of bias. Utilizing the method in these contexts could inadvertently justify discrimination or the negative targeting of specific groups. Second, in contrast to handcrafted features which are grounded in physical interpretations, the feature representations our framework learns can be opaque. This makes it difficult to understand and rationalize the model's predictions, particularly when trying to determine if any biases exist. Third, when our method is trained on datasets that do not have a balanced representation of minority groups, there's no assurance of its performance on these groups being reliable. It is essential to recognize that these ethical concerns apply to deep regression models at large, not solely our method. However, the continuous nature of our representation which facilitates interpolation and extrapolation might inadvertently make it more tempting to justify such unethical applications. Anyone seeking to implement or use our proposed method should be mindful of these concerns. Both our specific method and deep regression models, in general, should be used cautiously to avoid situations where their deployment might contribute to unethical outcomes or interpretations.
**Limitations.** Our proposed method presents some limitations. Firstly, the technique cannot discern spurious or incidental correlations between the input and the target within the dataset. As outlined in the Broader Impact section, this could result in incorrect conclusions potentially promoting discrimination or unjust treatment when utilized to deduce personal attributes. Future research should delve deeper into the ethical dimensions of this issue and explore strategies to ensure ethical regression learning. A second limitation is that our evaluation primarily focuses on general regression accuracy metrics (e.g., MAE) without considering potential disparities when evaluating specific subgroups (e.g., minority groups). Given that a regression model's performance can vary across demographic segments, subgroup analysis is an avenue that warrants exploration in subsequent studies. Lastly, our approach learns continuous representations by contrasting samples against one another based on their ranking in the target space, necessitating label information. To adapt it for representation learning with unlabeled data, our framework will need some modifications, which we reserve for future work.
We present Rank-N-Contrast (RNC), a framework that learns continuous representations for regression by ranking samples according to their labels and then contrasting them against each other based on their relative rankings. Extensive experiments on different datasets over various real-world tasks verify the superior performance of RNC, highlighting its intriguing properties such as better data efficiency, robustness to corruptions and spurious targets, and generalization to unseen targets.
Acknowledgements. We are grateful to Hao He for his invaluable assistance with the theoretical analysis in the paper. We thank the members of the NETMIT group for their constructive feedback on the draft of this paper. We extend our appreciation to the anonymous reviewers for their insightful comments and suggestions that greatly helped in improving the quality of the paper. Lastly, we acknowledge the generous support from the GIST-MIT program, which funded this project.
Recall that in Eqn. (3) we defined  $\mathcal{L}_{\mathrm{RNC}} := \frac{1}{2N} \sum_{i=1}^{2N} \frac{1}{2N-1} \sum_{j=1, j \neq i}^{2N} -\log \frac{\exp(s_{i,j})}{\sum_{v_k \in S_{i,j}} \exp(s_{i,k})}$ , where  $\mathcal{S}_{i,j} := \{\pmb{v}_k \mid k \neq i, d_{i,k} \geq d_{i,j}\}$ . We rewrite it as
$$
\begin{array}{l} \mathcal{L}_{\text{RNC}} = -\frac{1}{2N(2N - 1)}\sum_{i = 1}^{2N}\sum_{j\in [2N]\setminus \{i\}}\log \frac{\exp(s_{i,j})}{\sum_{k\in[2N]\setminus\{i\}, d_{i,k}\geq d_{i,j}}\exp(s_{i,k})} \\ = - \frac {1}{2 N (2 N - 1)} \sum_ {i = 1} ^ {2 N} \sum_ {m = 1} ^ {M _ {i}} \sum_ {j \in [ 2 N ] \backslash \{i \}, d _ {i, j} = D _ {i, m}} \log \frac {\exp \left(s _ {i , j}\right)}{\sum_ {k \in [ 2 N ] \backslash \{i \} , d _ {i , k} \geq D _ {i , m}} \exp \left(s _ {i , k}\right)} \\ = -\frac{1}{2N(2N - 1)}\sum_{i = 1}^{2N}\sum_{m = 1}^{M_{i}}\sum_{j\in [2N]\backslash \{i\}, d_{i,j} = D_{i,m}}\log \frac{\exp(s_{i,j})}{\sum_{k\in[2N]\backslash\{i\},d_{i,k} = D_{i,m}}\exp(s_{i,k})} \\ + \frac {1}{2 N (2 N - 1)} \sum_ {i = 1} ^ {2 N} \sum_ {m = 1} ^ {M _ {i}} \sum_ {j \in [ 2 N ] \backslash \{i \}, d _ {i, j} = D _ {i, m}} \log \left(1 + \frac {\sum_ {k \in [ 2 N ] \backslash \{i \} , d _ {i , k} > D _ {i , m}} \exp \left(s _ {i , k} - s _ {i , j}\right)}{\sum_ {k \in [ 2 N ] \backslash \{i \} , d _ {i , k} = D _ {i , m}} \exp \left(s _ {i , k} - s _ {i , j}\right)}\right) \\ > - \frac {1}{2 N (2 N - 1)} \sum_ {i = 1} ^ {2 N} \sum_ {m = 1} ^ {M _ {i}} \sum_ {j \in [ 2 N ] \backslash \{i \}, d _ {i, j} = D _ {i, m}} \log \frac {\exp \left(s _ {i , j}\right)}{\sum_ {k \in [ 2 N ] \backslash \{i \} , d _ {i , k} = D _ {i , m}} \exp \left(s _ {i , k}\right)}. \\ \end{array}
$$
$\forall i \in [2N], m \in [M_i]$ , from Jensen's Inequality we have