Rank-N-Contrast: Learning Continuous Representations for Regression
Authors: Kaiwen Zha
Peng Cao
Jeany Son
Yuzhe Yang
Dina Katabi
Venue: NeurIPS 2023 spotlight (2023)
Keywords: regression
representation learning
continuity
TL;DR: We propose a generic framework that learns continuous representations for regression, which not only boosts performance but also improves robustness, data efficiency, and generalization.

Abstract: Deep regression models typically learn in an end-to-end fashion without explicitly emphasizing a regression-aware representation. Consequently, the learned representations exhibit fragmentation and fail to capture the continuous nature of sample orders, inducing suboptimal results across a wide range of regression tasks. To fill the gap, we propose Rank-N-Contrast (RNC), a framework that learns continuous representations for regression by contrasting samples against each other based on their rankings in the target space. We demonstrate, theoretically and empirically, that RNC guarantees the desired order of learned representations in accordance with the target orders, enjoying not only better performance but also significantly improved robustness, efficiency, and generalization. Extensive experiments using five real-world regression datasets that span computer vision, human-computer interaction, and healthcare verify that RNC achieves state-of-the-art performance, highlighting its intriguing properties including better data efficiency, robustness to spurious targets and data corruptions, and generalization to distribution shifts.

Rank-N-Contrast. Learning Continuous Representations for Regression
Kaiwen Zha
Peng Cao
Jeany Son
Yuzhe Yang'
Dina Katabi
MIT CSAIL
2GIST
Abstract
Deep regression models typically learn in end-to-end fashion without explicitly emphasizing a regression-aware representation. Consequently. the learned repre- sentations exhibit fragmentation and fail capture the continuous nature of sample orders, inducing suboptimal results across a wide range of regression tasks. To fill the gap, we propose Rank-N-Contrast (RNC), framework that learns continuous representations for regression by contrasting samples against each other based on their rankings in the target space. We demonstrate, theoretically and empirically, that RNC guarantees the desired order of learned representations in accordance with the target orders, enjoying not only better performance but also significantly im- proved robustness, efficiency, and generalization. Extensive experiments using five real-world regression datasets that span computer vision, numan-computer interac- tion, and healthcare verify that RNC achieves state-of-the-arr performance, high- lighting its intriguing properties including better data efficiency, robustness to spu- rious targets and data corruptions, and generalization to distribution shifts. Code is available at: https /tte:://ggttttt github.. com/ wwwa///aawwnnaaaaanaaaa N Contr
b
SupCon
RNC (Ours)
Toup (C)
30
20
10
0
Figure 1: Learned representations of different methods on real-world temperature regression task [7] (details Sec. 5). Existing general regression learning (L1) representation learning (SupCon) schemes fail to recognize underlying continuous information in data. contrast, RNC learns continuous representations that capture the intrinsic sample orders w.r.t. the regression targets.
Introduction
Regression problems are pervasive and fundamental in the real world, spanning various tasks and domains including estimating age from human appearance [36], predicting health scores via human physiological signals [1 [1]], and detecting gaze directions using webcam images [52]. Due to the
'The two authors contributed equally and order was determined by a random coin flip. Correspondence to Kaiwen Zha <kzha @ mit.edux, Peng Cao <pengcao mit.edux.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

continuity in regression targets, the most widely adopted approach for training regression models is to directly predict the target value and employ a distance-basee based loss function, such as 11 or Lg distance, between the prediction and the ground-truth target [51, 52, 38]. Methods that tackle regression tasks using classification models trained with cross-entropy loss have also been studied [36, 33, 40].
However, previous methods focus on imposing constraints on the final predictions in an end-to-end fashion, but do not explicitly emphasize the representations learned by the model. Unfortunately, these representations are often fragmented and incapable of capturing the continuous relationships that underlie regression tasks. Fig. (a) highlights the representations learned by the L1 loss on SkyFinder [7], regression dataset for predicting weather temperature from webcam outdoor images captured different locations (details in Sec. 5). Rather than exhibiting the continuous ground -tuth temperatures, the learned representations are grouped by different webcams in a fragmented manner. Such unordered and fragmented representation is suboptimal for the regression task and can even hamper performance by including irrelevant information, such as the capturing wehcam.
Furthermore, despite the great success of representation learning schemes on solving discrete classifi- cation or segmentation tasks (e.g., contrastive learning [4, 20] and supervised contrastive learning (SupCon) [25]), less attention has been paid to designing algorithms that capture the intrinsic continu- ity in data for regression. Interestingly, we highlight that existing representation learning methods inevitably overlook the continuous nature in data: Fig. 1(b) shows the representation learned by SupCon on the SkyFinder dataset, where it again fails to capture the underlying continuous order between the samples, resulting in a suboptimal representation for regression tasks.
To fill the gap, we present Rank-N-Contrast (RNC), a novel framework for generic regression learning. RNC first learns a regression-aware representation that orders the distances in the embedding space based on the target values, and then leverages it to predict the continuous targets. To achieve this, we propose the Rank-N-Contrast loss (CrNc), which ranks the samples in batch according their labels and then contrasts them against each other based on their relative rankings. Theoretically, we prove that optimizing Luee results in features that are ordered according to the continuous labels, leading to improved performance in downstream regression tasks. As confirmed in Fig. 1(c). RNC learns continuous representations that capture the intrinsic ordered relationships between samples. Notably, our framework is orthogonal to existing regression methods, allowing for the use of any regression method to map the learned representation to the final prediction values.
To support practical evaluations, we benchmark RNC against state-of-the-ard (SOTA) regression and representation learning schemes five real-world regression datasets that span computer vision, human-computer interaction, and healthcare. Rigorous experiments verify the superior performance, robustness, and efficiency of RNC on learning continuous targets. Our contributions are as follows:
We identify the limitation of current regression and representation learning methods for continuous targets, and uncover intrinsic properties of learning regression-awe aware representations.
We design RNC, a simple & effective method that learns continuous representations for regression.
We conduct extensive experiments on five diverse regression datasets in vision, human-computer interaction, and healthcare, verifying the superior performance of RNC against SOTA schemes.
Further analyses reveal intriguing properties of RNC on its data efficiency, robustness to spurious targets & data corruptions, and better generalization to unseen targets.
2 Related Work
Regression Learning. Deep learning has achieved great success in addressing regression tasks [36, 51, 5,,2233 38, 11]. In regression learning, the final predictions of the model are typically trained in an end-to-end manner to be close to the targets. Standard regression losses include the L1 loss, the mean squared error (MSE) loss, and the Huber loss [22]. Past work has also proposed several variants. One branch of work [36, 13, 14, 35] divides the regression range into small bins, converting the problem into a classification task. Another line of work [12, 2, 40] casts regression as an ordinal classification problem [33] using ordered thresholds and employing multiple binary classifiers. Recently, a line of work proposes to regularize the embedding space for regression, ranging from modeling feature space uncertainty [28), encouraging higher-entropy feature spaces [50], to regularizing features for imbalanced regression [44, 17]. In contrast to existing works, we provide regression-aware representation learning approach that emphasizes the continuity in the features space w.r.t. the targets, which enjoys better performance while being compatible to prior regression schemes. In addition,
2

---

Anchor
Positive Pair
Corresponding Negative Pair(s)
20
0
Age:
Age:
Age: 30
Age: 70
20
20
70
(a) A Batch of Samples
(b) Pair Construction for RNC
Figure 2: Illustration of Lasc in the context of positive and negative pairs. (a) An example batch of input data and their labels.
(b) Two example positive pairs and corresponding negative pair(s) when the anchor is the 20-year-old man (shown in gray shading). When the anchor forms positive pair with a 30-year-old man, their label distance is 10, hence the corresponding negative samples are the O-ycar-old baby and the 70-year-old man, whose label distances to the anchor are larger than 10. When the O-ycar-old baby creates a positive pair with the anchor, only the 70-ycar-old man has a larger label distance the anchor, thus serving as negative sample.
C-Mixup [47] adapts the original mixup [49] by adjusting the sampling probability of the mixed pairs according to the target similarities. is also worth noting that our method is orthogonal and complementary to data augmentation algorithms for regression learning, such C-Mixup [47].
Representation Learning. Representation learning crucial in machine learning, often studied in the context of classification. Recently, contrastive learning has emerged as a popular technique for self-suuer representation learning [4, 20, 5, 8, 3]. The supervised version of contrastive learning.
SupCon [25], has been shown to outperform the conventional cross- croos-yn loss on multiple discrete classification tasks, including image recognition [25], noisy labels [26], long-tailed classification
[23, 27], and out-of-domain detection [48]. A few recent papers propose to adapt SupCon to tackle ordered labels specific downstream applications, including gaze estimation [42], medical imaging [9, 10], and neural behavior analysis [37]. Different from prior works that directly adapt SupCon, we incorporate the intrinsic property of label continuity for designing representation learning scheme tailored for regression, which offers a simple and principled approach for generic regression tasks.
3 Our Approach: Rank-N-Contrast (RNC)
Problem Setup. Given a regression task, we aim to train a neural network model composed of feature encoder f()) X + Rd and a predictor g(-): Rd. + gd to predict the target y E Rd based on the input data x E X.
1}. (1. Given randomly sampled batch of N input and label pairs {I Mn)) 3n)) we apply standard data augmentations to obtain a two-view batch ((t,, ÿe) beetee)))) where 22 = t(x,,, and 2 = t' (xx),, with t and t' being independently sampled augmentation operations, and 8/2n 92- = ya. Vn E [N]. The augmented batch is then fed into the encoder f to obtain the feature embedding for each augmented input data, i.e., en (mx) E IRd,, VI E [2N]. The representation learning phase is then performed over the feature embeddings. To harness the acquired representation for regression, we freeze the encoder f(.) and train the predictor 9()) on top of it using regression loss (e.g.. L1 loss).
For positive integer I. let [Z] denote the set {1,
In this context, a natural question arises: How to design regression representation learning scheme tailored for continuous and ordered samples?
The Lank-N-Contrase Loss. In order align distances in the embedding space ordered by distances in their labels, we propose Bank-N-Contran loss (CRwc), which first ranks the samples according to their target distances, and then contrasts them against each other based on their relative rankings.
Following [16, 45], given an anchor Us we model the likelihood of any other vy to increase expo- nentially with respect to their similarity in the representation space [15]. Inspired by the listwise ranking methods [43, 6], we introduce St,, := {wx k: & i, d(ğ., ÿn) 2 d( } to denote the set of samples that are of higher ranks than v, in terms of label distance w.r.t. U; where d(-,-) is the distance measure between two labels (e.g., L1 distance). Then the normalized likelihood of vy given v. and St can be written
exp(sim(u,, 0z)/T)
Po (1.))) Σ Eeges,, exp(sim(vi, vx (x)
(1)
3

---

where sim(:,-) is the similarity measure between two feature embeddings (e.g.. negative L2 norm) and T denotes the temperature parameter. Note that the denominator is a sum over the set of samples that possess higher ranks than ;j; Maximizing P(U 0; Si,) effectively increases the probability that tyy outperforms the other samples in the set and emerges the top rank within Si.j-
As result, we define the per-sample RNC loss as the average negative log-likelihood over all other samples in given batch:
exp(sim(v, v2)/T)
Σ J_11
- log Σ Uy,
(2)
RCC
2N
(.. bx)/T)
Intuitively, for an anchor sample any other sample j in the batch is contrasted with it, enforeing the feature similarity between and j to be larger than that of i and any other sample k in the batch,
if the label distance between and k is larger than that of and Minimizing & will align the
orders of feature embeddings with their corresponding orders in the label space w.r.t. anchor i.
LrC is then enumerating over all 2N samples as anchors to enforce the entire feature embeddings ordered according to their orders in the label space:
2N - log exp(sim(U,: v2y/T) J Emges., exp(sim U. T/)/T)
CRCC
( RNC
1
1
Σ
(3)
2N
2NV
2N
Interpretation. To exploit the inherent continuity underlying the labels, Lasc ranks samples in a batch with respect to their label distances to the anchor. When contrasting the anchor the sample in the batch that is closest in the label space, it enforces their similarity to be larger than all other samples in the batch. Similarly, when contrasting the anchor the second closest sample in the batch, it enforces their similarity to be larger than only those samples that have a rank of three or higher in terms of distance to the anchor. This process repeated for higher-rank samples (i.e., the third closest, fourth closest, etc.) and for all anchors in a batch.
Feature Ordinality. We further examine the impact of RNC on the ordinality of learned features. Fig. 3 visualizes the feature similarity matrices obtained from 2,000 randomly sampled data points in real- world temperature regression task [7] for models trained using the vanilla T, loss and RNC. For clarity, the data points sorted based on their ground-truth labels, with the expectation that the matrix values decrease progressively from the diagonal to the periphery. Notably, our method, RNC, exhibits a more discernible pattern compared to the L1 loss. Furthermore, we calculate two quantitative metrics, the Spearman's 's rank correlation coefficient [41] and the Kendall rank correlation coefficient [24], between the label similarities and the feature similarities for both methods. The results in Table confirm that the feature similarities learned by our method have a significantly higher correlation with label similarities than those by the L, loss.
Table Correlation between feature & label similarities.
Spearman's p
Kendall's 7
L RNC
0.822
0.664
0.971
0.870
4
RXC
Figure 3: Feature similarity matrices sorted by labels.
Connections to Contrastive Learning. The loss can also be explained in the context of positive and negative pairs in contrastive learning. Contrastive learning and SupCon are designed for classification tasks, where positive pairs consist of samples that belong to the same class the same input image, while all other samples are considered as negatives. In regression, however, there are no distinct classes but rather continuous labels [44, 46]. Thus in LRNG, any two samples can be considered as a positive or negative pair depending on the context. For given anchor sample i, any other sample j in the same batch can be used construct a positive pair with the corresponding negative samples set to all samples in the batch whose labels differ from i's label by more than the label of j. Fig. 2(a) shows an example batch, and Fig. 2(b) shows two positive pairs and their corresponding negative pair(s).
4 Theoretical Analysis
In this section, we theoretically prove that optimizing Crae results in an ordered feature embedding that corresponds to the ordering of the labels. All proofs are in Appendix A.
Notations. Let 8t.j = sim wy)/T and di,, == d(ğr, ÿ5). Vi,j E [2N]. Let DA,1 < Dr,2 <....... <...< D be the sorted label distances starting from the i-th sample (i.c., sort (ort(di E (2N])(

---

vi E [2N]. Let Ri.m := Hi dr, = Dim. j E [2.]X | be the number of samples whose distance from the i-th sample equals Di.m. Vi [2N]. [M.)
First, to formalize the concept of ordered feature embedding according to the order in the label space, we introduce property termed as ô-ordered for set of feature embeddings {u))
Definition (ó-ordered feature embeddings). For any 8 < the feature embeddings {Vr are 5-ordered if Vi E [2.]].j. 22]Mi.)
8t,j > 8i,k +
if di,j < di
8
|8(, - s < 8 if di di
1
81. < 8i,k - 8
if da.s > di,t
Definition implies that a set of feature embeddings that is 8-ordered satisfies the following properties: For any j and & such that jr,j = di the difference between $i,j and si,6 is no more than 8; and @ For any j and k such that di, < di the value of šiح exceeds that of 8i,k by at least 3. Note that 6, which indicates that the feature similarity gap between samples with different label distances to the anchor is always larger than that between samples with equal label distances to the anchor.
Next, we demonstrate that the batch of feature embeddings will be ó-ordered as the optimization of CrdC approaches its lower bound. In order to prove this, it is necessary to derive a tight lower bound for LRcC. Let I," == 2N(2N 1) Es Em= fi.m log fs we have:
Theorem (Lower bound of LRNC). L is lower bound of LRNC, i.e., LRNC L".
Theorem 2 (Lower bound tightness).. For any >> O, there exists set of feature embeddings such that CRvc L* +
The above theorems verify the lower bound of Lrce well as its tightness. Given that Lrwe can approach its lower bound L* arbitrarily, we demonstrate that the feature embeddings will be d-ordered when LrcC is sufficiently close to L* for any 0 < 8 < 1.
Theorem 3 (Main theorem). For any < 1, there exist c > O, such that if LRSc < L' + G, then the feature embeddings are 8-ordered.
From Batch to Entire Feature Space. Now we have examined the property of batch of feature embeddings optimized using CRcC. However, what will be the final outcome for the entire feature space when CrvC is optimized? In fact, if any batch of feature embeddings is optimized achieve a low enough loss such that it is d-ordered, the entire feature embedding will also be ô-ordered. This because any triplet (i. k) in the entire feature embeddings certain appear in some batch, thus their feature embeddings (V;, Uk) will satisfy the condition in Definition 1. Nevertheless, to achieve 5-ordered features for the entire feature embeddings, do we need to optimize all batches to achieve a sufficiently low loss? The answer is no. Optimizing every batch is not only unnecessary, but also practically infeasible. In fact, one should consider the training process as a cohesive whole, which is effectively optimizing the expectation of the loss over all possible random batches. Then, the Markov's inequality [18] guarantees that when the expectation of the loss optimized to be sufficiently low, the loss on any batch will be low enough with high probability.
Connections to Final Performance. Suppose we have d-ordered feature embedding. how can it help to boost the final performance of a regression task? In Appendix B, we present an analysis based on Rademacher Complexity [39] to prove that ô-ordered feature embedding results in better generalization bound. To put it intuitively, fitting an ordered feature embedding reduces the complexity of the regressor, which enables better generalization ability from training to testing, and ultimately leads to the final performance gain. Relatedly, we note that the enhanced generalization ability is further empirically verified in Sec. 5. Specifically, if not constrained, the learned feature embeddings could capture spurious or easy-to-learn features that are not generalizable to the real continuous targets. Moreoever, such property also leads to better robustness to data corruptions, better resilience to reduced training data, and better generalization to unseen targets.
5

---

Table 2: Comparison and compatibility to end-to-end regression methods. RNC is compatible to end-to-end regression methods, and consistently improves the performance over all datasets.
AgeDU
TUAB
NP XPIIPaceGaze
SkyFinder
Metrics
MAE!
R-T
MAE!
R2T
Angular
R²T
MAE!
R-T
663
0,828
7.46
0.655
5,97
0.774
2,95
(1860
RNC(L))
(40.8)
0,850 (-0022)
(48,8))
0.697 (4004))
5.27 (4:30)
0,815 (waans)
2,86 (400)
0.869 (+60600)
MSE
6.57
0.828
8.06
0.585
6.02
0.747
3.08
0.851
RNC(MSR)
6.19 (40.80)
0.849 (-0,221)
7.05 (+1/1))
0.692 (4(()7)
5.35 (+887)
0 (40855)
2.86 (-1.22)
(L869 (-08818)
HUBER
6.54
0.828
7.59
0.637
6.34
0.709
2.92
0.860
RNC(HUBER)
6,15 (+0.79)
0,850 (+8,822)
6,99 (+8,0)
1.69% (41L089)
5.15 (+1.39)
0,830 (40.121)
2,86 (40.0)
(L869 (+80809)
DEX [36]
7.29
0.787
8.01
0,537
5.72
0.776
3.58
0.778
RNC(DRX)
6.43
0.836
7.23
0.646
5.14
0.805
2.88
0.865
[1
6:00
0,827
7.91
0.560
5.47
0.799
2,99
O 0.856
RXC(DLDL-V2)
6,32 (+0.28)
0.844 (+8.177)
6.91 (+3,00)
0.697 (+0.137)
5,16 (+0,31)
0.802 (v0.083)
2,85 (40.14)
0.869 (+88813)
[38]
6.40
0,830
7.36
0.646
5,86
0.770
2,92
RNCIOR)
6.34 (-UU))
0.843 (-8L013)
7.01 (+8,355
0.688 (41L42)
5.13 (+8.75)
0,825 (40,885)
2.86 (-1100)
(.867 (-000000
CORN [40]
0.811
0.597
5.88
0.762
3.24
0.819
RNCICORN)
6.44 (40.28)
01838 (-8,827)
7.22 (40,89)
0.663 (-IL864)
5.18 (+8.700
0,820 (40,850)
2,89 (40.25)
(L862 (+80845)
Experiments
Datasets. We perform extensive experiments on five regression datasets that span different tasks and domains, including computer vision, human-computer interaction, and healthcare. Complete descriptions of each dataset and example inputs in Table Appendix C and D.
AgeDB (Age) [32, 44] is dataset for predicting age from face images, containing 16,488 in-the-wil images of celebrities and the corresponding age labels.
IMDB-WIKI (Age) [36, 44] is dataset for age prediction from face images, which contains 523,051 celebrity images and the corresponding age labels. We use this dataset only for the analysis.
TUAB (Brain-Age) [34, 11] aims for brain-age estimation from EEG resting state signals, with
1,385 21-channel EEG signals sampled 200Hz from individuals with age from 0 95.
MPIIFaceGaze (Gaze Direction) [51, 52] contains 213,659 face images collected from 15 partici- pants during natural everyday laptop use. The task is to predict the gaze direction.
SkyFinder (Temperature) [31, 7] contains 35,417 images captured by 44 outdoor webcam cameras for in-the-wild temperature prediction.
Metries. We report two distinct metrics for model evaluation: 1 Prediction error, and @ Coefficient of determination (R²). Prediction error provides practical insight for interpretation, while R² quantifies the difficulty of the task by measuring the extent to which the model outperforms a dummy regressor that always predicts the mean value of the training labels. We report the mean absolute error (MAE) for age, brain-age, and temperature prediction, and the angular error for gaze direction estimation.
Experiment Settings. We employ ResNet-18 [19] as the main backbone for AgeDB, IMDB-WIKI MPIIFaceGaze, and SkyFinder. In addition, we confirm that the results are consistent with other backbones (e.g.. ResNet-50), and report related results in Appendix G. For TUAB, a 24-layer ID ResNet [19] is used as the backbone model to process the EEG signals, with a linear regressor acting as the predictor. We fix data augmentations to be the same for methods, where the details are reported in Appendix D. Negative L2 norm (i.e.. -Ile.. in LRNC, with L, distance as the label distance measure for AgeDB, IMDB-WIKI, SkyFinder, and TUAB, and angular distance the label distance measure for MPIIFaceGaze. We provide complete experimental settings and
(|2)) is used as the feature similarity measure
choices in Appendix F.
5.1 Main Results
Comparison and compatibility to end-to-end regression methods. As explained earlier, our model learns a regression-suitabee representation that can be used by any end-to-end regression methods. Thus, we implement seven generic regression methods: L1, MSE, and HUBER [44] ask the model to directly predict the target and utilize an error-based loss function; DEX [36] and DI.DL-V2 [14] divide the regression range of each label dimension into pre-defined bins and learn the probability

---

Table 3: Comparisons state-of-the-art representation regression learning methods. MAE is used as metric AgeDB, TUAB, and SkyFinder, and Angular Errort is used for MPIIFaceGaze. L1 loss employed as the default regression loss if not specified. RNC surpasses state-of-the-art methods on all datasets.
Method
AgeDB
TUAB
MPIIFaceCaze
SkyFinder
Representation learning methods (Linear Probingl:
SIMCLR [4]
9.59
11.01
9.43
4.70
DINO [3]
10.26
11.62
11.92
5.63
SUPCON [25]
8.13
9.27
3.97
Representation learning methods (Fine-tuning):
SIMCLR [4]
6.57
7.57
5,50
2.93
DINO [3]
6.61
7.58
5.80
2.98
SUPCON [25]
6.55
7.41
5.54
2.95
Regression leurning methods:
In
6.63
7.46
5.97
2.95
LDS+EDS [44]
6.45
L2CS-NET [1]
5.45
LDE [7]
2.92
RANKSIM [17]
6.51
7.33
5.70
2.94
ORDINAL ENTROPY [50]
6.47
7.28
2.94
RNC(L.)
6.14
6.97
5.27
2.86
GAINS
+0.31
+0.31
+0.18
+0.06
distribution over them; OR [33] and CORN [40] design multiple ordered thresholds for each label dimension and learn a binary classifier for each threshold. Further details of these baselines are included in Appendix E. In comparison, then freeze the encoder and train a predictor on top of using each of the baseline methods. The original baseline without the RNC representation is then compared that with RNC. For instance, we denote the end-to-end training of the encoder and predictor using L1 loss as L1, while RNC(L,) denotes training the encoder with Lrec and subsequently training the predictor using L1 loss.
first train the encoder with the proposed CRCC. We
Table 2 summarizes the evaluation results on AgeDB, TUAB, MP MPIIFaceGaze and SkyFinder. Green numbers highlight the performance gains by using RNC representation. The standard deviations of the best results on each dataset are reported Appendix G.2. As the table indicates, RNC consistently achieves the best performance both metrics across all datasets. Moreover, incorporating RNC to learn the representation consistently reduces the prediction error of all baselines by 5.8 %o 9.3%, 11.7% 1.7%, and 7.0% average AgeDB, TUAB, MP MPIIFaceGaze, and SkyFinder, respectively.
Comparison state-of-the-art representation learning methods. We further compare RNC with ofate-of-the-ar representation learning methods, SIMCLR [4], DINO [3], and SUPCON [25], under both linear probing and fine-tuning schemes to evaluate their learned representations for regression tasks. Full details are in Appendix E.2. Note that SIMCLR and DINO do not use label information while SUPCON uses label information for training the encoder. The predictor is trained with L, loss. Table 3 demonstrates that RNC outperforms all other methods across all datasets. Note that some of the representation learning schemes even underperform the vanilla L1 method, which further verifies that the performance gain of RNC stems from our proposed loss rather than the pre-training scheme.
Comparison to state-of-the-art regression learning methods. We also compare RNC with state- of-the-an art regression learning schemes, including methods that perform regularization in the feature space [44, 17, 50] and methods that adopt dataset-specific techniques [1, 7]. We provide the details in Appendix E.3. L1 loss is employed as the default regression loss for all methods if not specified. The results are presented in Table 3, with a dash (-) indicating that the method is not applicable to the dataset. We observe that RNC achieves tate-of-the-arr performance across all datasets.
5.2 Analysis
Robustness to Data Corruptions. Deep neural networks are widely acknowledged to be vulnerable to
data and various forms of corruption, such as noise, blur, and color distortions [21]. To analyze the robustness RNC, we generate corruptions on the AgeDB test set using the comrpttin process from the ImageNet-C benchmark [21], incorporating 19 distinct types of corruption at varying severity levels. We compare (NC(L,) with I, by training both models on the original AgeDB training set, but directly testing them on the corrupted test set across all severity levels. The
7

---

13
20
11.75
RNC(L,)
RNC(L.)
Mean Absolute Enror
11
9:88
9.80
16
15.09
8.60
Mean Abooluts
13.00
8.63
6.65
7.78
12
10.74
11.94
14
9.78
9.30
16k
8k
4k
2k
Corruttion Severity Level
Number Training Samples
Figure 4: RNC is more robust to data corruptions.
Figure 5: RNC is more resilient to reduced train- ing data. We show MAE- as a function of the num- ber of training samples on IMDB-WII MIDB-WIKI.
We show MAB* as a function of corruption severity on AgeDB test set.
Table 4: Transfer learning results. RNC delivers better performance than L1 in all scenarios.
AgeD8 IMD8-MIKI (subsampled, 2k)
IIDB-WIKI (subsampled, 32k)
AgeDB
Linear Probing
Fine- Fine-tuning
Linear Probing
Fine-tuning
Metries
MAE
R-1
MAE
R2
MAB
R2
MAE
Ra1
Ln
12:25
0.496
11.57
0.528
7.36
0.801
6.36
0.848
R 11.12 (+1.3)
0.556 (+0.060)
11.09 (+0.48)
0.546 (+0.018)
7.06 (+0.30)
0.812 (+0.011)
6.13 (+0.23)
0.850 (+0.002)
reported results are averaged over all types of corruptions. Fig. 4 illustrates the results, where the representation learned by RNC is more robust to unforeseen data corruptions, with consistently less performance degradation when corruption severity increases.
Resilience to Reduced Training Data. The availability of massive training datasets has played an important role in the success of modern deep learning. However, in many real-world scenarios, the cost and time involved in labeling large training sets make infeasible to do so. As a result, there need enhance model resilience limited training data. To investigate this, we subsample IMDB-WIKI to generate training sets of varying sizes and compare the performance of RNC(L)) and L1. As Fig. 5 confirms, RNC is robust reduced training data and displays less performance degradation as the number of training samples decreases.
Transfer Learning. We evaluate whether the learned representations by RNC are transferable across datasets. To do so, we first pre-train the feature encoder on a large dataset, and subsequently utilize either linear probing (fixed encoder) fine-tuning to learn predictor
small dataset (which shares the same prediction task). We investigate two scenarios: 1 Transferring from AgeDB which contains 122 samples subsampled IMDB-WIK WIKI of 2K samples, and 2 Transferring from another subsampled IMDB-WIKI of 32K samples to AgeDB, As shown in Table 4, RNC(L) ) exhibits consistent and superior performance compared to L1 in both linear probing and fine-tuning settings for both of the aforementioned scenarios.
Robustness to Spurious Targets. We show that RNC is able to deal with spurious targets that arise in data [46], while existing regres- sion learning methods often fail to learn gen- eralizable features. Specifically, the SkyFinder dataset naturally has a spurious target: different webcams (with distinct locations). Therefore, we show in Fig. the UMAP [30] visualiza- tion of the learned features obtained from both L and RNC, using data from 10 webcams in SkyFing inder. The first column of the figure is colored by the ground-truth target (temperature), while the second column is colored by the 10 different webcams. Our results demonstrate that the representation learned by L1 is clustered by webcam, indicating that it is prone to capturing easy-to-learn features. In contrast, RNC learns the underlying continuous temperature informa- tion even in the presence of strong spurious tar-
Colored by real targets Colored by spurious targets (temperature)
(webcam)
L
RNC
Figure 6: Robustness spurious targets. RNC able to capture underlying continuous targets learn robust representations that generalize.
gets, confirming its ability to learn robust representations that generalize.
8

---

Table 6: Ablation studies for RNC. All experiments are performed on the AgeDB dataset and L is used as the default loss for training the predictor. Default settings used in the main experiments are marked in gray
(a) Number of Positives K
(b) Feature Similarity Measure
(c) Training Scheme
MAE-
R²7
MAE
R²1
MAE
R²7
128
6.46
0.828
cosine
6.51
0,836
linear prohing
6.14
0.850
256
6.43
0.833
negative L1 norm
6,25
0.842
fine-tuning regularization
6.36
0.844
384
6.29
0.845
negative Le norm
6.14
0.850
6.42
0.833
511
6.14
0.850
Generalization Unseen Targets. In practi- cal regression tasks, it is frequently encountered that some targets are unseen during training. To investigate zero-shot generalization to unseen targets, following [44], we curate two subsets of IMDB-WIKI that contain unseen targets during training, while keeping the test set uniformly dis- tributed across the target range. Table 5 shows the label distributions, where regions of unseen targets are marked with pink shading and those of seen targets are marked with blue shading.
Table 5:
Zero-shot generalization to unseen targets. We create two IMDB-WIK. training sets with missing targets, and keep test sets uniformly distributed across the target range. MAE* used the metric.
Label Distribution
Method
All
Secn
Unseen
L
12:53
10.82
18.40
RNC(L))
11.69
10.46
15,92
(+0.84)
(40.36)
(+2.48)
In
11.94
10.43
14,98
RNC(L,)
10.88
9.78
13.08
(+1.06)
(+0.64)
(+1.90)
The first training set has bi-modal Gaussian distribution, while the second one exhibits a tri-modal Gaussian distribution over the target space. The results confirm that RNC(L,) outperforms L1 by a larger margin the unseen targets without sacrificing the performance on the seen targets.
5.3 Ablation Studies
Ablation on Number of Positives. Recall that CRCC. all samples in the batch will be treated as the positive for each anchor. Here, we conduct an ablation study only considering the first K closest samples to the anchor as positive. Table 6a shows the results on AgeDB for different K, where K = 511 represents the scenario where all samples are considered as positive (default batch size N = 256). The experiments reveal that larger values of K lead to better performance, which aligns with the intuition behind the design of LRvc: each contrastive term ensures that group of orders related to the positive sample is maintained. Specifically, it guarantees that all samples that have larger label distance from the anchor than the positive sample are farther from the anchor than the positive sample in the feature space. Only when all samples are treated as positive and their corresponding groups of orders preserved can the order in the feature space be fully guaranteed.
A Ablation on Similarity Measure. Table 6b shows the performance of using different feature similarity measures sim((..). Compared to cosine similarity, both the negative L1 norm and L2 norm produce significantly better results. This improvement can attributed to the fact that cosine similarity only captures the directions of feature vectors, while the negative L1 or L2 norm takes both the direction and magnitude of the vectors into account. This finding highlights the potential differences between representation learning for classification and regression tasks, where the standard practice for classification is to use cosine similarity [4, 25], while our findings suggest the superiority of L2 and L2 norm for regression-basee representation learning.
Ablation on Training Scheme. There are typically three schemes to train the encoder: Linear probing, which first trains the feature encoder using the representation learning loss, then fixes the encoder and trains a linear regressor on top using a regression loss; Fine-tuning. which first trains the feature encoder using the representation learning loss, then fine-tunes the entire model using a regression loss: and Regularization, which trains the entire model while jointly optimizing the representation learning & the regression loss. Table 6c shows the results for the three schemes using CreC as the representation learning loss and L1 as the regression loss. All three schemes can improve performance over using L1 loss alone. Further, unlike classification problems where fine-tuning often delivers the best performance, freezing the feature encoder yields the best outcome in regression. This is because, in the case of regression, back- sack-propagating the L₁ loss to the representation can disrupt the order in the embedding space learned by CRVC, leading to inferior performance.

---

5.4 Further Discussions
Does RNC rely on data augmentation for superior performance (Appendix G.3)? Table 10 confirms that RNC surpasses the baseline, with or without data augmentation. In fact, unlike typical (unsupervised) contrastive learning techniques that rely on data augmentation for distinguishing the augmented views, data augmentation is not essential for RNC. This is because RNC contrasts samples according to label distance rather than the identity. The role of data augmentation for RNC similar to its role in typical regression learning, which is to enhance model generalization.
Does the benefit of RNC come from the two-stage training scheme (Appendix G.4)? We confirm in Table 11 that training predictor on top of the representation learned by the competing regression baselines does not improve performance. In fact, it can even be detrimental to performance. This finding validates that the benefit of RNC due to the proposed LrGC and not the training scheme. The generic regression losses are not designed for representation learning and are computed based on the final model predictions, which fails to guarantee the final learned representation.
Computational cost of RNC (Appendix G.5)? We verify in Table 12 that the training time of RNC is comparable to standard contrastive learning methods (e.g., SUPCON [25]), indicating that RNC offers similar training efficiency, while achieving notably better performance for regression tasks.
6 Broader Impacts and Limitations
Broader Impacts. We introduce novel framework designed to enhance the performance of generic deep regression learning. We believe this will significantly benefit regression tasks across various real-world applications. Nonetheless, several potential risks warrant discussion. First, when the framework is employed to regress sensitive personal attrihutes such as intellectual capabilities, health status, or financial standing from human data, there's a danger it might reinforce or even introduce new forms of bias. Utilizing the method in these contexts could inadvertently justiffy discrimination or the negative targeting of specific groups. Second, in contrast to handcrafted features which are grounded in physical interpretations, the feature representations our framework learns can be opaque. This makes it difficult to understand and rationalize the model's predictions, particularly when trying to determine if any biases exist. Third, when our method is trained on datasets that do not have a balanced representation of minority groups, there's 's no assurance of its performance on these groups being reliable. It is essential to recognize that these ethical concerns apply to deep regression models at large, not solely our method. However, the continuous nature of our representation which facilitates interpolation and extrapolation might inadvertently make it more tempting to justify such unethical applications. Anyone seeking to implement or use our proposed method should be mindful of these concerns. Both our specific method and deep regression models, in general, should be used cautiously to avoid situations where their deployment might contribute to unethical outcomes or interpretations.
Limitations. Our proposed method presents some limitations. Firstly, the technique cannot discern spurious or incidental correlations between the input and the target within the dataset. As outlined in the Broader Impact section, this could result in incorrect conclusions potentially promoting discrimination or unjust treatment when utilized to deduce personal attributes. Future research should delve deeper into the ethical dimensions of this issue and explore strategies to ensure ethical regression learning. A second limitation is that our evaluation primarily focuses on general regression accuracy metrics (e.g., MAE) without considering potential disparities when evaluating specific subgroups (e.g., minority groups). Given that regression model's 's performance can vary across demographic segments, subgroup analysis is an avenue that warrants exploration in subsequent studies. Lastly, our approach learns continuous representations by contrasting samples against one another based on their ranking in the target space, necessitating label information. To adapt it for representation learning with unlabeled data, our framework will need some modifications, which we reserve for future work.
Conclusion
(RNC). framework that learns continuous representations for regression by ranking samples according to their labels and then contrasting them against each other based on their relative rankings. Extensive experiments on different datasets over various real-world tasks verify the superior performance of RNC, highlighting its intriguing properties such as better data efficiency, robustness to corruptions and spurious targets, and generalization unseen targets.
We present
10

---

Acknowledgements. We are grateful to Hao He for his invaluable assistance with the theoretical analysis in the paper. We thank the members of the NETMIT group for their constructive feedback on the draft of this paper. We extend our appreciation to the anonymous reviewers for their insightful comments and suggestions that greatly helped in improving the quality of the paper. Lastly, we acknowledge the generous support from the GIST-MIT program, which funded this project.
References
[1] Ahmed A Abdelrahman, Thorsten Hempel, Aly Khalifa, and Ayoub Al-Hamadi. L2ccsnet: Fine-grained gaze estimation in unconstrained environments. arXiv preprint 1rXiv:2203.03339 2022.
[2] Wenzhi Cao, Vahid Mirjalili, and Sebastian Raschka. Rank consistent ordinal regression for neural networks with application to age estimation. Pattern Recognition Letters, 140:325-331, 2020.
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[4] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning. pages 1597-1607. PMLR, 2020.
[5] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422
2020.
[6] Weiwei Cheng, Eyke Hüllermeier, and Krzysztof J Dembezynski. Label ranking methods based on the plackett-luce model. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 215-222, 2010.
[7] Wei-Ta Chu, Kai-Chia Ho, and Ali Borji. Visual weather temperature prediction. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 234-241. IEEE, 2018
[8] Ching-Yao Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. Advances in neural information processing systems, 33:8765- 8775, 2020.
[9] Benoit Dufumier, Pietro Gori, Julie Victor, Antoine Grigis, and Edouard Duchesnay. Conditional aligmment and uniformity for contrastive learning with continuous proxy labels. arXiv preprint 1rX1v:2111.05643, 2021.
[10] Benoit Dufumier, Pietro Gori, Julie Victor, Antoine Grigis, Michele Wessa, Paolo Brambilla, Pauline Favre, Mircea Polosan, Colm Mcdonald, Camille Marie Piguet, et al. Contrastive learning with continuous proxy meta-data for 3d mri classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 58 68. Springer, 2021.
[11] Denis A Engemann, Apolline Mellot, Richard Höchenberger, Hubert Banville, David Sabbagh, Lukas Gemein, Tonio Ball, and Alexandre Gramfort. A reusable benchmark of brain-age prediction from m/eeg resting-state signals. Neurolmage, 199521, 2022.
[12] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2002-2011, 2018.
[13] Bin-Bin Gao, Chao Xing, Chen-Wei Xie, Jianxin Wu, and Xin Geng. Deep label distribution learning with label ambiguity. IEEE Transactions on Image Processing, (M6))2825-2836 2017.
[14] Bin-Bin Gao, Hong- Zhou, Jianxin Wu, and Xin Geng. Age estimation using expectation of label distribution learning. In IJCAI, pages 712-718, 2018.
[15] Amir Globerson, Gal Chechik, Fernando Pereira, and Naftali Tishby. Euclidean embedding of co-occurrence data. In NeurIPS, 2004.
[16] Jacob Goldberger, Geoffrey E Hinton, Sam Roweis, and Russ R Salakhutdinov. Neighbourhood components analysis. In NewrIPS, 2004.

---

[17] Yu Gong, Greg Mori, and Frederick Tung. Ranksim: Ranking similarity regularization for deep imbalanced regression. In Proceedings of the 39th International Conference on Machine Learning, pages 7634-7649. PMLR, 2022.
[18]
Geoffrey Grimmett and David Stirzaker. Probability and random processes. Oxford university press, 2020.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[20] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2020.
[21] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
[22] Peter Huber. Robust estimation of a location parameter. In Breakthroughs in statistics, pages 492-518. Springer, 1992.
[23] Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference on Learning Representations, 2020.
[24] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/)):81111111119110)99 1938.
[25] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Samaa, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33: 133116661-1673 2020.
[26] Shikun Li, Xiaobo Xia, Shiming Ge, and Tongliang Liu. Selective-superiieed contrastive learning with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 316-325, 2022.
[27] Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio S Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1928-6928, 2022.
[28] Wanhua Li, Xiaoke Huang, Jiwen Lu, Jianjiang Feng, and Jie Zhou. Learning probabilistic ordi- nal embeddings for uncertainty-awant aware regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13995- 2021.
[29] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint a6Xiv: 03983, 2016.
[30] Leland Mclnnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 0100//02200222 2018.
[31] Radu P Mihail, Scott Workman, Zach Bessinger, and Nathan Jacobs. Sky segmentation in the wild: An empirical study. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1-6. IEEE, 2016.
[32] Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop, volume 2, page 5, 2017.
[33] Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. Ordinal regression with multiple output enn for age estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4920-4928. 2016.
[34] Iyad Obeid and Joseph Picone. The temple university hospital eeg data corpus. Frontiers in neuroscience, 10:196, 2016.
[35] Hongyu Pan, Hu Han, Shiguang Shan, and Xilin Chen. Mean-variance loss for deep age estimation from a face. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5285-5294, 2018
12

---

[36] Rasmus Rothe, Radu Timofte. and Lue Van Gool. Dex: Deep expectation of apparent age from a single image. In Proceedings of the IEEE international conference on computer vision workshops, pages 10-15, 2015.
[37] Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioral and neural analysis. arXiv preprint arXiv:2204.00673 2022.
[38] Fabian Schrumpf, Patrick Frenzel. Christoph Aust, Georg Osterhoff, and Mirco Fuchs. Assess- ment of non-invasive blood pressure prediction from ppg and rppg signals using deep learning. Sensors, 21(18):6022, 2021.
[39] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
[40] Xintong Shi, Wenzhi Cao, and Sebastian Raschka. Deep neural networks for rank-consistent ordinal regression based on conditional probabilities, 2021.
[41] Charles Spearman. The proof and measurement of association between two things. The American journal of psychology, 100(3/4):441-771, 1987.
[42] Yaoming Wang. Yangzhou Jiang, Jin Li, Binghing Ni, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Teng Li. Contrastive regression for domain adaptation on gaze estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19376-19385, 2022.
[43] Fen Xia, Tie- Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwise approach to learning to rank: theory and algorithm. In Proceedings of the 25th international conference on Machine learning, pages 1 1192-1199 199, 2008
[44] Yuzhe Yang, Kaiwen Zha, Yingcong Chen, Hao Wang, and Dina Katabi. Delving into deep imbalanced regression. In International Conference on Machine Learning, pages 11842-11851. PMLR, 2021.
[45] Yuzhe Yang. Hao Wang, and Dina Katabi. On multi-domain long-tailed recognition, imbalanced domain generalization and beyond. In European Conference on Computer Vision (ECCV), 2022.
[46] Yuzhe Yang. Xin Liu, Jiang Wu, Silviu Borac, Dina Katabi, Ming-Z Poh, and Daniel McDuff. Simper: Simple sulf-super learning of periodic targets. In International Conference on Learning Representations, 2023.
[47] Huaxiu Yao, Yiping Wang, Linjun Zhang, James Y Zou, and Chelsea Finn. C-mixup: Improving generalization in regression. Advances in Neural Information Processing Systems, 35:3361- 3376, 2022.
[48] Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. Modeling discriminative representations for out-of -o-aann detection with supervised contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I Ith International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 870-878, 2021.
[49] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 2017.
[50] Shihao Zhang. Linlin Yang, Michael Bi Mi, Xiaoxu Zheng, and Angela Yao. Improving deep regression with ordinal entropy. arXiv preprint arXi::2301.08115, 2023.
[51] Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. Mpiigaze: Real-world dataset and deep appearancee-basd gaze estimation. IEEE transactions on paltern analysis and machine intelligence, 1121): 2017.
[52] Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. It's written all over your face: Full-face appearance-based gaze estimation. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on, pages 2299-2308. IEEE, 2017.
13

---

A Proofs
A.1 Proof of Theorem
Recall that in Eqn. (3) we defined CRcC == 2N
2N
2
2N
Σ
log
2o exp(ssx)) where
exp(*1,7)
2N
1=1
j=
iz
Sig = (th h: + i, di di.j.. We rewrite it as
2N
exp(si.)
CRCC 22 1) JCUNNR(
Σ
log
exp(81,E)
RE(2N]M 2,
ΣΣ
M,
Σ
log
exp(s.3)
2N(2N 1) 1m JERNM(() di De.m
Σ
exp(5iA)
kE( d 2..
ΣΣ
M.
E
log
exp( (8..))
2N(2N -1)
Σ
exp(si,k)
(E), Di.m
KE(NNTM d, D.
Σ
exp($i,e 81.5)
M.
ΣΣ
Σ
log 1+ (c)2N))))).
d.A.Dem Pam
+ 2N 1) 1 m=
exp(si,k si.))
JERNI di.=Da.m
kEj2NI di.
ΣΣ
M.
E
log
exp(81.5)
2N(2N 1)
exp(s,,))
m==
JENNNNEE Di.m
KC) d, D.m
(4)
Vi [2N]. m E [M.), from Jensen's Inequality we have
exp(sia)
Σ
log Σ
exp(si,)
jE(2N]((), diy Di.m
kE(2N]((), d,, Di.
(5)
cxp(si.j)
2 -14mm log
Σ
Nmm log mim
nimm
Σ
exp(81,R)
JE(2N]M()) DDmm kE|2NTU()) di.==D..
Thus, by plugging Eqn. (5) into Eqn. (4), we have
2N Mr
1
LASC
Σ
nim log f4,m D"
(6)
2N(2N - 1)
L= m=1
A.2 Proof of Theorem
We will show Ve 0, there of feature embeddings where
$ Si,k + if dr,j < di $1.j = si,k di,; = di
and 7 log
min mm MM] Agmmm Vi E [2N], [2N])((), such that Lave L t
2N
For such of feature embeddings, Vi [2N], m [M., E ti [2N] di.j Di,m),
exp($15)
loo
llog ni.m
Σ
exp(s1.))
AE d, Di.m
14

---

since 81, Si, for all & such that di = Dim = di and
exp(8i, - 82.5)
1 1 RERNNT( di >> D
log
exp(86,1 8t,5)
kE(2N])((), A.n=Dim
(8)
2N
2N 2N eex((->) S
< log
ni,m
Ri.m
since 8i,k 8i, < -y for all l such that di, > Dim di,5 and 8i, 8i.j = 0 for all & such that dı,t Di.m- di.j-
From Eqn. (4) we have
2N
exp( 8
LRNC 2N(2N
Σ
Σ
log
exp($i,k)
El m=1
JE(2NT 4., D
kE(2M]M()) de D..m
Σ
exp(s - 8i.)
2N M
ΣΣ
Σ
log
14 KE/2N) d. Σ
2N(2N 11
exp(8,, 81.)
E m
JCPNTM) 4., Di.m
NE(2N],(0)..... dr,
(9)
By plugging Eqn. (7) and Eqn. (8) into Eqn. (9) we have
2N Mr
LRNC < 2N(2N 1)
nim log n4,m + 1, +
(10)
1= m=1
A.3 Proof of Theorem 3
We will show v0 8 < there is
1 exp(0)
-8) > 0,
2N(22 - min min c=
log ni,m exp( 8)
222g
[e())]]me[MM]]]
2
such that when LRNN < L + the feature embeddings are 5-ordered.
We first show that (i.j si < 8 if dij di,s, vi E [2N], j, [2N] when LRNN < 1' +
From Eqn. (4) have
M, Σ
exp(sij)
LRCC 2N(2N 1)
Σ
log
Σ
exp(six))
(11)
de
Di.m
kE(2N]((), dr.
Let Pi.m
arg min JE(2NMM() =D,=
86.
arg JE(NNIV())
Siji Si, := Sip Ti.m
di,==D..m Vi [2N], m E [M.]., by splitting out the maximum term and the minimum term
Si.g..m
Si.pr.m
we have
1
M.
exp(Si.m)
LRSC
log
2N (2N 1)
Σ
exp(st,k)
KE D..m
exp
Σ
8i,
exp(Sim + Mmm)
(E(2N])(4PP... Ds.m
log
+ log
Σ
exp(si.k)
2
kE(DN]N(), di..D..m
Σ
exp(si,k)
kEj2N]VE).d di De...
(12)
Let 0zm Tlsmm
Σ (.m
exp(8i.) Sima) we have
exp(Gi.m)
-log Σ
log(1 + exp(n)..) + (niam 2)0mm
(13)
exp(si,t)
kE(2N]M(1), d DD
15

---

and
exp(Si.m + m)mm)
log
Σ
exp(ss,k) =
log(1 + exp(s..) + (nim 2) )- 7m
(14)
kε)aNT((s), di.=DD,m
Then, from Jensen's inequality, we know
exp
Σ JE(2N]M (i.pr.m P.ds, : m.
81.
S
Σ
exp(s15)
(15)
thus
log
"1 (Nam 2) log(1
+ cxp(lim)
(14m 2) (n4,m 2) (0.))
eap(ana))
(16)
By plugging Eqn. (13), Eqn. (14) and Eqn. (16) into Eqn. (12), we have
2N M.
CRNC 2N(2N - 1)
Σ
(ni, log(1 exp(Ni.m) + (na, 2) Ta (Nam. 22) (og(8i,)))
(17)
Let (0) = ni,m log(l+ (n,,m-m))))) --.m (nim 2) log(O). From derivative analysis
we know (0) decreases monotonically when A E 1-exp( and increases monotonically
when [+exp(m..))
exp(m).)) )
thus
2
h(0)= 2 h + exp(r.m) =
+ exp(Timm) Vm
nimm log Ni,m + 2 log
(18)
2
By plugging Eqn. (18) into Eqn. (17), we have
1
2N
exp(Ni.m)
LRNC >
nem log ni,m nig
+loo
T/.m
2N - 1)
2
(19)
2N M.
1 + e (Tk.)
= -t + 2N(2N 1)
2
7/
m=1
Then, since fi.m 2 0, we have 2og
1+exp(ve...
Bim 2 Thus, E [2N], m E [Mi].
2
1 + exp((imm) 1i
LRCC > L + 2N(2N 1)
2log
(20)
2
If LRNC < L + 22
2 22 1+exp(6)
-6)
6). then
+ exp(Ikm)
mi 2log + exp(o)
2 2og
(21)
2
Since y(x) = 2 log [+epp(x) x increases monotonically when x > 0, we have 7im < 6. Hence, Vi [2.]], j. [2N]] dr,j d₁ Ds |st.j st,b| I 7. 8.
Next, we show $u,j > si,k + if dr,, < di when LRNe < L + E.
From Eqn. (4) we have
LRNC = 2N(2N 1)
2N M
Σ
Σ
log Σ
exp(
exp(si.)
JE(N]](() D
AE(2N] de Di.m
Σ
exp(si,k 8i.)
2N
ΣΣ Σ
log 14
kE kE(2N]M()), de D.
2N - 1)
Σ
exp(31, 8tg)
JCPNNMA)) d. D..m
kE(2N]N(1), di.=D
(22)
16

---

and combining it with Eqn. (5) we have
Σ
exp(six 81,5)
2N M
Lave 2 L' 2N(2N m-
E
log 1+ MCRNNT d,, DM,.
Σ kEj2N] d,,.=D,m
exp(51.4 )63)
jE M d.
exp( 81,5)
> L' 2N(2N 1)
log 1+
exp(a1.) 815)
(E(AN]V(), dua=dia
(23)
WiE E [2N], j E
{k (2N]M des < d.,
When PrSE < T. e, we already have |8i, st,ع < 6, Vdl,1 di.j. which derives 8i,1 Sijj < d
and thus exp($t. - -.14) ) < exp(o). By putting this into Eqn. (22), we have Vi E [2.], j E
[2N]][i), E [k [2N]EE] dag < dí,t..
2N -1 log 1+ exp(si.k - 81.)) mirr, exp(8)
LRCC > L'
(24)
where ri, E [Mi is the index such that Di.ras = di.j-
Further, given Frve < L +<< 2N(2N-) log 1+ mirs, expló+3] we have
log 1+ exp(si, (6.5 < log 1+ Bir.. exp(6 + z))
1
(25)
ni,r, exp(o)
which derives 8t. > 8i,k + 3. Vi [2N], j [2N]] E (k E [2N]](i) des < dex).
Finally, Vi E [2N]], j,k E [2N] (i), 8ú, < 88,k if da.j > de directly follows from 8i.j > $t,k + 8 if dis < di...
B Additional Theoretical Analysis
In this section, we present an analysis based on Rademacher Complexity [39] to substantiate that 5-ordered feature embedding results in a better generalization bound.
regression learning task can be formulated as finding hypothesis h to predict the target y from the input æ. Suppose there are m data points in the training set S = {(TK, Hi) Let Hi be the class of all possible functions from the input space the target space.
d-ordered feature embedding is guaranteed with an encoder f mapping ag to Uk the set of candidate hypotheses can be reduced to all "5 "6-monotonic" functions h(x) g(f(x)), which satisfy d(g(u;), g(;))) < d(g(v;). (vx)) for $t > 8t.k + 3. d(g(v;). sc,l < 8, and d(g(v;), g(v,)) > d(g(v:), g(v))) for sijj < $t,K 4 for any j and k. We denote the class of all "ô-monotonic" functions by H2. Note that both Hi and H2 contain the optimal hypothesis h", which satisfies Vz, y, h²" hxxx)= =
= d(g(v:), g(v))) for |ss,]
(((mm Vn h)) h Hi} be the loss set for each hypothesis in H4 with respect to the training set S, where is the loss function. Let c, be the upper bound of [[([z, y); h))] for all x, y and h E H4, We introduce the Rademacher Complexity [39] of A,, denoted as R(A.). Then, the generalization bound based on Rademacher Complexity says that with high probability (at least e), the gap between the empirical risk (i.e.,
Further, for hypothesis set H4, let A, ((1((11,11)
training error) and the expected risk (i.e., test error) upper bounded by 2R(A,) + 4c;
2ln(4/c)
m7
Since H2 C Hi, we have A C Ai and 02 S ct. and from the monotonicity of Rademacher Complexity we have R(A2)= < R(Ar). Hence, with d-ordered feature embedding, the upper bound on the gap between the training error and the test error will be reduced, which leads to better regression performance.
17

---

Table Visualizations of original and augmented data samples all datasets.
Dataset
Original
Augmented
AgeDB
TUAB
MPIIFaceCaze
SkyFinder
C
Dataset Details
Five real-world datasets are used in the experiments:
AgeDB [32, 44] is a dataset for predicting age from face images. It contains 16,488 mn-the-will images of celebrities and the corresponding age labels. The age range is between 0 and 101. It is split into a 12,208-image training set, a 2140-image validation set, and 2140-image test set.
TUAB [34, 11] a dataset for brain-age estimation from EEG resting-state signals. The dataset comes from EEG exams at the Temple University Hospital in Philadelphia. Following Engemann al. [11], we use only the oon-pathological subjects, so that we may consider their chronological age their brain-age label. The dataset includes 1,385 21-channel EEG signals sampled at 200Hz from individuals whose age ranges from to 95. is split into a 1,246-subject training set and 139-subject test set.
MPIIFaceGaze (51,52) is dataset for gaze direction estimation from face images. It contains 213,659 face images collected from 15 participants during their natural everyday laptop use. We subsample and split it into 33,000-immage training set, 6.000-image validation set, and a 6,000- image test set with no overlapping participants. The gaze direction is described as a 2-dimensional vector with the pitch angle in the first dimension and the yaw angle in the second dimension. The range of the pitch angle -40" to and the range of the yaw angle is -450 45".
SkyFinder [31, 7] a dataset temperature prediction from outdoor webcam images. contains 35,417 images captured by 44 cameras around I on each day under a wide range of weather and illumination conditions. The temperature range -20 49 CC. It split into 28,373 -mage training set, 3,522-image validation set, and 3.522-image test set.
MIDB-WIKI [36, 44] is a large dataset for predicting age from face images, which contains 523,05 celebrity images and the corresponding age labels. The age range between 0 and 186 (some images are mislabeled). We use this dataset to test our method's resilience to reduced training data, performance on transfer learning, and ability to generalize to unseen targets. We subsample the dataset to create variable size training set, and keep the validation set and test set unchanged with 11,022 images in each.
18

---

D Details of Data Augmentation
Table shows examples of original and augmented data samples on each dataset. The data augmenta- tions used on each dataset are listed below:
For AgeDB and SkyFinder, random crop and resize (with random horizontal flip), color distortions used as data augmentation;
For TUAB, random crop is used as data augmentation;
For MPI MPIIFaceGaze, random crop and resize (without random horizontal flip), color distortions are used as data augmentation.
E Details of Competing Methods
E.1 End-to-End Regression Methods
We compared with seven end-to-end regression methods:
L1. MSE and HUBER have the model directly predict the target value and train the model with an error-based loss function, where L1 uses the mean absolute error, MSE uses the mean squared error and HUBER uses an MSE term when the error is below threshold and an L1 term otherwise.
DEX [36] and DLDL-V2 [14] divide the regression range of each label dimension into several bins and learn the probability distribution over the bins. DEX [36] optimizes a cross-entropy loss between the predicted distribution and the one-hot ground-truth labels, while DLDL-V2 [14] jointly loss between the predicted distribution and a normal distribution centered at the ground-truth value, as well as L1 loss between the expectation of the predicted distribution and the ground-truth value. During inference, they output the expectation of the predicted distribution for each label dimension.
optimizes
OR [33] and CORN [40] design multiple ordered thresholds for each label dimension, and learn binary classifier for each threshold. OR [33] optimizes a binary cross-entropy loss for each binary classifier to learn whether the target value is larger than each threshold, while CORN [40] learns whether the target value is larger than each threshold conditioning on it is larger than the previous threshold. During inference, they aggregate all binary classification results to produce the final results.
For the classification based baselines, the regression range is divided into small bins, and each bin is considered as class. Details for each dataset are as follows:
For AgeDB, the age range is 0 ~ 101 and the bin size is set to 1;
For TUAB, the brain-age range is 0 95 and the bin size is set to 1;
For MPIIFaceGaze, the target range is -40 10 (") for the pitch angle and -45 45 (?) for the yaw angle, and the bin size is set to 0.5 for the pitch angle and is set to for the yaw angle;
For SkyFinder, the temperature range is -2 ~ 49 (*C) and the bin size is set to 1.
E.2 State-of-the-art Representation Learning Methods
We compared with three state-of-the-art representation learning methods:
SIMCLR [4] is a contrastive learning method that learns representations by aligning positive pairs and repulsing negative pairs. Positive pairs are defined as different augmented views from the same data input, while negative pairs are defined as augmented views from different data inputs.
DINO [3] is a self-supervised representation learning method using relf-distillation. It passes two different augmented views from the same data input to both the student and the teacher networks and maximizes the similarity between the output features of the student network and those of the teacher network. The gradients are propagated only through the student network and the teacher parameters updated with an exponential moving average of the student parameters.
19

---

SUPCON [25] extends SIMCLR [4] to the fully-supervised setting, where positive pairs are defined augmented data samples from the same class and negative pairs are defined as augmented data samples from different classes. To adapt SupCon to the regression task, we follow the standard routine of classification-base and regard each bin as class.
regression methods to divide the regression range into small bins
E.3 State-of-the-aar Regression Learning Methods
We also compared with five state-of-the-art regression learning methods:
LDS+FDS [44] is the state-of oftte-of-the-art method on the AgeDB dataset. It addresses the data imbalance issue by performing distribution smoothing for both labels and features.
L2CS-NET [1] is the state-of-the-art method the MPIIFaceGaze dataset. It regresses each gaze angle separately and applies both a cross-entropy loss and an MSE loss on the predictions.
LDE [7] is the state-of-the-art method on the SkyFinder dataset. It converts temperature estima- tion to a classification task, and the class label is encoded by a Gaussian distribution centered at the ground-truth label.
RANKSIM [17] is state-of-the-art regression method that proposes regularization loss match the sorted list of a given sample's neighbors in the feature space with the sorted list of its neighbors the label space.
ORDINAL ENTROPY [50] is state-of-the- art regression method that proposes regularization loss to encourage higher-entropy feature spaces while maintaining ordinal relationships
Details of Experiment Settings
All experiments are trained using 8 NVIDIA TITAN RTX GPUs. We use the SGD optimizer and cosine learning rate annealing [29] for training. The batch size is set to 256. For one-stage methods and encoder training of two-stage methods, we select the best learning rates and weight decays for each dataset by grid search, with a grid of learning rates from {0.01, 0.05, 0.1, 0.2, 0.5. 1.0) and weight decays from (10-6, 10-5, 10 10-2, 10-$), J. For the predictor training of two-stage methods, we adopt the same search setting as above except for adding no weight decay to the search choices of weight decays. For temperature parameter т, we search from {0.1, 0.2, 0.5, 1.0, 2.0, 5.0} and select
the best, which 2.0. We train all one-stage methods and the encoder of two-stage methods for 400 epochs, and the linear regressor of two-stage methods for 100 epochs.
G Additional Experiments and Analyses
G.1 Impact of Model Architectures
In the main paper, we use 18 the default encoder backbone for three visual datasets (AgeDB, MPIIFaceGaze and SkyFinder) In this section, we study the impact of backbone architectures on the experiment results. As Table 8 reports, the results of using ResNet-50 as the encoder backbone are consistent with the results using ResNet-18 in Table 2, indicating that our method is compatible with different model architectures.
Table Evaluation results using R5sNet-50 as the encoder backbone for visual datasets. The results are consistent with the results using ResNet- 18 as the encoder backhone.
AgeDB
MP IPIIFaceGaze
SkyFinder
Metries
MAE
R21
Angular
R²7
MAE.
R²7
L1
6.49
0.830
5.74
0.748
2.88
0.863
RNC(LI)
6.10
0.851
5.16
0.819
2.78
0.877
(+0.39)
(+0.021)
(+0.58)
(+0.071)
(+0.10)
(+0.014)
20

---

G.2 Standard Deviations of Results
In this section, we study the standard deviations of the best results on each dataset with different random seeds. Table 9 shows their average prediction errors and standard deviations. These results are aligned with the results we reported in the main paper.
Table 9: Average prediction errors and standard deviations of the best results on each dataset.
AgeDB: RNC(L,)
TUAE: RxC(DLDL-v2)
MPIIFaceCaze: RNC(OR)
SkyFinder: Rx RxC(DLDL-v2)
6.19 = 0.08
7.00 =
5.24 0.13
2.87 +
G.3
Impact of Data Augmentation
In this section, we study the impact of data augmentation on RNC. The following ablations are considered:
(NC(Li)(wthonn augmentation): Data augmentation is removed and CRRC computed over N feature embeddings.
RNC(Li)ooneviie) augmentation): Data augmentation is only performed once for each data point and Crcc computed over N feature embeddings.
RNC(L; (twoo-iew augmentation); Data augmentation is performed twice to create two-view batch and Lace computed over feature embeddings.
Liwithouut augmentation): Data augmentation removed and L1 is computed over samples.
(iwwthh augmentation): Data augmentation is performed for each datapoint and Lı is computed over N samples.
Table 10 reports the results on each dataset, which demonstrate that removing data augmentation will result in a decrease in performance for both L1 and RNC(L1), and method outperforms the baseline with or without data augmentation.
Here we further discuss the different roles played by data augmentation in (unsupervised) contrastive learning methods, end-to-end regression learning methods, and our method respectively:
For (unsupervised) contrastive learning methods, data augmentation is essential for creating the pretext task, which is to distinguish whether the augmented views belong to the same identity or not. Therefore, removing the data augmentation will result in complete collapse of the model.
For end-to-end regression learning methods, data augmentation helps the model generalize better and become more robust unseen variations. However, without augmentation, the model still perform reasonably well. The improvement brought by the augmentation is often correlated to how much the data augmentation can compensate for the gap between training data and testing data.
For RNC, data augmentation is also not necessary since loss contrasts samples according to label distance rather than the identity. Thus, creating augmented views is not crucial to our method. The role of data augmentation for method is similar to its role in regression learning methods, namely, improving the generalization ability. The experiment results also confirm this: removing data augmentation will lead to similar drop in performance for both RNC(L,) and L1-
Table 10: Impact of data augmentation. Data augmentation is not essential for RNC's superior performance.
Method
Augmentation
AgeDB
TUAB
MPIIFaceCaze
SkyFinder
LA
9.53
10.24
6.93
3.14
RNC(L.)
8.96
9.88
6.31
2.97
L
6.63
7.46
5.97
2.95
RNC(L.)
view
6.40
7.29
5.46
2.89
RNC(L,)
Two-view
6.14
6.97
5.27
2.86
21

---

G.4 Two-Stage Training Scheme for End-to-End Regression Methods
RNC employs a two-stage training scheme, where the encoder is trained with Cree in the first stage and predictor is trained using regression loss on top of the encoder in the second stage. In Table 11, we also train a predictor using L1 loss on top of the encoder learned by the competing end-to-end regression methods and compare with the performance of RNC((,)) ) on the AgeDB dataset. The results show that the two-stage training scheme does increase the performance of end-to-end regression methods. In fact, it can even be detrimental to performance. This is because these methods are not designed for representation learning, and their loss functions are calculated w.r.t. the final model predictions. As result, there is no guarantee for the representation learned by these methods. This finding validates that the benefit of RNC due to the proposed CrıC and not the training scheme.
Table 11:
Two-stage training results for end-to-end regression methods on AgeDB. The two-stage training scheme does not increase or even harms the performance of end-to-end regression methods.
Method
End-i to End
Two-Stage
L1
6.63
6.68
MSE
6.57
6.57
HUBER
6.54
6.63
DEX [36]
7.29
7.42
DLDL-V2 [14]
6.60
7.28
OR [33]
6.40
6.72
CORN [40]
6.72
RNC(L1)
6.14
G.5 Training Efficiency
We compute the average wall-clock running time (in seconds) per training epoch on 8 NVIDIA TITAN RTX GPUs for RNC and compare it with SUPCON [25] all four datasets, as shown in Table 12. Results indicate that the training efficiency RNC is comparable to SUPCON.
Table 12:
Average wall-clock running time (in seconds) per training epoch RNC and SUPCON [25] on all datasets. The training efficiency of RNC is comparable to SUPCON.
Method
AgeDB
TUAB
MP MPIIFaceGaze
SkyPinder
SUPCON [25]
23.1
25.3
69.1
55.6
RNC
26.2
27.3
75.4
61.8
RATIO
1.13
1.08
1.09
1.11
22


Reviews:
[{"review_id": "wmqJLxhsL7", "reviewer": "Reviewer_6JYz", "summary": "In this paper, the authors proposed a novel framework that learns continuous representations for regression problems, by contrasting samples against each other based on the rankings induced by the target values. The proposed method is evaluated on several regression tasks and the results show that the proposed method can achieve competitive performance compared to the state-of-the-art methods.", "questions": "See the comments in \"weaknesses\".", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper tackles a very interesting problem of representation fragmentation in deep regression models. The proposed method is simple and very effective to learn continuous representations that fit the regression task. It is very interesting to see that the traditional methods learn representations clustered based on spurious targets (i.e., camera location in the SkyFinder dataset) while the proposed method learns a nice continuous representation that captures the target values.\n\n2. The authors present a theoretical analysis of the proposed loss function and show that it can learn delta-ordered feature embeddings when sufficiently trained.\n\n3. The proposed loss function is conceptually simple and easy to implement. The proposed method is thoroughly evaluated on several regression tasks and the results show that the proposed method can achieve competitive performance compared to the state-of-the-art methods. The authors further conducted ablation studies to show the effectiveness of the proposed method under data corruption, unseen or spurious targets.", "weaknesses": "1. While it is nice to see the proposed method leads to delta-ordered feature embeddings. It would be nice if the authors can further theoretically connect the delta-ordered feature embeddings to the final performance of the regression task. This would help to justify that the delta-ordered feature embeddings are indeed useful for the regression task.\n\n2. For the remark for Thm 3, to achieve delta-ordered features for the entire dataset, is it that we need to optimize all batches to achieve low enough loss? Is there any guarantee/insight that there is a feasible solution that can achieve low enough loss for all batches?\n\n3. It seems the IMDB-WIKI performance is missing in Table 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 6JYz, \n\nThanks for your constructive comments and insightful questions. We are delighted to see that you appreciate the contributions of our work. Below, we address your concerns in detail. \n> *It would be nice if the authors can further theoretically connect the delta-ordered feature embeddings to the final performance of the regression task.*\n\nThank you for the valuable suggestion. The delta-ordered feature embeddings can *indeed* be theoretically connected to the final performance: In our **Global Response**, we provide an analysis based on Rademacher Complexity to show that ***$\\delta$-ordered feature embedding leads to better generalization bound***. We will expand on the theoretical analysis therein and include it as a new theorem following Theorem 3 in the revised paper.\n\n> *For the remark for Thm 3, to achieve delta-ordered features for the entire dataset, is it that we need to optimize all batches to achieve low enough loss? Is there any guarantee/insight that there is a feasible solution that can achieve low enough loss for all batches?*\n\nThanks for the great question. We do not need to conduct optimizations for all batches, which is also practically impossible. In fact, one should consider the training process as a cohesive whole, which is effectively optimizing the **expectation** of the loss over all possible random batches. In addition, Markov's inequality [1] guarantees that when the expectation of the loss is optimized to be sufficiently low, the loss on any batch will be low enough with a high probability. We will add this point to the revised paper.\n\n> *It seems the IMDB-WIKI performance is missing in Table 2.*\n\nWe apologize for the confusion. As we mentioned in Appendix B, we used IMDB-WIKI only for the **analysis**: testing our method’s resilience to reduced training data, performance on transfer learning, and the ability to generalize to unseen targets. We didn’t include it in the main results because we already incorporated AgeDB in the main results for the task of age estimation from face images; in addition, the age labels in AgeDB have been manually cleaned by other researchers while the age labels in IMDB-WIKI contain noise [2]. We will make this point clearer in the revised paper and properly refer to them in the main text. \n\n[1] Grimmett & Stirzaker. Probability and random processes. Oxford University Press 2020. \n\n[2] Moschoglou et al. AgeDB: the first manually collected, in-the-wild age database. CVPR Workshop 2017. \n\n---\nWe hope that our response has addressed all of your concerns and offered any needed clarifications, and that you may consider a favorable increase of the score. Please do not hesitate to discuss with us if you have other comments. We are always happy to take any questions or suggestions.\n"}, {"review_id": "PAzEW7MLSU", "reviewer": "Reviewer_D7S9", "summary": "The authors discuss the benefits of contrastive learning for learning structured representations in a regression setting. While contrastive losses are typically formulated in terms of “similar” and “dissimilar” examples, the authors make use of the extra information conveyed by the continuous target label. They show how adding the RnC contrastive term to training is beneficial for a suite of high-dimensional regression tasks.\n", "questions": "* How is the temperature parameter \\tau tuned? [line 106]\n", "limitations": "Limitations and broader impacts are discussed [Sec. 5.4, App. H]. However, given that images of faces are used frequently in the experiments, I think that a more complete discussion about broader impacts, especially as it relates to task definition for regression, should be included (see “Weaknesses” above and my request for an ethics review below).", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This is a technically strong paper that, in my opinion, makes a contribution towards the important and understudied problem of representation learning for regression. The qualitative results [Fig 1] are compelling. The proposed method is intuitive and the paper is well written. The theoretical analysis and extensive empirical results suggest to me that the proposed method will be of interest to NeurIPS attendees. I was especially interested in the suggestion that contrastive training yields more robust representations [lines 259--279], although I think further work will be needed in the future to validate this finding in more sophisticated OOD generalization settings.\n", "weaknesses": "To my knowledge the technical contributions here are sound. However, after reading the paper, I found myself a bit concerned about how neural net-based regression models are going to be used in the near future. And I think the paper would benefit from a head-on discussion of the potential ethical issues at play. \n\nRegression is notoriously difficult for neural networks (for example, many baseline methods convert the regression into a classification problem). A technological advance here could open up entirely new application areas. However, the use of datasets involving human faces suggest to me that some of these new applications could bring up new ethical concerns as well. While Predicting age and gaze direction from face images (as done in this paper) might seem reasonable, there are existing critiques in the literature [e.g. https://ir.lawnet.fordham.edu/cgi/viewcontent.cgi?article=1804&context=iplj, https://medium.com/@blaisea/physiognomys-new-clothes-f2d4b59fdd6a, https://dl.acm.org/doi/pdf/10.1145/3375627.3375820] arguing that predicting other target variables, such as a continuous proxy for emotion, are ethically fraught. In the authors’ opinion, are there application areas where RnC should *not* be applied?\n\nI think the paper would benefit from a discussion of these issues. While the lack of such discussion (the broader impact section is, frankly, rather boilerplate) represents a weakness in my view, it probably should not stand in the way of the paper’s acceptance, since it is more of a critique of the whole subfield rather than this specific paper. However, I am going to request an ethics review in order to get a second opinion about this.\n\nThe related works are generally well covered. One exception is the recent C-mixup paper [https://arxiv.org/abs/2210.05775], which also uses label similarity in regression to group “similar” data points. However their approach uses these similarities for data augmentation rather than contrastive learning. A discussion of how the two approaches differ would benefit the reader.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer D7S9,\n\nThank you very much for your valuable feedback. We are delighted to see that you found the method intuitive, the results compelling and the paper well-written, and we wish to express our gratitude for bringing the ethics considerations to our attention. Here we address your concerns one by one.\n\n> *Given that images of faces are used frequently in the experiments, I think that a more complete discussion about broader impacts, especially as it relates to task definition for regression, should be included.*\n\nThank you for the dedicated discussion about ethical considerations. We certainly agree that these are essential points that deserve being addressed in the main body of the paper. We have **revised our Broader Impact and Limitations sections**, as shown below, and will move them to the **main body** of the revised manuscript. We hope that the revised sections address your concerns.\n\n**Broader Impacts.** \n*We introduce a novel framework designed to enhance the performance of generic deep regression learning. We believe this will significantly benefit regression tasks across various real-world applications. Nonetheless, several potential risks warrant discussion. First, when the framework is employed to regress sensitive personal attributes such as intellectual capabilities, health status, or financial standing from human data (like facial images or physiological signals), there's a danger it might reinforce or even introduce new forms of bias. Utilizing the method in these contexts could inadvertently justify discrimination or the negative targeting of specific groups. Second, in contrast to handcrafted features which are grounded in physical interpretations, the feature representations our framework learns can be opaque. This makes it difficult to understand and rationalize the model’s predictions, particularly when trying to determine if any biases exist. Third, when our method is trained on datasets that do not have a balanced representation of minority groups, there's no assurance of its performance on these groups being reliable. It is essential to recognize that these ethical concerns apply to deep regression (and classification) models at large, not solely our method. However, the continuous nature of our representation which facilitates interpolation and extrapolation might inadvertently make it more tempting to justify such unethical applications. Anyone seeking to implement or use our proposed method should be mindful of these concerns. Both our specific method and deep regression models, in general, should be used cautiously to avoid situations where their deployment might contribute to unethical outcomes or interpretations.*\n\n**Limitations.** \n*Our proposed method presents some limitations. Firstly, the technique cannot discern spurious or incidental correlations between the input and the target within the dataset. As outlined in the Broader Impact section, this could result in incorrect conclusions potentially promoting discrimination or unjust treatment when utilized to deduce personal attributes. Future research should delve deeper into the ethical dimensions of this issue and explore strategies to ensure ethical regression learning. A second limitation is that our evaluation primarily focuses on general regression accuracy metrics (e.g., MAE) without considering potential disparities when evaluating specific subgroups (e.g., minority groups). Given that a regression model's performance can vary across demographic segments, subgroup analysis is an avenue that warrants exploration in subsequent studies. Lastly, our approach learns continuous representations by contrasting samples against one another based on their ranking in the target space, necessitating label information. To adapt it for representation learning with unlabeled data, our framework will need some modifications, which we reserve for future work.*\n\nWe certainly welcome any further suggestions from the reviewer, and are more than happy to incorporate them to make the statements more comprehensive.\n\n> *The recent C-mixup paper also uses label similarity in regression to group “similar” data points. A discussion of how the two approaches differ would benefit the reader.*\n\nThanks for pointing out the missing reference. We sincerely apologize for the oversight during the submission phase. We will cite and discuss the C-mixup paper in our revised manuscript. Here is a draft of the discussion to be added to the **Related Work** section:\n\n*C-mixup [1] leverages label similarity for regression tasks. Specifically, it adapts the original mixup [2] data augmentation technique for regression learning by adjusting the sampling probability of the mixed pairs according to the label similarities. In contrast, our method contrasts samples against each other based on the rankings of label similarities. It is also worth noting that our method is orthogonal and complementary to data augmentation algorithms for regression, such as C-mixup.* \n\n> *How is the temperature parameter \\tau tuned? [line 106]*\n\nAs discussed in Appendix E, we performed standard hyper-parameter search for the temperature parameter $\\tau$ in {0.1, 0.2, 0.5, 1.0, 2.0, 5.0} and selected one with the best performance, which is 2.0.\n\n[1] Yao et al. C-mixup: Improving generalization in regression. NeurIPS 2022. \n\n[2] Zhang et al. Mixup: Beyond Empirical Risk Minimization. ICLR 2018.\n\n---\nWe hope our response has addressed all of your concerns and can lead to a favorable increase of the score. Please feel free to let us know if you have other questions or suggestions. We are more than willing to discuss more with you.\n"}, {"review_id": "TZIQXVutI3", "reviewer": "Reviewer_wKSy", "summary": "The authors present a new loss for representation learning in regression, RNC (Rank-n-contrast).  RNC can be seen as the SupCon loss adapted to the regression setting, where the labels given are not hard class labels but rather continuous regression labels.  In SupCon the negatives for each example are members of other classes and the positives come from members of the same class.  In RNC positive pairs are formed with every example to the anchor, and the negatives are those examples with larger label distances to the anchor than the positives.  By doing this they are able to train a representation for the data that is continuous in nature, which is hypothesized to represent the data better.", "questions": "Questions and possible directions of improvement are listed under \"Weaknesses\".", "limitations": "I believe this was sufficient.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.   I believe the method is sufficiently original as a nontrivial adaptation of the SupCon methodology to the regression setting.  The designation of what the positives and negatives are for the RNC contrastive loss are well motivated and I can see why the loss would give the kind of continuous representations that it does.\n\n2.  The method seems to be pretty simple to implement.  I particularly like that the RNC loss is the only loss the authors pretrained with and it wasn't some highly tuned composition of many different losses.  It gives me more confidence that the RNC loss is providing the gain in performance.\n\n3.  The theory is simple and well-motivated in that it proves the loss is doing what the authors are claiming it does.  The theory combined with the visualization of the representations give me confidence that the representations are in fact ordered continuously.\n\n4.  The experimental evaluation is fairly thorough.  The standard questions are answered, i.e. standard deviation error bars, how does it perform against other pretraining tasks, how does a plain two-stage training compare, whether augmentation is important, etc.   ", "weaknesses": "1.  I think the experimental evaluation can be improved to improve the generality of the method.  Currently the datasets that are evaluated are on computer vision regression tasks, which confines the evaluation to just ResNet-based models when we compare against the different losses and training methods in the paper.  In particular, one area where regression tasks are popular is the tabular setting.  I would be very interested to see a comparison between RNC and tree-based methods on popular tabular methods.  For examples of models and datasets to compare on the authors can check https://arxiv.org/abs/2106.11959 (not my paper btw)\n\n2.  It is great to see that the method is able to consistently perform well when compared against other baselines.  But as someone who is not familiar with these particular regression tasks I am not sure whether the magnitude of improvement is substantial.  Could the authors help me understand the scale of improvement?\n\n3.  Returning to my point on how the evaluation is restricted to ResNet-based architectures, I wonder if given that ResNet was designed to classify images that it has some inductive bias on clustering sets of images based on semantic similarities, which is what causes the disjointed representations in Figure 1 (left).  Expanding past the computer vision domain would be very valuable in making the contribution more impactful and general.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer wKSy,\n\nThanks for the constructive comments and insightful feedback. We are glad that you found the method novel and simple to implement, the theory simple and well-motivated and the evaluation thorough. Here we address your concerns one by one. \n\n> *Currently the datasets that are evaluated are on computer vision … I would be very interested to see a comparison between RNC and tree-based methods on popular tabular methods.*\n\nFirst, we would like to clarify that **not** all datasets included in our evaluation are image datasets - TUAB is a dataset consisting of EEG signals (i.e., time-series data). Besides, our method is **not** restricted to ResNet, but rather applicable to various deep architectures, including Transformer-based architectures. We chose ResNet as our main backbone because ResNet is the most commonly used architecture for mainstream regression methods and tasks [1, 2, 3, 4]. Here we further ran experiments with **ViT-Small** as the backbone on two datasets. The results (metric: MAE) are shown in the table below. \n\n||AgeDB|SkyFinder|\n|-|:-:|:-:|\n|L1|10.18|3.63|\n|**RnC(L1)**|**9.58**|**3.50**|\n\nNote that the performance with ViT could be worse than that with ResNet since ViT typically requires larger amounts of training data and usually performs worse on smaller datasets [5]. Nevertheless, RnC(L1) still performs *significantly better* than L1, showing the generality of RnC. \n\nBesides, following the reviewer’s suggestion, we further conducted experiments on a **tabular** dataset and compared RnC with tree-based methods and competitive deep models. Specifically, we followed the same evaluation protocols in [6] and evaluated on a subset (10% random subsampled due to time and computation limit) of Microsoft (MI, search queries) dataset [7]. The results are shown in the table below. \n\n|Method|RMSE|\n|-|:-:|\n|CatBoost|0.758 $\\pm$ 4.8e-4|\n|XGBoost|0.756 $\\pm$ 6.7e-4|\n|ResNet|0.762 $\\pm$ 5.3e-4|\n|**RnC(ResNet)**|0.756 $\\pm$ 6.4e-4|\n|FT-Transformer|0.760 $\\pm$ 8.9e-4|\n|**RnC(FT-Transformer)**|**0.753 $\\pm$ 7.7e-4**|\n\nThe results show that applying RnC to the deep models *significantly improved* their performance and allowed them to *outperform* the popular tree-based methods on the tabular dataset. We will add the above results and discussions in the revised paper. \n\n> *As someone who is not familiar with these particular regression tasks I am not sure whether the magnitude of improvement is substantial. Could the authors help me understand the scale of improvement?*\n\nThanks for the great question. Here we discuss the improvement scale for each dataset:\n\n- **AgeDB**: The incorporation of RnC reduces the prediction error by 5.8% on average for all regression methods in Table 2, and Table 3 shows that our performance gain against the best SOTA method is 0.31 years, while the performance gap between the best and the second-best SOTA method is 0.02 years.\n\n- **TUAB**: The incorporation of RnC reduces the prediction error by 9.3% on average for all regression methods in Table 2, and Table 3 shows that our performance gain against the best SOTA method is 0.31 years, while the performance gap between the best and the second-best SOTA method is 0.05 years.\n\n- **MPIIFaceGaze**: The incorporation of RnC reduces the prediction error by 11.7% on average for all regression methods in Table 2, and Table 3 shows that our performance gain against the best SOTA method is 0.18 degrees, while the performance gap between the best and the second-best SOTA method is 0.05 degrees.\n\n- **SkyFinder**: The incorporation of RnC reduces the prediction error by 7.0% on average for all regression methods in Table 2, and Table 3 shows that our performance gain against the best SOTA method is 0.06 degrees Celsius, while the performance gap between the best and the second-best SOTA method is 0.01 degree Celsius.\n\nWe believe that the performance gain from the proposed RnC framework is *significant* and *substantial*, and we will expand upon these explanations in the revision.\n\n> *I wonder if given that ResNet was designed to classify images that it has some inductive bias on clustering sets of images based on semantic similarities, which is what causes the disjointed representations in Figure 1 (left).*\n\nFirst, we would like to clarify that the three plots in Fig.1 are generated with the **same** architecture and only differ by the **losses** used to train them, which indicates that it is the different losses that lead to different structures (continuous or fragmented) in the representations. Second, regarding the reviewer’s concern on whether it is the inductive bias in ResNet that leads to the fragmented representations in Fig.1, we generated the same plots for L1 and RnC using **ViT-Small** as the backbone (**Please see the PDF attached in the Global Response**), which reveals similar structures and verifies that the presence of fragmented representations in existing general regression learning schemes stems from their inability to capture the underlying continuous order between samples. \n\n[1] Yang et al. Delving into deep imbalanced regression. ICML 2021.\n\n[2] Gong et al. Ranksim: Ranking similarity regularization for deep imbalanced regression. ICML 2022.\n\n[3] Zhang et al. Improving deep regression with ordinal entropy. ICLR 2023.\n\n[4] Engemann et al. A reusable benchmark for brain-age prediction from M/EEG resting-state signals. NeuroImage 2022.\n\n[5] Zhu et al. Understanding Why ViT Trains Badly on Small Datasets: An Intuitive Perspective. Arxiv 2023.\n\n[6] Gorishniy et al. Revisiting Deep Learning Models for Tabular Data. NeurIPS 2021.\n\n[7] Qin et al. Introducing LETOR 4.0 datasets. Arxiv 2013.\n\n---\nWe hope our response has thoroughly addressed your concerns, and would really appreciate it if you could consider raising your score accordingly. If you have any further questions or suggestions, please do not hesitate to share them. We are eager to engage in further discussions with you."}, {"review_id": "XJWP3mAneX", "reviewer": "Reviewer_dBkT", "summary": "The paper introduces a deep learning method, Rank-N-Contrast, for regression tasks. This method aims to capture continuity in data, something existing methods struggle with. The authors define a concept of $\\delta$-ordered feature embedding and show theoretically that if Rank-N-Contrast loss is minimized, feature embeddings will be $\\delta$-ordered. The proposed method excels in several regression tasks.", "questions": "1. **About Embedding Dimensions:** The learned representations in Fig. 1 are impressive, but the embedding dimension isn't specified. Were these embeddings trained directly in two dimensions, or were they first embedded in a high-dimensional space and then visualized with methods like UMAP? If UMAP was used, are the apparent clustering and lack of continuity in L1 and SupCon embeddings an artifact of the visualization process, or do these characteristics persist in the high-dimensional embeddings?\n2. **About the Construction of RNC loss:** In l.104, just before Equation 1, it's stated that the normalized likelihood of $v_j$ \"can be written as …\" Is Equation 1 derived from some definition or axiom? At least the normalized likelihood is “defined” in the reference [41].\n3. **About Trainability:** Theorem 3 claims that if RNC loss is sufficiently minimized, a desirable representation can be obtained. But how much can RNC loss be practically minimized? Can any insights be provided from a theoretical standpoint on the trainability of the proposed loss?", "limitations": "One potential limitation is the difference in the number of training epochs between the one-stage and two-stage methods. According to Appendix l.629, the one-stage methods were trained for 400 epochs, whereas the two-stage methods, including the proposed approach, were trained for a total of 500 epochs. This longer training period for the proposed method could potentially be contributing to the improvements observed in table 2. A further analysis or an ablation study might help to confirm whether these enhancements are genuinely due to the method itself and not merely a consequence of the extended training period.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper provides a novel perspective to regression tasks using deep learning models, a well-studied and widely-acknowledged problem setting. The paper's unique approach lies in its application of contrastive learning methods, which is a departure from conventional techniques that alter the loss function or incorporate specific regularization.\n\nThe paper sets itself apart by offering an extensive comparison to existing methodologies, using both theoretical and empirical results from regression and representation learning tasks. This comprehensive approach underlines the proposed method's potential and relevance.\n\nThe ubiquity of regression tasks in the realm of deep learning models implies that this paper's approach could contribute meaningfully to the field. Furthermore, the paper is well-composed, making it easy for readers to grasp the fundamental concept of the study.", "weaknesses": "The paper positions \"regression-aware representation\" as vital, but a slight disparity between the theoretical and experimental results is apparent.\n\nTheoretically, it suggests that a deep learning model can fulfill \"regression-aware embedding\" characteristics by minimizing the RNC loss appropriately. Experimentally, the paper exhibits consistent performance enhancements over existing approaches in both regression tasks and representation learning frameworks. Furthermore, it confirms that the implementation of RNC provides resilience against data corruption, improves performance with limited training data, and boosts transfer learning.\n\nWhile the paper implies the potential to secure $\\delta$-ordered feature embeddings by reducing RNC loss, and evidences steady performance improvement in real-world data contexts, the precise relationship between the continual characteristics of the \"regression-aware representations\" and the observed performance improvements is not fully clear yet. Additional discussion around this subject could offer more insights and, thereby, enrich our comprehension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer dBkT,\n\nThank you very much for acknowledging the novelty and the contributions of our work. We sincerely appreciate the time and effort you have dedicated to evaluating our work. In the following, we address your concerns in detail. \n\n> *The precise relationship between the continual characteristics of the \"regression-aware representations\" and the observed performance improvements is not fully clear yet. Additional discussion around this subject could offer more insights and, thereby, enrich our comprehension.*\n\nThanks for the valuable suggestion. Indeed, theoretical connections **do** exist between regression-aware representations and the observed enhancement in performance: In our **Global Response**, we provide a theoretical analysis based on Rademacher Complexity to show that ***$\\delta$-ordered feature embedding leads to better generalization bound***. We will expand on the analysis therein and include it in the revised manuscript as a new theorem following Theorem 3.\n\n> ***About Embedding Dimensions**: The learned representations in Fig. 1 are impressive, but the embedding dimension isn't specified. Were these embeddings trained directly in two dimensions, or were they first embedded in a high-dimensional space and then visualized with methods like UMAP? If UMAP was used, are the apparent clustering and lack of continuity in L1 and SupCon embeddings an artifact of the visualization process, or do these characteristics persist in the high-dimensional embeddings?*\n\nThanks for pointing this out, and we apologize for the confusion. The embeddings in Fig.1 are first embedded in a 512-dimensional embedding space, then visualized using UMAP. The clustering and lack of continuity are **unlikely** to be artifacts, because if the points are clustered / far apart in the UMAP visualization, it suggests that those points were close to / distant from each other in the high-dimensional space as well [29]. \n\nFurthermore, we calculated the Spearman’s rank correlation coefficient and the Kendall rank correlation coefficient between label similarities and feature similarities on that dataset in **Sec. 3 - Feature Ordinality**, where the feature similarities are computed from the **original** 512-dimensional feature vectors. The results in Table 1 confirm that the feature similarities learned by our method have significantly higher correlations with the label similarities than those by the L1 loss, which further verifies that the embeddings learned by RnC are indeed more continuous. We will make these points clearer in the revised paper.\n\n> ***About the Construction of RNC loss**: In l.104, just before Equation 1, it's stated that the normalized likelihood of v_j  \"can be written as …\" Is Equation 1 derived from some definition or axiom? At least the normalized likelihood is “defined” in the reference [41].*\n\nThank you for the great question. As stated in l.99, the **form** of the normalized likelihood is commonly used by related metric learning literatures [16, 43], where the likelihood is modeled to increase exponentially with respect to the feature similarity. Furthermore, we are inspired by [41] – where the denominator contains a **subset** of samples for ranking purposes – to introduce an adaptive set $\\mathcal{S_{i, j}}$ that contains the samples of higher rank than $v_j$ given $v_i$, and define a **customized** likelihood $\\mathbb{P}(v_j | v_i, \\mathcal{S_{i, j}})$ accordingly that is suitable for our problem settings.\n\nWe apologize for the potential confusion and will make these points clearer in the revised paper.\n\n> ***About Trainability**: Theorem 3 claims that if RNC loss is sufficiently minimized, a desirable representation can be obtained. But how much can RNC loss be practically minimized? Can any insights be provided from a theoretical standpoint on the trainability of the proposed loss?*\n\nThanks for the insightful comment. Actually, Theorem 2 proves that the RnC loss can be *arbitrarily* close to its lower bound, which means that the RnC loss can be sufficiently minimized for any $0 < \\delta < 1$ from a **theoretical** perspective.\n\nHowever, in practice, for any loss function, how much the loss can be minimized **empirically** depends a lot on the model, task, and data, such as whether the model capacity is large enough, whether the input contains sufficient information about the task, and whether the label is clean or not. Thus it usually cannot be *simply and/or universally* guaranteed. We will make a remark in the revised paper to discuss this point.\n\n> *One potential limitation is the difference in the number of training epochs between the one-stage and two-stage methods .... A further analysis or an ablation study might help to confirm whether these enhancements are genuinely due to the method itself and not merely a consequence of the extended training period.*\n\nThanks for pointing out the difference between training epochs. In fact, we have already included that ablation study in **Appendix F.4 of the submission**. In this section, we adopted the two-stage training scheme for each of the one-stage methods, i.e., training the predictor for 100 more epochs on top of the encoder which was trained for 400 epochs. The results show that the two-stage training scheme does **not** help improve the performance of those one-stage methods, which further validates that the benefit of RnC stems from the proposed loss function rather than the training scheme / number of training epochs.\n\n---\nWe thank you again for your time and feedback. We hope that our response has adequately answered your questions, and would lead to a favorable increase of the score. We are happy to discuss more if you have any further questions. \n"}, {"review_id": "9ylFOCpO7k", "reviewer": "Reviewer_2Kp1", "summary": "The paper proposes the framework Rank-N-Contrast in order to learn regression-aware feature representations. Authors claim that this representation learning mechanism captures the continuous nature of sample orders and helps achieve better performance in downstream regression task.", "questions": "See Weakness section.", "limitations": "I don't think authors have adequately addressed their limitations and weaknesses in the paper.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper provides both experimental results and theoretical proof to support their method. \n2. The paper includes relatively comprehensive ablation study and analysis section.", "weaknesses": "1. The paper is in lack high-level intuition and in-depth analysis of why the method work and only have low-level interpretation on the experimental results. Following are some example questions I hope authors could have addressed (and go one-step further than just stating their claims): Why preserving order in feature representation help to learn continuous nature? In fig. 3, why discernible pattern is better? What are the theorems trying to convey (my understanding is that feature embedding follows order inherited from the labels)?\n\n2. Although authors have conducted experiments on 4 different datasets (where is the results of IMDB-WIKI b.t.w.?) and with 7 different regression losses, the datasets mostly have scalar responses (except for MPIFaceGaze) and seem repetitive to me. I think the experimental results will be more convincing if authors include experiments with higher-dimensional responses.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 2Kp1,\n\nThank you for your valuable questions and thoughtful feedback. Your comments have helped us to further improve the quality of our paper. However, we believe that there are several **important misunderstandings** which we would like to clarify and address point-by-point. We hope you could re-evaluate the paper based on our clarifications, and would really appreciate it if you could consider updating the review, and raising your score accordingly.\n\n> *Why preserving order in feature representation help to learn continuous nature?*\n\nThe continuous nature in data means that the target value is continuous, and thus the target distances between different data samples have **orders**. This intuition motivates us to learn a representation that preserves the order in the label space. \n\nIn our **Global Response**, we provide a theoretical analysis based on Rademacher Complexity to demonstrate that a feature embedding that preserves the order in the label space enables a better **generalization bound**, leading to better regression performance. Further, in our experiments, the superior performance (Sec. 5.1), as well as the improved data efficiency, robustness to spurious targets and data corruptions, and generalization to distribution shifts (Sec. 5.2) confirm that preserving order in the feature space *indeed* helps to learn the continuous nature of regression tasks.\n\nTo summarize, we’ve justified both **theoretically** and **empirically** that preserving order in feature representation helps to learn continuous targets. \n\n> *In fig. 3, why discernible pattern is better?*\n\nAt a high level, Fig.3 illustrates the **correlation** between feature similarities and label similarities for both RnC and L1.  Each entry (i,j) in the matrix refers to the feature similarity between samples i and j.  Samples in the rows and columns of the matrix are ordered according to increased label values, i.e., the first row/column refers to the sample with the smallest label and the last row/column to the sample with the largest label. Thus, the samples that are most similar in the label space are the ones along the diagonal and the least similar in the label space are the two corners farthest away from the diagonal, with other points between the diagonal and these two corners gradually decreasing in their similarity in the label space.  The **more** the feature similarity pattern (i.e., the color pattern in the figure) follows the label similarity pattern (described above), the **higher** the correlation between the similarity in the feature space and the label space. We will make them clearer in the revised paper.\n\n> *What are the theorems trying to convey?*\n\nThe theoretical analysis is trying to prove that optimizing the proposed RnC loss will lead to an ordered feature embedding from a theoretical perspective. Thus, we first formalize the description of ordered feature embedding as $\\delta$-ordered (Definition 1), and then demonstrate that when the proposed RnC loss is optimized to be sufficiently low, the feature embedding will be $\\delta$-ordered (Theorem 3). We also prove that the proposed RnC loss is able to be optimized to be low enough (Theorem 2). Besides, as highlighted in the global response, we further prove that learning $\\delta$-ordered feature embeddings can indeed lead to better regression performance based on Rademacher Complexity. Combining all above, we conclude that optimizing the RnC loss will lead to better regression performance.\n\n> *Although ..., the datasets mostly have scalar responses (except for MPIFaceGaze) and seem repetitive to me. I think ... experiments with higher-dimensional responses.*\n\nWe want to clarify that the datasets we included in the experiments are **not** repetitive. To comprehensively evaluate the performance over regression tasks, we should not only take into consideration the output dimension, but also other important factors, such as application domain and input dimensions/modalities. The datasets in our evaluation are carefully selected considering all these factors together. They cover:\n\n- Diversity in **application domains**: computer vision (AgeDB), human-computer interaction (MPIIFaceGaze), healthcare (TUAB) and weather monitoring (SkyFinder).\n- Diversity in **input dimensions/modalities**: 2D images (AgeDB, MPIIFaceGaze, SkyFinder), and 1D time series (TUAB).\n- Diversity in **output dimensions**: scalar values (AgeDB, TUAB, SkyFinder) and higher-dimensional vectors (MPIIFaceGaze).\n\nPlease note that the level of diversity in our datasets is significantly high. Nonetheless, if the reviewer has specific suggestions on datasets with higher output dimensions, we are happy to include more results.\n\n> *Where is the results of IMDB-WIKI?*\n\nWe apologize for the confusion. As we mentioned in Appendix B, we used IMDB-WIKI only for the **analysis**: testing our method’s resilience to reduced training data, performance on transfer learning, and the ability to generalize to unseen targets. We didn’t include it in the main results because we already incorporated AgeDB in the main results for the task of age estimation from face images; in addition, the age labels in AgeDB have been manually cleaned by other researchers while the age labels in IMDB-WIKI contain noise [1]. We will make this point clearer in the revised paper and properly refer to them in the main text. \n\n[1] Moschoglou et al. AgeDB: the first manually collected, in-the-wild age database. CVPR Workshop 2017. \n\n---\nWe hope the above clarifications have addressed all of your concerns, and made you more confident about the novelty, significance, and completeness of our paper. If you have more questions or suggestions, please do not hesitate to discuss with us. We thank you again for your time and feedback. We hope you could re-evaluate the paper based on our clarifications, and would really appreciate it if you could consider updating the review, and raising your score accordingly.\n\n"}]

Author rebuttal:
We are grateful to all the reviewers for the time and effort they invested in reviewing our paper. It is heartening to note that the reviewers found:
- The paper addresses an **important** (D7S9), **ubiquitous** (dBkT), and **interesting** (6JYz) problem.
- The proposed method is **novel** (wKSy, dBkT), **well-motivated** (wKSy), **intuitive** (D7S9), and **easy to implement** (6JYz, wKSy).
- The proposed method is **justified theoretically** (6JYz, D7S9, wKSy, dBkT, 2Kp1), and the theories are **well-motivated** (wKSy).
- The empirical evaluations are **comprehensive** (6JYz, D7S9, wKSy, dBkT, 2Kp1), and the results are **compelling** (6JYz, D7S9).
- The paper is **well-written** and **easy-to-follow** (D7S9, dBkT).

We made a concerted effort to provide a comprehensive response to each reviewer, with point-to-point answers following each review. We hope that our response adequately addresses the reviewers’ concerns and would be happy to answer any additional questions you may have.

---
In this **Global Response** section, we would like to answer a common question raised by Reviewers 6JYz, dBkT and 2Kp1:
> *How does the delta-ordered feature embeddings relate to final performance gain for the regression tasks from a theoretical perspective?*

We thank the reviewers for highlighting this insightful question, which helps to further enhance the completeness of our paper. Learning an ordered feature embedding can *indeed* boost the performance of the regression task.
Below, we present an analysis based on Rademacher Complexity [1] to substantiate that ***$\delta$-ordered feature embedding results in a better generalization bound***:

Specifically, regression learning can be formulated as finding a hypothesis $h$ to predict the target $y$ from the input $x$. Suppose there are $m$ data points in the training set $\mathcal{S}=\\{(x_k, y_k)\\}^m_{k=1}$. Let $\mathcal{H}_1$ be the class of all possible functions from the input space to the target space.

If a $\delta$-ordered feature embedding is guaranteed with an encoder $f$ mapping $x_k$ to $v_k$, the set of candidate hypotheses can be reduced to all "$\delta$-monotonic" functions $h(x) = g(f(x))$, which satisfy $\forall i, j$ and $k$, $d(g(v_i), g(v_j)) < d(g(v_i), g(v_k))$ for $s_{i,j} > s_{i,k} + \frac{1}{\delta}$, $d(g(v_i), g(v_j)) = d(g(v_i), g(v_k))$ for $|s_{i, j} - s_{i, k}| < {\delta}$, and $d(g(v_i), g(v_j)) > d(g(v_i), g(v_k))$ for $s_{i,j} < s_{i,k} - \frac{1}{\delta}$, where $d(\cdot,\cdot)$ is the target distance measure and $s_{i, j}$ is the feature similarity between $v_i$ and $v_j$. We denote the class of all "$\delta$-monotonic" functions by $\mathcal{H}_2$. Note that the optimal hypothesis, i.e., $\forall x, y, h^*(x) = y$, is in both $\mathcal{H}_1$ and $\mathcal{H}_2$.

Further, for a hypothesis set $\mathcal{H}_i$, let $\mathcal{A}_i = \\{(l((x_1, y_1); h), ..., l((x_m, y_m); h)): h\in\mathcal{H}_i\\}$ be the loss set for each hypothesis in $\mathcal{H}_i$ with respect to the training set $\mathcal{S}$, where $l$ is the loss function. Let $c_i$ be the upper bound of $|l((x, y); h))|$ for all $x, y$ and $h \in \mathcal{H}_i$.
We introduce the Rademacher Complexity [1] of $\mathcal{A}_i$, denoted as $R(\mathcal{A}_i)$.
Then, from the generalization bound based on Rademacher Complexity [1], we have, with a high probability (at least $1-\epsilon$), the gap between the empirical risk (i.e., training error) and the expected risk (i.e., test error) is upper bounded by $2R(\mathcal{A}_i) + 4c_i\sqrt{\frac{2\ln(4/\epsilon)}{m}}$.

Since $\mathcal{H}_2 \subset \mathcal{H}_1$, we have $\mathcal{A}_2 \subset \mathcal{A}_1$ and $c_2 \leq c_1$, and from the monotonicity of Rademacher Complexity we have $R(\mathcal{A}_2) \leq R(\mathcal{A}_1)$. Hence, with a $\delta$-ordered feature embedding, the upper bound on the gap between the training error and the test error will be reduced, which leads to better regression performance.

---
To put it more intuitively, fitting an ordered feature embedding **reduces** the **complexity** of the regressor, which enables **better generalization ability** from training to testing, and ultimately leads to the **final performance gain**.

Relatedly, we note that the enhanced generalization ability is further **empirically verified** in our paper (see **Sec. 5.2** in our main paper). Specifically, if not constrained, the learned feature embeddings could capture spurious or easy-to-learn features that are not generalizable to the real continuous targets (see **Robustness to Spurious Targets**). Such property also leads to better robustness to data corruptions, better resilience to reduced training data, and better generalization to unseen targets.

To summarize, learning an ordered feature embedding can indeed lead to better performance for regression tasks. We will describe these results formally as a new theorem following Theorem 3 in the revised paper. We believe the new results will make our paper more significant and comprehensive.

[1] Shalev-Shwartz & Ben-David. Understanding machine learning: From theory to algorithms. Cambridge University Press 2014.

---
We hope our response has adequately addressed the reviewers’ question(s), and would really appreciate it if the reviewers could consider raising their scores after reading our response.  We are happy to take any further questions from the reviewers. 


Meta-review: This paper proposes a strategy for learning embeddings for regression tasks. Reviewers appreciated the originality of the method, and the empirical and theoretical results. Although several concerns (including ethical) were raised, they seemed to properly addressed during the review period. All reviewers unanimously suggest that this paper be accepted.
Decision: Accept (spotlight)