Title: Rank-N-Contrast: Learning Continuous Representations for Regression
Venue: NeurIPS 2023
Accepted: yes
Keywords: regression

representation learning

continuity
Abstract: Deep regression models typically learn in an end-to-end fashion without explicitly emphasizing a regression-aware representation. Consequently, the learned representations exhibit fragmentation and fail to capture the continuous nature of sample orders, inducing suboptimal results across a wide range of regression tasks. To fill the gap, we propose Rank-N-Contrast (RNC), a framework that learns continuous representations for regression by contrasting samples against each other based on their rankings in the target space. We demonstrate, theoretically and empirically, that RNC guarantees the desired order of learned representations in accordance with the target orders, enjoying not only better performance but also significantly improved robustness, efficiency, and generalization. Extensive experiments using five real-world regression datasets that span computer vision, human-computer interaction, and healthcare verify that RNC achieves state-of-the-art performance, highlighting its intriguing properties including better data efficiency, robustness to spurious targets and data corruptions, and generalization to distribution shifts.

Rank-N-Contrast: Learning Continuous Representations for Regression

Abstract

Deep regression models typically learn in an end-to-end fashion without explicitly emphasizing a regression-aware representation . Consequently, the learned representations exhibit fragmentation and fail to capture the continuous nature of sample orders, inducing suboptimal results across a wide range of regression tasks. To fill the gap, we propose Rank-N-Contrast (RNC), a framework that learns continuous representations for regression by contrasting samples against each other based on their rankings in the target space. We demonstrate, theoretically and empirically, that RNC guarantees the desired order of learned representations in accordance with the target orders, enjoying not only better performance but also significantly improved robustness, efficiency, and generalization. Extensive experiments using five real-world regression datasets that span computer vision, human-computer interaction, and healthcare verify that RNC achieves state-of-the-art performance, highlighting its intriguing properties including better data efficiency, robustness to spurious targets and data corruptions, and generalization to distribution shifts. Code is available at: https://github.com/kaiwenzha/Rank-N-Contrast .

L

1

Temp. (

℃

)

30

20

10

0

SupCon

R

N

C

(Ours)

1 Introduction

Regression problems are pervasive and fundamental in the real world, spanning various tasks and domains including estimating age from human appearance [36], predicting health scores via human physiological signals [11], and detecting gaze directions using webcam images [52]. Due to the

continuity in regression targets, the most widely adopted approach for training regression models is to directly predict the target value and employ a distance-based loss function, such as L 1 or L 2 distance, between the prediction and the ground-truth target [51, 52, 38]. Methods that tackle regression tasks using classification models trained with cross-entropy loss have also been studied [36, 33, 40].

However, previous methods focus on imposing constraints on the final predictions in an end-to-end fashion, but do not explicitly emphasize the representations learned by the model. Unfortunately, these representations are often fragmented and incapable of capturing the continuous relationships that underlie regression tasks. Fig. 1(a) highlights the representations learned by the L 1 loss on SkyFinder [7], a regression dataset for predicting weather temperature from webcam outdoor images captured at different locations (details in Sec. 5). Rather than exhibiting the continuous ground-truth temperatures, the learned representations are grouped by different webcams in a fragmented manner. Such unordered and fragmented representation is suboptimal for the regression task and can even hamper performance by including irrelevant information, such as the capturing webcam.

Furthermore, despite the great success of representation learning schemes on solving discrete classification or segmentation tasks (e.g., contrastive learning [4, 20] and supervised contrastive learning (SupCon) [25]), less attention has been paid to designing algorithms that capture the intrinsic continuity in data for regression. Interestingly, we highlight that existing representation learning methods inevitably overlook the continuous nature in data: Fig. 1(b) shows the representation learned by SupCon on the SkyFinder dataset, where it again fails to capture the underlying continuous order between the samples, resulting in a suboptimal representation for regression tasks.

To fill the gap, we present Rank-N-Contrast (RNC), a novel framework for generic regression learning. RNC first learns a regression-aware representation that orders the distances in the embedding space based on the target values, and then leverages it to predict the continuous targets. To achieve this, we propose the Rank-N-Contrast loss ( L RNC), which ranks the samples in a batch according to their labels and then contrasts them against each other based on their relative rankings. Theoretically, we prove that optimizing L RNC results in features that are ordered according to the continuous labels, leading to improved performance in downstream regression tasks. As confirmed in Fig. 1(c), RNC learns continuous representations that capture the intrinsic ordered relationships between samples. Notably, our framework is orthogonal to existing regression methods, allowing for the use of any regression method to map the learned representation to the final prediction values.

To support practical evaluations, we benchmark RNC against state-of-the-art (SOTA) regression and representation learning schemes on five real-world regression datasets that span computer vision, human-computer interaction, and healthcare. Rigorous experiments verify the superior performance, robustness, and efficiency of RNC on learning continuous targets. Our contributions are as follows:

We identify the limitation of current regression and representation learning methods for continuous targets, and uncover intrinsic properties of learning regression-aware representations.

We conduct extensive experiments on five diverse regression datasets in vision , human-computer interaction , and healthcare , verifying the superior performance of RNC against SOTA schemes.

We design RNC, a simple & effective method that learns continuous representations for regression.

Further analyses reveal intriguing properties of RNC on its data efficiency, robustness to spurious targets & data corruptions, and better generalization to unseen targets.

2 Related Work

Regression Learning. Deep learning has achieved great success in addressing regression tasks [36, 51, 52, 38, 11]. In regression learning, the final predictions of the model are typically trained in an end-to-end manner to be close to the targets. Standard regression losses include the L 1 loss, the mean squared error (MSE) loss, and the Huber loss [22]. Past work has also proposed several variants. One branch of work [36, 13, 14, 35] divides the regression range into small bins, converting the problem into a classification task. Another line of work [12, 2, 40] casts regression as an ordinal classification problem [33] using ordered thresholds and employing multiple binary classifiers. Recently, a line of work proposes to regularize the embedding space for regression, ranging from modeling feature space uncertainty [28], encouraging higher-entropy feature spaces [50], to regularizing features for imbalanced regression [44, 17]. In contrast to existing works, we provide a regression-aware representation learning approach that emphasizes the continuity in the features space w.r.t. the targets, which enjoys better performance while being compatible to prior regression schemes. In addition,

Age:

20

Age:

30

Age:

70

Age:

0

,

Anchor

,

,

{

}

(       ,       )

Positive Pair

Corresponding Negative Pair(s)

(       ,       )

(       ,        )

(

,        )

20

30

20

0

70

20

0

70

20

(       ,       )

20

(a)

A Batch of Samples

(b)

Pair Construction for

RNC

C-Mixup [47] adapts the original mixup [49] by adjusting the sampling probability of the mixed pairs according to the target similarities. It is also worth noting that our method is orthogonal and complementary to data augmentation algorithms for regression learning, such as C-Mixup [47].

Representation Learning. Representation learning is crucial in machine learning, often studied in the context of classification. Recently, contrastive learning has emerged as a popular technique for self-supervised representation learning [4, 20, 5, 8, 3]. The supervised version of contrastive learning, SupCon [25], has been shown to outperform the conventional cross-entropy loss on multiple discrete classification tasks, including image recognition [25], noisy labels [26], long-tailed classification [23, 27], and out-of-domain detection [48]. A few recent papers propose to adapt SupCon to tackle ordered labels in specific downstream applications, including gaze estimation [42], medical imaging [9, 10], and neural behavior analysis [37]. Different from prior works that directly adapt SupCon, we incorporate the intrinsic property of label continuity for designing representation learning scheme tailored for regression, which offers a simple and principled approach for generic regression tasks.

3 Our Approach: Rank-N-Contrast (RNC)

Problem Setup. Given a regression task, we aim to train a neural network model composed of a feature encoder f ( · ) : X → R d e and a predictor g ( · ) : R d e → R d t to predict the target y ∈ R d t based on the input data x ∈ X .

In this context, a natural question arises: How to design a regression-aware representation learning scheme tailored for continuous and ordered samples?

For a positive integer I , let [ I ] denote the set { 1 , 2 , · · · , I } . Given a randomly sampled batch of N input and label pairs { ( x n , y n ) } n ∈ [ N ] , we apply standard data augmentations to obtain a two-view batch { (˜ x ℓ , ˜ y ℓ ) } ℓ ∈ [2 N ] , where ˜ x 2 n = t ( x n ) and ˜ x 2 n -1 = t ′ ( x n ) , with t and t ′ being independently sampled augmentation operations, and ˜ y 2 n = ˜ y 2 n -1 = y n , ∀ n ∈ [ N ] . The augmented batch is then fed into the encoder f to obtain the feature embedding for each augmented input data, i.e., v l = f (˜ x l ) ∈ R d e , ∀ l ∈ [2 N ] . The representation learning phase is then performed over the feature embeddings. To harness the acquired representation for regression, we freeze the encoder f ( · ) and train the predictor g ( · ) on top of it using a regression loss (e.g., L 1 loss).

The Rank-N-Contrast Loss. In order to align distances in the embedding space ordered by distances in their labels, we propose Rank-N-Contrast loss ( L RNC), which first ranks the samples according to their target distances, and then contrasts them against each other based on their relative rankings.

̸

Following [16, 45], given an anchor v i , we model the likelihood of any other v j to increase exponentially with respect to their similarity in the representation space [15]. Inspired by the listwise ranking methods [43, 6], we introduce S i,j := { v k | k = i, d (˜ y i , ˜ y k ) ≥ d (˜ y i , ˜ y j ) } to denote the set of samples that are of higher ranks than v j in terms of label distance w.r.t. v i , where d ( · , · ) is the distance measure between two labels (e.g., L 1 distance). Then the normalized likelihood of v j given v i and S i,j can be written as

where sim( · , · ) is the similarity measure between two feature embeddings (e.g., negative L 2 norm) and τ denotes the temperature parameter. Note that the denominator is a sum over the set of samples that possess higher ranks than v j ; Maximizing P ( v j | v i , S i,j ) effectively increases the probability that v j outperforms the other samples in the set and emerges at the top rank within S i,j .

As a result, we define the per-sample RNC loss as the average negative log-likelihood over all other samples in a given batch:

̸

Intuitively, for an anchor sample i , any other sample j in the batch is contrasted with it, enforcing the feature similarity between i and j to be larger than that of i and any other sample k in the batch, if the label distance between i and k is larger than that of i and j . Minimizing l ( i ) RNC will align the orders of feature embeddings with their corresponding orders in the label space w.r.t. anchor i .

L RNC is then enumerating over all 2 N samples as anchors to enforce the entire feature embeddings ordered according to their orders in the label space:

̸

Interpretation. To exploit the inherent continuity underlying the labels, L RNC ranks samples in a batch with respect to their label distances to the anchor. When contrasting the anchor to the sample in the batch that is closest in the label space, it enforces their similarity to be larger than all other samples in the batch. Similarly, when contrasting the anchor to the second closest sample in the batch, it enforces their similarity to be larger than only those samples that have a rank of three or higher in terms of distance to the anchor. This process is repeated for higher-rank samples (i.e., the third closest, fourth closest, etc.) and for all anchors in a batch.

Feature Ordinality. We further examine the impact of RNC on the ordinality of learned features. Fig. 3 visualizes the feature similarity matrices obtained from 2,000 randomly sampled data points in a realworld temperature regression task [7] for models trained using the vanilla L 1 loss and RNC. For clarity, the data points are sorted based on their ground-truth labels, with the expectation that the matrix values decrease progressively from the diagonal to the periphery. Notably, our method, RNC, exhibits a more discernible pattern compared to the L 1 loss. Furthermore, we calculate two quantitative metrics, the Spearman's rank correlation coefficient [41] and the Kendall rank correlation coefficient [24], between the label similarities and the feature similarities for both methods. The results in Table 1 confirm that the feature similarities learned by our method have a significantly higher correlation with label similarities than those by the L 1 loss.

L

RNC(

L

)

1

1

Connections to Contrastive Learning. The loss can also be explained in the context of positive and negative pairs in contrastive learning. Contrastive learning and SupCon are designed for classification tasks, where positive pairs consist of samples that belong to the same class or the same input image, while all other samples are considered as negatives. In regression, however, there are no distinct classes but rather continuous labels [44, 46]. Thus in L RNC, any two samples can be considered as a positive or negative pair depending on the context. For a given anchor sample i , any other sample j in the same batch can be used to construct a positive pair with the corresponding negative samples set to all samples in the batch whose labels differ from i 's label by more than the label of j . Fig. 2(a) shows an example batch, and Fig. 2(b) shows two positive pairs and their corresponding negative pair(s).

4 Theoretical Analysis

In this section, we theoretically prove that optimizing L RNC results in an ordered feature embedding that corresponds to the ordering of the labels. All proofs are in Appendix A.

Notations. Let s i,j := sim( v i , v j ) /τ and d i,j := d (˜ y i , ˜ y j ) , ∀ i, j ∈ [2 N ] . Let D i, 1 < D i, 2 < · · · < D i,M i be the sorted label distances starting from the i -th sample (i.e., sort ( { d i,j | j ∈ [2 N ] \{ i }} ) ),

∀ i ∈ [2 N ] . Let n i,m := |{ j | d i,j = D i,m , j ∈ [2 N ] \{ i }}| be the number of samples whose distance from the i -th sample equals D i,m , ∀ i ∈ [2 N ] , m ∈ [ M i ] .

First, to formalize the concept of ordered feature embedding according to the order in the label space, we introduce a property termed as δ -ordered for a set of feature embeddings { v l } l ∈ [2 N ] .

Definition 1 ( δ -ordered feature embeddings) . For any 0 < δ < 1 , the feature embeddings { v l } l ∈ [2 N ] are δ -ordered if ∀ i ∈ [2 N ] , j, k ∈ [2 N ] \{ i } ,

Definition 1 implies that a set of feature embeddings that is δ -ordered satisfies the following properties: ❶ For any j and k such that d i,j = d i,k , the difference between s i,j and s i,k is no more than δ ; and ❷ For any j and k such that d i,j < d i,k , the value of s i,j exceeds that of s i,k by at least 1 δ . Note that 1 δ > δ , which indicates that the feature similarity gap between samples with different label distances

to the anchor is always larger than that between samples with equal label distances to the anchor.

Next, we demonstrate that the batch of feature embeddings will be δ -ordered as the optimization of L RNC approaches its lower bound. In order to prove this, it is necessary to derive a tight lower bound for L RNC. Let L ⋆ := 1 2 N (2 N -1) ∑ 2 N i =1 ∑ M i m =1 n i,m log n i,m , we have:

Theorem 1 (Lower bound of L RNC) . L ⋆ is a lower bound of L RNC , i.e., L RNC > L ⋆ .

Theorem 2 (Lower bound tightness) . For any ϵ > 0 , there exists a set of feature embeddings such that L RNC < L ⋆ + ϵ .

The above theorems verify the lower bound of L RNC as well as its tightness. Given that L RNC can approach its lower bound L ⋆ arbitrarily, we demonstrate that the feature embeddings will be δ -ordered when L RNC is sufficiently close to L ⋆ for any 0 < δ < 1 .

Theorem 3 (Main theorem) . For any 0 < δ < 1 , there exist ϵ > 0 , such that if L RNC < L ⋆ + ϵ , then the feature embeddings are δ -ordered.

From Batch to Entire Feature Space. Now we have examined the property of a batch of feature embeddings optimized using L RNC. However, what will be the final outcome for the entire feature space when L RNC is optimized? In fact, if any batch of feature embeddings is optimized to achieve a low enough loss such that it is δ -ordered, the entire feature embedding will also be δ -ordered. This is because any triplet ( i, j, k ) in the entire feature embeddings is certain to appear in some batch, thus their feature embeddings ( v i , v j , v k ) will satisfy the condition in Definition 1. Nevertheless, to achieve δ -ordered features for the entire feature embeddings, do we need to optimize all batches to achieve a sufficiently low loss? The answer is no. Optimizing every batch is not only unnecessary, but also practically infeasible. In fact, one should consider the training process as a cohesive whole, which is effectively optimizing the expectation of the loss over all possible random batches. Then, the Markov's inequality [18] guarantees that when the expectation of the loss is optimized to be sufficiently low, the loss on any batch will be low enough with a high probability.

Connections to Final Performance. Suppose we have a δ -ordered feature embedding, how can it help to boost the final performance of a regression task? In Appendix B, we present an analysis based on Rademacher Complexity [39] to prove that a δ -ordered feature embedding results in a better generalization bound. To put it intuitively, fitting an ordered feature embedding reduces the complexity of the regressor, which enables better generalization ability from training to testing, and ultimately leads to the final performance gain. Relatedly, we note that the enhanced generalization ability is further empirically verified in Sec. 5. Specifically, if not constrained, the learned feature embeddings could capture spurious or easy-to-learn features that are not generalizable to the real continuous targets. Moreoever, such property also leads to better robustness to data corruptions, better resilience to reduced training data, and better generalization to unseen targets.

5 Experiments

Datasets. We perform extensive experiments on five regression datasets that span different tasks and domains, including computer vision, human-computer interaction, and healthcare. Complete descriptions of each dataset and example inputs are in Table 7, Appendix C and D.

AgeDB ( Age ) [32, 44] is a dataset for predicting age from face images, containing 16,488 in-the-wild images of celebrities and the corresponding age labels.

TUAB ( Brain-Age ) [34, 11] aims for brain-age estimation from EEG resting-state signals, with 1,385 21-channel EEG signals sampled at 200Hz from individuals with age from 0 to 95.

IMDB-WIKI ( Age ) [36, 44] is a dataset for age prediction from face images, which contains 523,051 celebrity images and the corresponding age labels. We use this dataset only for the analysis.

MPIIFaceGaze ( Gaze Direction ) [51, 52] contains 213,659 face images collected from 15 participants during natural everyday laptop use. The task is to predict the gaze direction.

SkyFinder ( Temperature ) [31, 7] contains 35,417 images captured by 44 outdoor webcam cameras for in-the-wild temperature prediction.

Metrics. Wereport two distinct metrics for model evaluation: ❶ Prediction error, and ❷ Coefficient of determination (R 2 ). Prediction error provides practical insight for interpretation, while R 2 quantifies the difficulty of the task by measuring the extent to which the model outperforms a dummy regressor that always predicts the mean value of the training labels. We report the mean absolute error (MAE) for age, brain-age, and temperature prediction, and the angular error for gaze direction estimation.

Experiment Settings. We employ ResNet-18 [19] as the main backbone for AgeDB , IMDB-WIKI , MPIIFaceGaze , and SkyFinder . In addition, we confirm that the results are consistent with other backbones (e.g., ResNet-50), and report related results in Appendix G.1. For TUAB , a 24-layer 1D ResNet [19] is used as the backbone model to process the EEG signals, with a linear regressor acting as the predictor. We fix data augmentations to be the same for all methods, where the details are reported in Appendix D. Negative L 2 norm (i.e., -∥ v i -v j ∥ 2 ) is used as the feature similarity measure in L RNC, with L 1 distance as the label distance measure for AgeDB , IMDB-WIKI , SkyFinder , and TUAB , and angular distance as the label distance measure for MPIIFaceGaze . We provide complete experimental settings and hyper-parameter choices in Appendix F.

5.1 Main Results

Comparison and compatibility to end-to-end regression methods. As explained earlier, our model learns a regression-suitable representation that can be used by any end-to-end regression methods. Thus, we implement seven generic regression methods: L 1 , MSE, and HUBER [44] ask the model to directly predict the target and utilize an error-based loss function; DEX [36] and DLDL-V2 [14] divide the regression range of each label dimension into pre-defined bins and learn the probability

distribution over them; OR [33] and CORN [40] design multiple ordered thresholds for each label dimension and learn a binary classifier for each threshold. Further details of these baselines are included in Appendix E.1. In our comparison, we first train the encoder with the proposed L RNC. We then freeze the encoder and train a predictor on top of it using each of the baseline methods. The original baseline without the RNC representation is then compared to that with RNC. For instance, we denote the end-to-end training of the encoder and predictor using L 1 loss as L 1 , while RNC( L 1 ) denotes training the encoder with L RNC and subsequently training the predictor using L 1 loss.

Table 2 summarizes the evaluation results on AgeDB , TUAB , MPIIFaceGaze and SkyFinder . Green numbers highlight the performance gains by using RNC representation. The standard deviations of the best results on each dataset are reported in Appendix G.2. As the table indicates, RNC consistently achieves the best performance on both metrics across all datasets. Moreover, incorporating RNC to learn the representation consistently reduces the prediction error of all baselines by 5.8% , 9.3% , 11.7% , and 7.0% on average on AgeDB , TUAB , MPIIFaceGaze , and SkyFinder , respectively.

Comparison to state-of-the-art representation learning methods. We further compare RNC with state-of-the-art representation learning methods, SIMCLR [4], DINO [3], and SUPCON [25], under both linear probing and fine-tuning schemes to evaluate their learned representations for regression tasks. Full details are in Appendix E.2. Note that SIMCLR and DINO do not use label information while SUPCON uses label information for training the encoder. The predictor is trained with L 1 loss. Table 3 demonstrates that RNC outperforms all other methods across all datasets. Note that some of the representation learning schemes even underperform the vanilla L 1 method, which further verifies that the performance gain of RNC stems from our proposed loss rather than the pre-training scheme.

Comparison to state-of-the-art regression learning methods. We also compare RNC with stateof-the-art regression learning schemes, including methods that perform regularization in the feature space [44, 17, 50] and methods that adopt dataset-specific techniques [1, 7]. We provide the details in Appendix E.3. L 1 loss is employed as the default regression loss for all methods if not specified. The results are presented in Table 3, with a dash ( -) indicating that the method is not applicable to the dataset. We observe that RNC achieves state-of-the-art performance across all datasets.

5.2 Analysis

Robustness to Data Corruptions. Deep neural networks are widely acknowledged to be vulnerable to out-of-distribution data and various forms of corruption, such as noise, blur, and color distortions [21]. To analyze the robustness of RNC, we generate corruptions on the AgeDB test set using the corruption process from the ImageNet-C benchmark [21], incorporating 19 distinct types of corruption at varying severity levels. We compare RNC( L 1 ) with L 1 by training both models on the original AgeDB training set, but directly testing them on the corrupted test set across all severity levels. The

6.63

7.47

8.60

9.86

11.75

6.14

6.85

7.78

8.78

9.60

5

7

9

11

13

0

1

2

3

4

Mean Absolute Error (MAE)

Corruption Severity Level

L

1

RNC(

L

1

)

9.78

10.74

13.00

18.60

9.36

10.23

11.94

15.09

8

12

16

20

16k

8k

4k

2k

Mean Absolute Error (MAE)

Number of Training Samples

L

1

RNC(

L

1

)

reported results are averaged over all types of corruptions. Fig. 4 illustrates the results, where the representation learned by RNC is more robust to unforeseen data corruptions, with consistently less performance degradation when corruption severity increases.

Resilience to Reduced Training Data. The availability of massive training datasets has played an important role in the success of modern deep learning. However, in many real-world scenarios, the cost and time involved in labeling large training sets make it infeasible to do so. As a result, there is a need to enhance model resilience to limited training data. To investigate this, we subsample IMDB-WIKI to generate training sets of varying sizes and compare the performance of RNC( L 1 ) and L 1 . As Fig. 5 confirms, RNC is more robust to reduced training data and displays less performance degradation as the number of training samples decreases.

Transfer Learning. We evaluate whether the learned representations by RNC are transferable across datasets. To do so, we first pre-train the feature encoder on a large dataset, and subsequently utilize either linear probing (fixed encoder) or fine-tuning to learn a predictor on a small dataset (which shares the same prediction task). We investigate two scenarios: ❶ Transferring from AgeDB which contains ∼ 12K samples to a subsampled IMDB-WIKI of 2K samples, and ❷ Transferring from another subsampled IMDB-WIKI of 32K samples to AgeDB . As shown in Table 4, RNC( L 1 ) exhibits consistent and superior performance compared to L 1 in both linear probing and fine-tuning settings for both of the aforementioned scenarios.

Robustness to Spurious Targets. We show that RNC is able to deal with spurious targets that arise in data [46], while existing regression learning methods often fail to learn generalizable features. Specifically, the SkyFinder dataset naturally has a spurious target: different webcams (with distinct locations). Therefore, we show in Fig. 6 the UMAP [30] visualization of the learned features obtained from both L 1 and RNC, using data from 10 webcams in SkyFinder . The first column of the figure is colored by the ground-truth target (temperature), while the second column is colored by the 10 different webcams. Our results demonstrate that the representation learned by L 1 is clustered by webcam, indicating that it is prone to capturing easy-to-learn features. In contrast, RNC learns the underlying continuous temperature information even in the presence of strong spurious tar-

L

1

RNC(

Colored by real targets

(temperature)

Colored by spurious targets

(webcam)

L

1

)

gets, confirming its ability to learn robust representations that generalize.

Generalization to Unseen Targets. In practical regression tasks, it is frequently encountered that some targets are unseen during training. To investigate zero-shot generalization to unseen targets, following [44], we curate two subsets of IMDB-WIKI that contain unseen targets during training, while keeping the test set uniformly distributed across the target range. Table 5 shows the label distributions, where regions of unseen targets are marked with pink shading and those of seen targets are marked with blue shading. The first training set has a bi-modal Gaussian

distribution, while the second one exhibits a tri-modal Gaussian distribution over the target space. The results confirm that RNC( L 1 ) outperforms L 1 by a larger margin on the unseen targets without sacrificing the performance on the seen targets.

5.3 Ablation Studies

Ablation on Number of Positives. Recall that in L RNC, all samples in the batch will be treated as the positive for each anchor. Here, we conduct an ablation study on only considering the first K closest samples to the anchor as positive. Table 6a shows the results on AgeDB for different K , where K = 511 represents the scenario where all samples are considered as positive (default batch size N = 256 ). The experiments reveal that larger values of K lead to better performance, which aligns with the intuition behind the design of L RNC: each contrastive term ensures that a group of orders related to the positive sample is maintained. Specifically, it guarantees that all samples that have a larger label distance from the anchor than the positive sample are farther from the anchor than the positive sample in the feature space. Only when all samples are treated as positive and their corresponding groups of orders are preserved can the order in the feature space be fully guaranteed.

Ablation on Similarity Measure. Table 6b shows the performance of using different feature similarity measures sim( · , · ) . Compared to cosine similarity, both the negative L 1 norm and L 2 norm produce significantly better results. This improvement can be attributed to the fact that cosine similarity only captures the directions of feature vectors, while the negative L 1 or L 2 norm takes both the direction and magnitude of the vectors into account. This finding highlights the potential differences between representation learning for classification and regression tasks, where the standard practice for classification is to use cosine similarity [4, 25], while our findings suggest the superiority of L 1 and L 2 norm for regression-based representation learning.

Ablation on Training Scheme. There are typically three schemes to train the encoder: ❶ Linear probing , which first trains the feature encoder using the representation learning loss, then fixes the encoder and trains a linear regressor on top of it using a regression loss; ❷ Fine-tuning . which first trains the feature encoder using the representation learning loss, then fine-tunes the entire model using a regression loss; and ❸ Regularization , which trains the entire model while jointly optimizing the representation learning & the regression loss. Table 6c shows the results for the three schemes using L RNC as the representation learning loss and L 1 as the regression loss. All three schemes can improve performance over using L 1 loss alone. Further, unlike classification problems where fine-tuning often delivers the best performance, freezing the feature encoder yields the best outcome in regression. This is because, in the case of regression, back-propagating the L 1 loss to the representation can disrupt the order in the embedding space learned by L RNC, leading to inferior performance.

5.4 Further Discussions

Does RNC rely on data augmentation for superior performance (Appendix G.3)? Table 10 confirms that RNC surpasses the baseline, with or without data augmentation. In fact, unlike typical (unsupervised) contrastive learning techniques that rely on data augmentation for distinguishing the augmented views, data augmentation is not essential for RNC. This is because RNC contrasts samples according to label distance rather than the identity. The role of data augmentation for RNC is similar to its role in typical regression learning, which is to enhance model generalization.

Does the benefit of RNC come from the two-stage training scheme (Appendix G.4)? We confirm in Table 11 that training a predictor on top of the representation learned by the competing regression baselines does not improve performance. In fact, it can even be detrimental to performance. This finding validates that the benefit of RNC is due to the proposed L RNC and not the training scheme. The generic regression losses are not designed for representation learning and are computed based on the final model predictions, which fails to guarantee the final learned representation.

Computational cost of RNC (Appendix G.5)? We verify in Table 12 that the training time of RNC is comparable to standard contrastive learning methods (e.g., SUPCON [25]), indicating that RNC offers similar training efficiency, while achieving notably better performance for regression tasks.

6 Broader Impacts and Limitations

Broader Impacts. We introduce a novel framework designed to enhance the performance of generic deep regression learning. We believe this will significantly benefit regression tasks across various real-world applications. Nonetheless, several potential risks warrant discussion. First, when the framework is employed to regress sensitive personal attributes such as intellectual capabilities, health status, or financial standing from human data, there's a danger it might reinforce or even introduce new forms of bias. Utilizing the method in these contexts could inadvertently justify discrimination or the negative targeting of specific groups. Second, in contrast to handcrafted features which are grounded in physical interpretations, the feature representations our framework learns can be opaque. This makes it difficult to understand and rationalize the model's predictions, particularly when trying to determine if any biases exist. Third, when our method is trained on datasets that do not have a balanced representation of minority groups, there's no assurance of its performance on these groups being reliable. It is essential to recognize that these ethical concerns apply to deep regression models at large, not solely our method. However, the continuous nature of our representation which facilitates interpolation and extrapolation might inadvertently make it more tempting to justify such unethical applications. Anyone seeking to implement or use our proposed method should be mindful of these concerns. Both our specific method and deep regression models, in general, should be used cautiously to avoid situations where their deployment might contribute to unethical outcomes or interpretations.

Limitations. Our proposed method presents some limitations. Firstly, the technique cannot discern spurious or incidental correlations between the input and the target within the dataset. As outlined in the Broader Impact section, this could result in incorrect conclusions potentially promoting discrimination or unjust treatment when utilized to deduce personal attributes. Future research should delve deeper into the ethical dimensions of this issue and explore strategies to ensure ethical regression learning. A second limitation is that our evaluation primarily focuses on general regression accuracy metrics (e.g., MAE) without considering potential disparities when evaluating specific subgroups (e.g., minority groups). Given that a regression model's performance can vary across demographic segments, subgroup analysis is an avenue that warrants exploration in subsequent studies. Lastly, our approach learns continuous representations by contrasting samples against one another based on their ranking in the target space, necessitating label information. To adapt it for representation learning with unlabeled data, our framework will need some modifications, which we reserve for future work.

7 Conclusion

Wepresent Rank-N-Contrast (RNC), a framework that learns continuous representations for regression by ranking samples according to their labels and then contrasting them against each other based on their relative rankings. Extensive experiments on different datasets over various real-world tasks verify the superior performance of RNC, highlighting its intriguing properties such as better data efficiency, robustness to corruptions and spurious targets, and generalization to unseen targets.

Acknowledgements. We are grateful to Hao He for his invaluable assistance with the theoretical analysis in the paper. We thank the members of the NETMIT group for their constructive feedback on the draft of this paper. We extend our appreciation to the anonymous reviewers for their insightful comments and suggestions that greatly helped in improving the quality of the paper. Lastly, we acknowledge the generous support from the GIST-MIT program, which funded this project.

References

Ahmed A Abdelrahman, Thorsten Hempel, Aly Khalifa, and Ayoub Al-Hamadi. L2cs-net: Fine-grained gaze estimation in unconstrained environments. arXiv preprint arXiv:2203.03339 , 2022.

Wenzhi Cao, Vahid Mirjalili, and Sebastian Raschka. Rank consistent ordinal regression for neural networks with application to age estimation. Pattern Recognition Letters , 140:325-331, 2020.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9650-9660, 2021.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 , 2020.

Weiwei Cheng, Eyke Hüllermeier, and Krzysztof J Dembczynski. Label ranking methods based on the plackett-luce model. In Proceedings of the 27th International Conference on Machine Learning (ICML-10) , pages 215-222, 2010.

Wei-Ta Chu, Kai-Chia Ho, and Ali Borji. Visual weather temperature prediction. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 234-241. IEEE, 2018.

Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. Advances in neural information processing systems , 33:87658775, 2020.

Benoit Dufumier, Pietro Gori, Julie Victor, Antoine Grigis, and Edouard Duchesnay. Conditional alignment and uniformity for contrastive learning with continuous proxy labels. arXiv preprint arXiv:2111.05643 , 2021.

Benoit Dufumier, Pietro Gori, Julie Victor, Antoine Grigis, Michele Wessa, Paolo Brambilla, Pauline Favre, Mircea Polosan, Colm Mcdonald, Camille Marie Piguet, et al. Contrastive learning with continuous proxy meta-data for 3d mri classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 58-68. Springer, 2021.

Denis A Engemann, Apolline Mellot, Richard Höchenberger, Hubert Banville, David Sabbagh, Lukas Gemein, Tonio Ball, and Alexandre Gramfort. A reusable benchmark of brain-age prediction from m/eeg resting-state signals. NeuroImage , 262:119521, 2022.

Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2002-2011, 2018.

Bin-Bin Gao, Chao Xing, Chen-Wei Xie, Jianxin Wu, and Xin Geng. Deep label distribution learning with label ambiguity. IEEE Transactions on Image Processing , 26(6):2825-2838, 2017.

Bin-Bin Gao, Hong-Yu Zhou, Jianxin Wu, and Xin Geng. Age estimation using expectation of label distribution learning. In IJCAI , pages 712-718, 2018.

Amir Globerson, Gal Chechik, Fernando Pereira, and Naftali Tishby. Euclidean embedding of co-occurrence data. In NeurIPS , 2004.

Jacob Goldberger, Geoffrey E Hinton, Sam Roweis, and Russ R Salakhutdinov. Neighbourhood components analysis. In NeurIPS , 2004.

Yu Gong, Greg Mori, and Frederick Tung. Ranksim: Ranking similarity regularization for deep imbalanced regression. In Proceedings of the 39th International Conference on Machine Learning , pages 7634-7649. PMLR, 2022.

Geoffrey Grimmett and David Stirzaker. Probability and random processes . Oxford university press, 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9729-9738, 2020.

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations , 2019.

Peter J Huber. Robust estimation of a location parameter. In Breakthroughs in statistics , pages 492-518. Springer, 1992.

Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference on Learning Representations , 2020.

Maurice G Kendall. A new measure of rank correlation. Biometrika , 30(1/2):81-93, 1938.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems , 33:18661-18673, 2020.

Shikun Li, Xiaobo Xia, Shiming Ge, and Tongliang Liu. Selective-supervised contrastive learning with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 316-325, 2022.

Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio S Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6918-6928, 2022.

Wanhua Li, Xiaoke Huang, Jiwen Lu, Jianjiang Feng, and Jie Zhou. Learning probabilistic ordinal embeddings for uncertainty-aware regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13896-13905, 2021.

Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 , 2016.

Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 , 2018.

Radu P Mihail, Scott Workman, Zach Bessinger, and Nathan Jacobs. Sky segmentation in the wild: An empirical study. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 1-6. IEEE, 2016.

Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop , volume 2, page 5, 2017.

Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. Ordinal regression with multiple output cnn for age estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4920-4928, 2016.

Iyad Obeid and Joseph Picone. The temple university hospital eeg data corpus. Frontiers in neuroscience , 10:196, 2016.

Hongyu Pan, Hu Han, Shiguang Shan, and Xilin Chen. Mean-variance loss for deep age estimation from a face. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5285-5294, 2018.

Rasmus Rothe, Radu Timofte, and Luc Van Gool. Dex: Deep expectation of apparent age from a single image. In Proceedings of the IEEE international conference on computer vision workshops , pages 10-15, 2015.

Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioral and neural analysis. arXiv preprint arXiv:2204.00673 , 2022.

Fabian Schrumpf, Patrick Frenzel, Christoph Aust, Georg Osterhoff, and Mirco Fuchs. Assessment of non-invasive blood pressure prediction from ppg and rppg signals using deep learning. Sensors , 21(18):6022, 2021.

Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

Xintong Shi, Wenzhi Cao, and Sebastian Raschka. Deep neural networks for rank-consistent ordinal regression based on conditional probabilities, 2021.

Charles Spearman. The proof and measurement of association between two things. The American journal of psychology , 100(3/4):441-471, 1987.

Yaoming Wang, Yangzhou Jiang, Jin Li, Bingbing Ni, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Teng Li. Contrastive regression for domain adaptation on gaze estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19376-19385, 2022.

Fen Xia, Tie-Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwise approach to learning to rank: theory and algorithm. In Proceedings of the 25th international conference on Machine learning , pages 1192-1199, 2008.

Yuzhe Yang, Kaiwen Zha, Yingcong Chen, Hao Wang, and Dina Katabi. Delving into deep imbalanced regression. In International Conference on Machine Learning , pages 11842-11851. PMLR, 2021.

Yuzhe Yang, Hao Wang, and Dina Katabi. On multi-domain long-tailed recognition, imbalanced domain generalization and beyond. In European Conference on Computer Vision (ECCV) , 2022.

Yuzhe Yang, Xin Liu, Jiang Wu, Silviu Borac, Dina Katabi, Ming-Zher Poh, and Daniel McDuff. Simper: Simple self-supervised learning of periodic targets. In International Conference on Learning Representations , 2023.

Huaxiu Yao, Yiping Wang, Linjun Zhang, James Y Zou, and Chelsea Finn. C-mixup: Improving generalization in regression. Advances in Neural Information Processing Systems , 35:33613376, 2022.

Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. Modeling discriminative representations for out-of-domain detection with supervised contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) , pages 870-878, 2021.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

Shihao Zhang, Linlin Yang, Michael Bi Mi, Xiaoxu Zheng, and Angela Yao. Improving deep regression with ordinal entropy. arXiv preprint arXiv:2301.08915 , 2023.

Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. Mpiigaze: Real-world dataset and deep appearance-based gaze estimation. IEEE transactions on pattern analysis and machine intelligence , 41(1):162-175, 2017.

Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. It's written all over your face: Full-face appearance-based gaze estimation. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on , pages 2299-2308. IEEE, 2017.

A Proofs

A.1 Proof of Theorem 1

Recall that in Eqn. (3) we defined L RNC := 1 2 N 2 N ∑ i =1 1 2 N -1 2 N ∑ j =1 , j = i -log exp( s i,j ) ∑ v k ∈S i,j exp( s i,k ) , where S i,j := { v k | k = i, d i,k ≥ d i,j } . We rewrite it as

̸

∀ i ∈ [2 N ] , m ∈ [ M i ] , from Jensen's Inequality we have

Thus, by plugging Eqn. (5) into Eqn. (4), we have

A.2 Proof of Theorem 2

We will show ∀ ϵ > 0 , there is a set of feature embeddings where

and γ := log 2 N min i ∈ [2 N ] ,m ∈ [ Mi ] n i,m ϵ , ∀ i ∈ [2 N ] , j, k ∈ [2 N ] \{ i } , such that L RNC < L ⋆ + ϵ .

For such a set of feature embeddings, ∀ i ∈ [2 N ] , m ∈ [ M i ] , j ∈ { j ∈ [2 N ] \{ i } | d i,j = D i,m } ,

̸

since s i,k = s i,j for all k such that d i,k = D i,m = d i,j , and

since s i,k -s i,j < -γ for all k such that d i,k > D i,m = d i,j and s i,k -s i,j = 0 for all k such that d i,k = D i,m = d i,j .

From Eqn. (4) we have

By plugging Eqn. (7) and Eqn. (8) into Eqn. (9) we have

A.3 Proof of Theorem 3

We will show ∀ 0 < δ < 1 , there is a such that when L We first show that if , when .

| s i,j -s i,k | < δ d i,j = d i,k ∀ i ∈ [2 N ] , j, k ∈ [2 N ] \{ i } L RNC < L ⋆ + ϵ From Eqn. (4) we have

, , by splitting out the maximum term and the minimum term s i,q i,m -s i,p i,m ∀ i ∈ [2 N ] , m ∈ [ M i ] we have

and

Then, from Jensen's inequality, we know

thus

By plugging Eqn. (13), Eqn. (14) and Eqn. (16) into Eqn. (12), we have

Let h ( θ ) := n i,m log(1+exp( η i,m )+( n i,m -2) θ ) -η i,m -( n i,m -2) log( θ ) . From derivative analysis we know h ( θ ) decreases monotonically when θ ∈ [ 1 , 1+exp( η i,m ) 2 ] and increases monotonically when θ ∈ [ 1+exp( η i,m ) 2 , exp( η i,m ) ] , thus

By plugging Eqn. (18) into Eqn. (17), we have

Since y ( x ) = 2 log 1+exp( x ) 2 -x increases monotonically when x > 0 , we have η i,m < δ . Hence, ∀ i ∈ [2 N ] , j, k ∈ [2 N ] \{ i } , if d i,j = d i,k = D i,m , | s i,j -s i,k | ≤ η i,m < δ . Next, we show s i,j > s i,k + δ if d i,j < d i,k when L RNC < L ⋆ + ϵ .

From Eqn. (4) we have

and combining it with Eqn. (5) we have

When L RNC < L ⋆ + ϵ , we already have | s i,l -s i,j | < δ , ∀ d i,l = d i,j , which derives s i,l -s i,j < δ and thus exp( s i,l -s i,j ) < exp( δ ) . By putting this into Eqn. (22), we have ∀ i ∈ [2 N ] , j ∈ [2 N ] \{ i } , k ∈ { k ∈ [2 N ] \{ i } | d i,j < d i,k } ,

∀ i ∈ [2 N ] , j ∈ [2 N ] \{ i } , k ∈ { k ∈ [2 N ] \{ i } | d i,j < d i,k } .

where r i,j ∈ [ M i ] is the index such that D i,r i,j = d i,j .

Further, given L RNC < L ⋆ + ϵ < L ⋆ + 1 2 N (2 N -1) log ( 1 + 1 n i,r i,j exp( δ + 1 δ ) ) , we have

which derives s i,j > s i,k + 1 δ , ∀ i ∈ [2 N ] , j ∈ [2 N ] \{ i } , k ∈ { k ∈ [2 N ] \{ i } | d i,j < d i,k } . Finally, ∀ i ∈ [2 N ] , j, k ∈ [2 N ] \{ i } , s i,j < s i,k -1 δ if d i,j > d i,k directly follows from s i,j > s i,k + 1 δ if d i,j < d i,k .

B Additional Theoretical Analysis

In this section, we present an analysis based on Rademacher Complexity [39] to substantiate that δ -ordered feature embedding results in a better generalization bound.

A regression learning task can be formulated as finding a hypothesis h to predict the target y from the input x . Suppose there are m data points in the training set S = { ( x k , y k ) } m k =1 . Let H 1 be the class of all possible functions from the input space to the target space.

If a δ -ordered feature embedding is guaranteed with an encoder f mapping x k to v k , the set of candidate hypotheses can be reduced to all ' δ -monotonic' functions h ( x ) = g ( f ( x )) , which satisfy d ( g ( v i ) , g ( v j )) < d ( g ( v i ) , g ( v k )) for s i,j > s i,k + 1 δ , d ( g ( v i ) , g ( v j )) = d ( g ( v i ) , g ( v k )) for | s i,j -s i,k | < δ , and d ( g ( v i ) , g ( v j )) > d ( g ( v i ) , g ( v k )) for s i,j < s i,k -1 δ for any i, j and k . We denote the class of all ' δ -monotonic' functions by H 2 . Note that both H 1 and H 2 contain the optimal hypothesis h ∗ , which satisfies ∀ x, y, h ∗ ( x ) = y .

Since H 2 ⊂ H 1 , we have A 2 ⊂ A 1 and c 2 ≤ c 1 , and from the monotonicity of Rademacher Complexity we have R ( A 2 ) ≤ R ( A 1 ) . Hence, with a δ -ordered feature embedding, the upper bound on the gap between the training error and the test error will be reduced, which leads to better regression performance.

Further, for a hypothesis set H i , let A i = { ( l (( x 1 , y 1 ); h ) , ..., l (( x m , y m ); h )) : h ∈ H i } be the loss set for each hypothesis in H i with respect to the training set S , where l is the loss function. Let c i be the upper bound of | l (( x, y ); h )) | for all x, y and h ∈ H i . We introduce the Rademacher Complexity [39] of A i , denoted as R ( A i ) . Then, the generalization bound based on Rademacher Complexity says that with a high probability (at least 1 -ϵ ), the gap between the empirical risk (i.e., training error) and the expected risk (i.e., test error) is upper bounded by 2 R ( A i ) + 4 c i √ 2 ln(4 /ϵ ) m .

Dataset

Original

Augmented

AgeDB

TUAB

……

21

channels

……

……

……

……

MPIIFaceGaze

SkyFinder

C Dataset Details

Five real-world datasets are used in the experiments:

AgeDB [32, 44] is a dataset for predicting age from face images. It contains 16,488 in-the-wild images of celebrities and the corresponding age labels. The age range is between 0 and 101. It is split into a 12,208-image training set, a 2140-image validation set, and a 2140-image test set.

TUAB [34, 11] is a dataset for brain-age estimation from EEG resting-state signals. The dataset comes from EEG exams at the Temple University Hospital in Philadelphia. Following Engemann et al. [11], we use only the non-pathological subjects, so that we may consider their chronological age as their brain-age label. The dataset includes 1,385 21-channel EEG signals sampled at 200Hz from individuals whose age ranges from 0 to 95. It is split into a 1,246-subject training set and a 139-subject test set.

MPIIFaceGaze [51, 52] is a dataset for gaze direction estimation from face images. It contains 213,659 face images collected from 15 participants during their natural everyday laptop use. We subsample and split it into a 33,000-image training set, a 6,000-image validation set, and a 6,000image test set with no overlapping participants. The gaze direction is described as a 2-dimensional vector with the pitch angle in the first dimension and the yaw angle in the second dimension. The range of the pitch angle is -40° to 10° and the range of the yaw angle is -45° to 45°.

SkyFinder [31, 7] is a dataset for temperature prediction from outdoor webcam images. It contains 35,417 images captured by 44 cameras around 11am on each day under a wide range of weather and illumination conditions. The temperature range is -20 ◦ C to 49 ◦ C . It is split into a 28,373-image training set, a 3,522-image validation set, and a 3,522-image test set.

IMDB-WIKI [36, 44] is a large dataset for predicting age from face images, which contains 523,051 celebrity images and the corresponding age labels. The age range is between 0 and 186 (some images are mislabeled). We use this dataset to test our method's resilience to reduced training data, performance on transfer learning, and ability to generalize to unseen targets. We subsample the dataset to create a variable size training set, and keep the validation set and test set unchanged with 11,022 images in each.

D Details of Data Augmentation

Table 7 shows examples of original and augmented data samples on each dataset. The data augmentations used on each dataset are listed below:

For AgeDB and SkyFinder , random crop and resize (with random horizontal flip), color distortions are used as data augmentation;

For TUAB , random crop is used as data augmentation;

For MPIIFaceGaze , random crop and resize (without random horizontal flip), color distortions are used as data augmentation.

E Details of Competing Methods

E.1 End-to-End Regression Methods

We compared with seven end-to-end regression methods:

L 1 , MSE and HUBER have the model directly predict the target value and train the model with an error-based loss function, where L 1 uses the mean absolute error, MSE uses the mean squared error and HUBER uses an MSE term when the error is below a threshold and an L 1 term otherwise.

DEX [36] and DLDL-V2 [14] divide the regression range of each label dimension into several bins and learn the probability distribution over the bins. DEX [36] optimizes a cross-entropy loss between the predicted distribution and the one-hot ground-truth labels, while DLDL-V2 [14] jointly optimizes a KL loss between the predicted distribution and a normal distribution centered at the ground-truth value, as well as an L 1 loss between the expectation of the predicted distribution and the ground-truth value. During inference, they output the expectation of the predicted distribution for each label dimension.

OR [33] and CORN [40] design multiple ordered thresholds for each label dimension, and learn a binary classifier for each threshold. OR [33] optimizes a binary cross-entropy loss for each binary classifier to learn whether the target value is larger than each threshold, while CORN [40] learns whether the target value is larger than each threshold conditioning on it is larger than the previous threshold. During inference, they aggregate all binary classification results to produce the final results.

For the classification-based baselines, the regression range is divided into small bins, and each bin is considered as a class. Details for each dataset are as follows:

For AgeDB , the age range is 0 ∼ 101 and the bin size is set to 1;

For MPIIFaceGaze , the target range is -40 ∼ 10 (°) for the pitch angle and -45 ∼ 45 (°) for the yaw angle, and the bin size is set to 0.5 for the pitch angle and is set to 1 for the yaw angle;

For TUAB , the brain-age range is 0 ∼ 95 and the bin size is set to 1;

For SkyFinder , the temperature range is -20 ∼ 49 (°C) and the bin size is set to 1.

E.2 State-of-the-art Representation Learning Methods

We compared with three state-of-the-art representation learning methods:

SIMCLR [4] is a contrastive learning method that learns representations by aligning positive pairs and repulsing negative pairs. Positive pairs are defined as different augmented views from the same data input, while negative pairs are defined as augmented views from different data inputs.

DINO [3] is a self-supervised representation learning method using self-distillation. It passes two different augmented views from the same data input to both the student and the teacher networks and maximizes the similarity between the output features of the student network and those of the teacher network. The gradients are propagated only through the student network and the teacher parameters are updated with an exponential moving average of the student parameters.

SUPCON [25] extends SIMCLR [4] to the fully-supervised setting, where positive pairs are defined as augmented data samples from the same class and negative pairs are defined as augmented data samples from different classes. To adapt SupCon to the regression task, we follow the standard routine of classification-based regression methods to divide the regression range into small bins and regard each bin as a class.

E.3 State-of-the-art Regression Learning Methods

We also compared with five state-of-the-art regression learning methods:

LDS+FDS [44] is the state-of-the-art method on the AgeDB dataset. It addresses the data imbalance issue by performing distribution smoothing for both labels and features.

L2CS-NET [1] is the state-of-the-art method on the MPIIFaceGaze dataset. It regresses each gaze angle separately and applies both a cross-entropy loss and an MSE loss on the predictions.

LDE [7] is the state-of-the-art method on the SkyFinder dataset. It converts temperature estimation to a classification task, and the class label is encoded by a Gaussian distribution centered at the ground-truth label.

RANKSIM [17] is a state-of-the-art regression method that proposes a regularization loss to match the sorted list of a given sample's neighbors in the feature space with the sorted list of its neighbors in the label space.

ORDINAL ENTROPY [50] is a state-of-the-art regression method that proposes a regularization loss to encourage higher-entropy feature spaces while maintaining ordinal relationships.

F Details of Experiment Settings

All experiments are trained using 8 NVIDIA TITAN RTX GPUs. We use the SGD optimizer and cosine learning rate annealing [29] for training. The batch size is set to 256. For one-stage methods and encoder training of two-stage methods, we select the best learning rates and weight decays for each dataset by grid search, with a grid of learning rates from { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 5 , 1 . 0 } and weight decays from { 10 -6 , 10 -5 , 10 -4 , 10 -3 } . For the predictor training of two-stage methods, we adopt the same search setting as above except for adding no weight decay to the search choices of weight decays. For temperature parameter τ , we search from { 0 . 1 , 0 . 2 , 0 . 5 , 1 . 0 , 2 . 0 , 5 . 0 } and select the best, which is 2 . 0 . We train all one-stage methods and the encoder of two-stage methods for 400 epochs, and the linear regressor of two-stage methods for 100 epochs.

G Additional Experiments and Analyses

G.1 Impact of Model Architectures

In the main paper, we use ResNet-18 as the default encoder backbone for three visual datasets ( AgeDB , MPIIFaceGaze and SkyFinder ). In this section, we study the impact of backbone architectures on the experiment results. As Table 8 reports, the results of using ResNet-50 as the encoder backbone are consistent with the results using ResNet-18 in Table 2, indicating that our method is compatible with different model architectures.

G.2 Standard Deviations of Results

In this section, we study the standard deviations of the best results on each dataset with 5 different random seeds. Table 9 shows their average prediction errors and standard deviations. These results are aligned with the results we reported in the main paper.

G.3 Impact of Data Augmentation

In this section, we study the impact of data augmentation on RNC. The following ablations are considered:

RNC( L 1 ) (without augmentation) : Data augmentation is removed and L RNC is computed over N feature embeddings.

RNC( L 1 ) (one-view augmentation) : Data augmentation is only performed once for each data point and L RNC is computed over N feature embeddings.

L 1 (without augmentation) : Data augmentation is removed and L 1 is computed over N samples.

RNC( L 1 ) (two-view augmentation) : Data augmentation is performed twice to create a two-view batch and L RNC is computed over 2 N feature embeddings.

L 1 (with augmentation) : Data augmentation is performed for each datapoint and L 1 is computed over N samples.

Table 10 reports the results on each dataset, which demonstrate that removing data augmentation will result in a decrease in performance for both L 1 and RNC( L 1 ), and our method outperforms the baseline with or without data augmentation.

Here we further discuss the different roles played by data augmentation in (unsupervised) contrastive learning methods, end-to-end regression learning methods, and our method respectively:

For (unsupervised) contrastive learning methods, data augmentation is essential for creating the pretext task, which is to distinguish whether the augmented views belong to the same identity or not. Therefore, removing the data augmentation will result in a complete collapse of the model.

For end-to-end regression learning methods, data augmentation helps the model generalize better and become more robust to unseen variations. However, without augmentation, the model can still perform reasonably well. The improvement brought by the augmentation is often correlated to how much the data augmentation can compensate for the gap between training data and testing data.

For RNC, data augmentation is also not necessary since our loss contrasts samples according to label distance rather than the identity. Thus, creating augmented views is not crucial to our method. The role of data augmentation for our method is similar to its role in regression learning methods, namely, improving the generalization ability. The experiment results also confirm this: removing data augmentation will lead to a similar drop in performance for both RNC( L 1 ) and L 1 .

G.4 Two-Stage Training Scheme for End-to-End Regression Methods

RNC employs a two-stage training scheme, where the encoder is trained with L RNC in the first stage and a predictor is trained using a regression loss on top of the encoder in the second stage. In Table 11, we also train a predictor using L 1 loss on top of the encoder learned by the competing end-to-end regression methods and compare with the performance of RNC( L 1 ) on the AgeDB dataset. The results show that the two-stage training scheme does not increase the performance of end-to-end regression methods. In fact, it can even be detrimental to performance. This is because these methods are not designed for representation learning, and their loss functions are calculated w.r.t. the final model predictions. As a result, there is no guarantee for the representation learned by these methods. This finding validates that the benefit of RNC is due to the proposed L RNC and not the training scheme.

G.5 Training Efficiency

We compute the average wall-clock running time (in seconds) per training epoch on 8 NVIDIA TITAN RTX GPUs for RNC and compare it with SUPCON [25] on all four datasets, as shown in Table 12. Results indicate that the training efficiency of RNC is comparable to SUPCON.

Peer reviews:
comments: SUMMARY:
The paper proposes the framework Rank-N-Contrast in order to learn regression-aware feature representations. Authors claim that this representation learning mechanism captures the continuous nature of sample orders and helps achieve better performance in downstream regression task.

STRENGTHS:
1. The paper provides both experimental results and theoretical proof to support their method. 
2. The paper includes relatively comprehensive ablation study and analysis section.

WEAKNESSES:
1. The paper is in lack high-level intuition and in-depth analysis of why the method work and only have low-level interpretation on the experimental results. Following are some example questions I hope authors could have addressed (and go one-step further than just stating their claims): Why preserving order in feature representation help to learn continuous nature? In fig. 3, why discernible pattern is better? What are the theorems trying to convey (my understanding is that feature embedding follows order inherited from the labels)?

2. Although authors have conducted experiments on 4 different datasets (where is the results of IMDB-WIKI b.t.w.?) and with 7 different regression losses, the datasets mostly have scalar responses (except for MPIFaceGaze) and seem repetitive to me. I think the experimental results will be more convincing if authors include experiments with higher-dimensional responses.

QUESTIONS:
See Weakness section.

LIMITATIONS:
I don't think authors have adequately addressed their limitations and weaknesses in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a deep learning method, Rank-N-Contrast, for regression tasks. This method aims to capture continuity in data, something existing methods struggle with. The authors define a concept of $\\delta$-ordered feature embedding and show theoretically that if Rank-N-Contrast loss is minimized, feature embeddings will be $\\delta$-ordered. The proposed method excels in several regression tasks.\n\nSTRENGTHS:\nThis paper provides a novel perspective to regression tasks using deep learning models, a well-studied and widely-acknowledged problem setting. The paper's unique approach lies in its application of contrastive learning methods, which is a departure from conventional techniques that alter the loss function or incorporate specific regularization.\n\nThe paper sets itself apart by offering an extensive comparison to existing methodologies, using both theoretical and empirical results from regression and representation learning tasks. This comprehensive approach underlines the proposed method's potential and relevance.\n\nThe ubiquity of regression tasks in the realm of deep learning models implies that this paper's approach could contribute meaningfully to the field. Furthermore, the paper is well-composed, making it easy for readers to grasp the fundamental concept of the study.\n\nWEAKNESSES:\nThe paper positions \"regression-aware representation\" as vital, but a slight disparity between the theoretical and experimental results is apparent.\n\nTheoretically, it suggests that a deep learning model can fulfill \"regression-aware embedding\" characteristics by minimizing the RNC loss appropriately. Experimentally, the paper exhibits consistent performance enhancements over existing approaches in both regression tasks and representation learning frameworks. Furthermore, it confirms that the implementation of RNC provides resilience against data corruption, improves performance with limited training data, and boosts transfer learning.\n\nWhile the paper implies the potential to secure $\\delta$-ordered feature embeddings by reducing RNC loss, and evidences steady performance improvement in real-world data contexts, the precise relationship between the continual characteristics of the \"regression-aware representations\" and the observed performance improvements is not fully clear yet. Additional discussion around this subject could offer more insights and, thereby, enrich our comprehension.\n\nQUESTIONS:\n1. **About Embedding Dimensions:** The learned representations in Fig. 1 are impressive, but the embedding dimension isn't specified. Were these embeddings trained directly in two dimensions, or were they first embedded in a high-dimensional space and then visualized with methods like UMAP? If UMAP was used, are the apparent clustering and lack of continuity in L1 and SupCon embeddings an artifact of the visualization process, or do these characteristics persist in the high-dimensional embeddings?\n2. **About the Construction of RNC loss:** In l.104, just before Equation 1, it's stated that the normalized likelihood of $v_j$ \"can be written as …\" Is Equation 1 derived from some definition or axiom? At least the normalized likelihood is “defined” in the reference [41].\n3. **About Trainability:** Theorem 3 claims that if RNC loss is sufficiently minimized, a desirable representation can be obtained. But how much can RNC loss be practically minimized? Can any insights be provided from a theoretical standpoint on the trainability of the proposed loss?\n\nLIMITATIONS:\nOne potential limitation is the difference in the number of training epochs between the one-stage and two-stage methods. According to Appendix l.629, the one-stage methods were trained for 400 epochs, whereas the two-stage methods, including the proposed approach, were trained for a total of 500 epochs. This longer training period for the proposed method could potentially be contributing to the improvements observed in table 2. A further analysis or an ablation study might help to confirm whether these enhancements are genuinely due to the method itself and not merely a consequence of the extended training period.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a new loss for representation learning in regression, RNC (Rank-n-contrast).  RNC can be seen as the SupCon loss adapted to the regression setting, where the labels given are not hard class labels but rather continuous regression labels.  In SupCon the negatives for each example are members of other classes and the positives come from members of the same class.  In RNC positive pairs are formed with every example to the anchor, and the negatives are those examples with larger label distances to the anchor than the positives.  By doing this they are able to train a representation for the data that is continuous in nature, which is hypothesized to represent the data better.\n\nSTRENGTHS:\n1.   I believe the method is sufficiently original as a nontrivial adaptation of the SupCon methodology to the regression setting.  The designation of what the positives and negatives are for the RNC contrastive loss are well motivated and I can see why the loss would give the kind of continuous representations that it does.\n\n2.  The method seems to be pretty simple to implement.  I particularly like that the RNC loss is the only loss the authors pretrained with and it wasn't some highly tuned composition of many different losses.  It gives me more confidence that the RNC loss is providing the gain in performance.\n\n3.  The theory is simple and well-motivated in that it proves the loss is doing what the authors are claiming it does.  The theory combined with the visualization of the representations give me confidence that the representations are in fact ordered continuously.\n\n4.  The experimental evaluation is fairly thorough.  The standard questions are answered, i.e. standard deviation error bars, how does it perform against other pretraining tasks, how does a plain two-stage training compare, whether augmentation is important, etc.\n\nWEAKNESSES:\n1.  I think the experimental evaluation can be improved to improve the generality of the method.  Currently the datasets that are evaluated are on computer vision regression tasks, which confines the evaluation to just ResNet-based models when we compare against the different losses and training methods in the paper.  In particular, one area where regression tasks are popular is the tabular setting.  I would be very interested to see a comparison between RNC and tree-based methods on popular tabular methods.  For examples of models and datasets to compare on the authors can check https://arxiv.org/abs/2106.11959 (not my paper btw)\n\n2.  It is great to see that the method is able to consistently perform well when compared against other baselines.  But as someone who is not familiar with these particular regression tasks I am not sure whether the magnitude of improvement is substantial.  Could the authors help me understand the scale of improvement?\n\n3.  Returning to my point on how the evaluation is restricted to ResNet-based architectures, I wonder if given that ResNet was designed to classify images that it has some inductive bias on clustering sets of images based on semantic similarities, which is what causes the disjointed representations in Figure 1 (left).  Expanding past the computer vision domain would be very valuable in making the contribution more impactful and general.\n\nQUESTIONS:\nQuestions and possible directions of improvement are listed under \"Weaknesses\".\n\nLIMITATIONS:\nI believe this was sufficient.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors discuss the benefits of contrastive learning for learning structured representations in a regression setting. While contrastive losses are typically formulated in terms of “similar” and “dissimilar” examples, the authors make use of the extra information conveyed by the continuous target label. They show how adding the RnC contrastive term to training is beneficial for a suite of high-dimensional regression tasks.\n\nSTRENGTHS:\nThis is a technically strong paper that, in my opinion, makes a contribution towards the important and understudied problem of representation learning for regression. The qualitative results [Fig 1] are compelling. The proposed method is intuitive and the paper is well written. The theoretical analysis and extensive empirical results suggest to me that the proposed method will be of interest to NeurIPS attendees. I was especially interested in the suggestion that contrastive training yields more robust representations [lines 259--279], although I think further work will be needed in the future to validate this finding in more sophisticated OOD generalization settings.\n\nWEAKNESSES:\nTo my knowledge the technical contributions here are sound. However, after reading the paper, I found myself a bit concerned about how neural net-based regression models are going to be used in the near future. And I think the paper would benefit from a head-on discussion of the potential ethical issues at play. \n\nRegression is notoriously difficult for neural networks (for example, many baseline methods convert the regression into a classification problem). A technological advance here could open up entirely new application areas. However, the use of datasets involving human faces suggest to me that some of these new applications could bring up new ethical concerns as well. While Predicting age and gaze direction from face images (as done in this paper) might seem reasonable, there are existing critiques in the literature [e.g. https://ir.lawnet.fordham.edu/cgi/viewcontent.cgi?article=1804&context=iplj, https://medium.com/@blaisea/physiognomys-new-clothes-f2d4b59fdd6a, https://dl.acm.org/doi/pdf/10.1145/3375627.3375820] arguing that predicting other target variables, such as a continuous proxy for emotion, are ethically fraught. In the authors’ opinion, are there application areas where RnC should *not* be applied?\n\nI think the paper would benefit from a discussion of these issues. While the lack of such discussion (the broader impact section is, frankly, rather boilerplate) represents a weakness in my view, it probably should not stand in the way of the paper’s acceptance, since it is more of a critique of the whole subfield rather than this specific paper. However, I am going to request an ethics review in order to get a second opinion about this.\n\nThe related works are generally well covered. One exception is the recent C-mixup paper [https://arxiv.org/abs/2210.05775], which also uses label similarity in regression to group “similar” data points. However their approach uses these similarities for data augmentation rather than contrastive learning. A discussion of how the two approaches differ would benefit the reader.\n\nQUESTIONS:\n* How is the temperature parameter \\tau tuned? [line 106]\n\nLIMITATIONS:\nLimitations and broader impacts are discussed [Sec. 5.4, App. H]. However, given that images of faces are used frequently in the experiments, I think that a more complete discussion about broader impacts, especially as it relates to task definition for regression, should be included (see “Weaknesses” above and my request for an ethics review below).", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors proposed a novel framework that learns continuous representations for regression problems, by contrasting samples against each other based on the rankings induced by the target values. The proposed method is evaluated on several regression tasks and the results show that the proposed method can achieve competitive performance compared to the state-of-the-art methods.\n\nSTRENGTHS:\n1. This paper tackles a very interesting problem of representation fragmentation in deep regression models. The proposed method is simple and very effective to learn continuous representations that fit the regression task. It is very interesting to see that the traditional methods learn representations clustered based on spurious targets (i.e., camera location in the SkyFinder dataset) while the proposed method learns a nice continuous representation that captures the target values.\n\n2. The authors present a theoretical analysis of the proposed loss function and show that it can learn delta-ordered feature embeddings when sufficiently trained.\n\n3. The proposed loss function is conceptually simple and easy to implement. The proposed method is thoroughly evaluated on several regression tasks and the results show that the proposed method can achieve competitive performance compared to the state-of-the-art methods. The authors further conducted ablation studies to show the effectiveness of the proposed method under data corruption, unseen or spurious targets.\n\nWEAKNESSES:\n1. While it is nice to see the proposed method leads to delta-ordered feature embeddings. It would be nice if the authors can further theoretically connect the delta-ordered feature embeddings to the final performance of the regression task. This would help to justify that the delta-ordered feature embeddings are indeed useful for the regression task.\n\n2. For the remark for Thm 3, to achieve delta-ordered features for the entire dataset, is it that we need to optimize all batches to achieve low enough loss? Is there any guarantee/insight that there is a feasible solution that can achieve low enough loss for all batches?\n\n3. It seems the IMDB-WIKI performance is missing in Table 2.\n\nQUESTIONS:\nSee the comments in \"weaknesses\".\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-13"}