Supervised Contrastive Regression
Authors: Kaiwen Zha
Peng Cao
Yuzhe Yang
Dina Katabi
Venue: Submitted to ICLR 2023 (2023)
Keywords: regression
regression learning

Abstract: Deep regression models typically learn in an end-to-end fashion and do not explicitly try to learn a regression-aware representation. Their representations tend to be fragmented and fail to capture the continuous nature of regression tasks. In this paper, we propose Supervised Contrastive Regression (SupCR), a framework that learns a regression-aware representation by contrasting samples against each other based on their target distance. SupCR is orthogonal to existing regression models, and can be used in combination with such models to improve performance. Extensive experiments using five real-world regression datasets that span computer vision, human-computer interaction, and healthcare show that using SupCR achieves the state-of-the-art performance and consistently improves prior regression baselines on all datasets, tasks, and input modalities. SupCR also improves robustness to data corruptions, resilience to reduced training data, performance on transfer learning, and generalization to unseen targets.

Under review as a conference paper at ICLR 2023
SUPERVISED CONTRASTIVE REGRESSION WITH SAMPLE RANKING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep regression models typically learn in end-to-end fashion and do not ex- plicitly try to learn a regression-awan representation. Their representations tend be fragmented and fail to capture the continuous nature of regression tasks. In this paper, we propose Supervised Contrastive Regression (SupCR), a framework that learns a regression-awe aware representation by contrasting samples against each other based on their target distance. SupCR is orthogonal to existing regression models, and can be used in combination with such models improve performance. Extensive experiments using five real-world regression datasets that span com- puter vision, human-computer interaction, and healthcare show that using SupCR achieves the state-of-the-an performance and consistently improves prior regres- sion baselines on all datasets, tasks, and input modalities. SupCR also improves robustness data corruptions, resilience to reduced training data, performance on transfer learning, and generalization to unseen targets.
INTRODUCTION
Regression problems are ubiquitous and fundamental in the real world. They include estimating age from human appearance (Rothe et al., 2015), predicting health scores from physiological signals (En- gemann al., 2022), and detecting gaze directions from webcam images (Zhang et al., 2017b). Since regression targets are continuous, the most widely used approach to train a regression model is to have the model predict the target value, and use the distance (e.g., L1 or L2 distance) between the prediction and the ground-truth target as the loss function (Zhang et al., 20;7;;b; Schrumpf et al., 2021; Engemann et al., 2022). There are also works that control the relationship between predictions and targets by converting the regression task into a classification task and training the model with the cross-cntropy loss (Rothe et al., 2015; Niu et al., 2016; Shi et al., 2021).
However, all previous methods focus on imposing constraints on the final predictions in end-to-end fashion, but do not explicitly consider the representations learned by the model. Their representations tend to be fragmented and fail to capture the continuous relationships underlying regression tasks. For example, Figure (a) shows the representation learned by the L1 loss in the task of predicting weather temperature from webcam outdoor images (Chu et al., 2018), where the images outdoor webcams at different locations. The representation learned by the LI model does not exhibit the continuous ground-trute temperatures; rather it is grouped by different webcams manner. Such unordered and fragmented representation is sub-optimal for the regression task and can even hamper performance since it contains distracting information (e.g.. the capturing webcam).
captured by 44
fragmented
While there is a rich literature on representation learning, past methods are focused on classification problems. particular, contrastive learning (Chen al., 2020a: He al., 2020: Chen et al., 2020b) and supervised contrastive learning (SupCon) (Khosla et al., 2020) have been proven highly effective in representation learning. However, as shown in Figure ((b), which plots the representation learned by SupCon for the visual temperature prediction task mentioned above, such method produces sub-optimal representation for regression problems because it ignores the continuous order between the samples in a regression task. Besides, there are several recent works (Wang al., 2022; Dufumier et al., 202 la;;; Schneider et al., 2022) adopting contrastive learning in the context of continuous labels, but they not doing regression learning tasks.
In this paper, we introduce Supervised Contrastive Regression (SupCR), a new framework for deep regression learning, where first learn a representation that ensures the distances in the embedding

---

Under review as a conference paper at ICLR 2023
10
-
(a)
(b) SupCon
SupCR
Figure UMAP (McInnes al., 2018) visualization of the representation learned by LI, Sup- Con (Khosla et al., 2020), and the proposed SupCR for the task of predicting the temperature from webcam outdoor images (Chu et al., 2018). The representation of each image is visualized as a dot and the color indicates the ground-truth temperature. SupCR can learn representation that captures the intrinsic ordered relationships between the samples while L.1 and SupCon fail to do so.
space are ordered according to the target values. We then this representation predict the targets. To learn such regression-ane ewwre representation, we contrast samples against each other based on their label/target distances. Our method explicitly leverages the ordered relationships between the samples to optimize the representation for the downstream regression task. As shown in Figure (cc), SupCR leads to representation that captures the intrinsic ordered relationships between the samples. Moreover, our framework is orthogonal to existing regression methods as one can use any kind of regression method to map our learned representation to the prediction values.
Extensive experiments using five real-world regression datasets that span computer vision, human- computer interaction, and healthcare verify the superior performance of our framework. The results show that SupCR achieves the state-of of- totte--of-heattt the-art performance and consistently improves the performance of previous regression methods by 8.7 on average across all datasets, tasks, and input modalities. Furthermore, experimental results also show that SupCR delivers multiple desirable properties: 1) improved robustness to data corruptions; 2) enhanced resilience to reduced training data; 3) higher performance transfer learning; and 4) better generalization unseen targets.
RELATED WORK
Regression Learning. Deep learning has achieved a great success in addressing regression tasks (Rothe al., 2015; Zhang et al., 20;7;;;; Schrumpf et al., 2021; Engemann et al., 2022). The most straightforward and widely used way to train a regression model is to have the model predict the target value, and the distance between the prediction and the ground-truth target the loss function. Common distance metrics used as the loss function in regression tasks are the LI loss, the MSE loss and the Huber loss (Huber, 1992).
Past work has also proposed variants to the basic methods described above. One branch of prior works (Rothe et al., 2015; Gao al., 2017: 2018; Pan al., 2018) divide the regression range into small bins to convert the problem into classification problem, and use classification loss, e.g., the cross-entropy loss, to train the model. Another branch of prior work (Niu et al., 2016; Fu al., 2018; Cao et al., 2020; Shi et al., 2021) casts regression as an ordinal classification problem. They design multiple ordered thresholds and use multiple binary classifiers, one for each threshold, to learn whether the target larger than each threshold.
Past work however focuses on forcing the final predictions of the model be close to the target and does so in an end-to-end manner. In contrast, this paper focuses ordering the samples in the embedding space in accordance with the label order, and it does so by first learning a representation that contrasts samples against each other, and then learning predictor map the representation to the prediction values. Therefore, our method is orthogonal to prior methods for regression learning as those prior methods be applied to training our predictor and learning the final predictions with the help of our ordered representation.
Contrastive Learning. Contrastive learning, which aligns positive pairs and repulses negative pairs in the representation space, has demonstrated improved performance for self-supervised learn
2

---

Under review as a conference paper at ICLR 2023
ing (Chen et al., 2020a; He et al., 2020; Chen et al., 2020b; Chuang et al., 2020). Its supervised version, named supervised contrastive learning (SupCon) (Khosla et al., 2020), which defines samples from the same class positive pairs and samples from different classes negative pairs, was also shown to outperform the cross-entropy loss on image classification tasks (Khosla et al., 2020). is also heneficial for learning in challenging settings such noisy labels (Li al., 2022a), long-tailed classification (Kang et al., 2020; Li et al., 2022b) and out-of-domain detection (Zeng et al., 2021).
couple of recent papers have used the concept of "contrastive" or contrastive learning in the context of continuous labels. In particular, Yu et al. (2021) learns a model for action quality assessment by regressing the relative scores between two input videos. This work differs from ours that does not use the standard contrastive learning framework. Wang et al. (2022) proposes to improve domain adaptation for gaze estimation by adding a contrastive loss term to the I.1 loss. They show that their approach is beneficial for adapting a gaze estimation model from one dataset (i.e., one domain) to another, but the approach produces no benefits and even reduces performance for the source dataset. In contrast, our approach improves performance for the source dataset opposed to being specific to domain adaptation. Besides, Dufumier et al. (2021a;b) use a contrastive loss re- weighted by continuous meta-data for classification. Schneider et al. (2022) learns lin-dimensional and interpretable embeddings to encode behavioral and neural data using a generalized contrastive loss which samples positives and negatives according continuous behavior or time labels. However, the continuous meta-data in (Dufumier et al., 202 labb) and the continuous labels in (Schneider et al., 2022) are not the targets in their tasks and they are not working on regression problems.
METHOD
For a regression task, we aim to train a neural network composed of a feature encoder f()) : X + Rdd and predictor p(.) Rdd. + Rd to predict the target y E Rd based on the input data Z E X.
Given an input batch of data, similar to contrastive learning methods (Chen et al., 2020a; Khosla et al., 2020), we first apply data augmentation twice obtain two views of the batch. The two views are fed into the encoder f(-) to obtain a d, dimeensiooal feature embedding for each augmented input data. Our supervised contrastive regression loss, LsupCR. is computed on the feature embeddings. To use the learned representation for regression, we freeze the encoder f(-) and train the predictor on top of it using a regression loss (e.g., LI loss).
3.1 SUPERVISED CONTRASTIVE REGRESSION LOSS
We would like our loss function to ensure distances in the embedding space ordered according to distances in the label space. But, how can we design a contrastive loss that delivers this property? Below, we first define our loss function, then explain in the context of positive and negative contrastive samples, and finally provide theoretical analysis to prove that as our loss approaches its minimum, the learned representation will be ordered according to label distances.
For a positive integer I, let [4] denote the set (1.2......4 Given a randomly sampled batch of N input and label pairs {<rn yn)] we apply data augmentation the batch and obtain a two-view batch {i (t. heejanj- where T2n. = t(xn),. x2n-1 +((x,) (t and are independently sampled data augmentations operations) and G2n J2n-1 = yn. Vn E [N]. The augmented batch fed into the encoder f()) to obtain vi = f()) E RRd V/ E [2.]. Our supervised contrastive regression loss is then computed over the feature embeddings {17 ) as:
2
2N
exp(sim(u, v,)/T)
LSapCR 2N
Σ
log
(I)
2N
E 1 (...0.222200. (2)d( (s,(sim(v,, Ux)/T)
j=1, ##i
where sim(..)) is the function measuring the similarity between two feature embeddings (c.g.. negative L. norm), -) is the function measuring the distance between two labels (e.g., L1 distance), 11 E {0, 13 is an indicator function evaluating to iff the conditions in the square brackets satisfied, and is the temperature parameter.
At a high level, our loss can be explained in the context of positive/negative pairs, commonly used in contrastive learning. Contrastive learning and SupCon focus classification. Hence, positive
3

---

Under review as a conference paper at ICLR 2023
Anchor
Positive Pair
Corresponding Negative Pair(s)
]
20 30
20
Age:
Age: 20
Age:
Age: 70
A Batch of Samples
20 0
20 70
(a)
(b)
Figure 2: Illustration of L (a) An example batch of input data and their age labels. (b) Two example positive pairs and the corresponding negative pair(s) when the anchor is the 20-year old man (shown in gray shading). When the 30-year old man creates positíve pair with the anchor, their label distance is 10, so the corresponding negative samples are the O-year old baby and the 70-year old man, whose label distances to the anchor are 20 and 50 respectively. When the O-year old baby creates a positive pair with the anchor, their label distance is 20. Only one sample in the batch has a larger label distance the anchor, namely the 70-year old man, and it acts negative sample.
samples are those that come from the same class the same input image, and all other samples are negative samples. In regression problems, there are no classes but continuous labels. Any two samples can be thought of positive pair or a negative pair, depending the inherent continuity underlying the labels, we define positive and negative samples in a relative way, based on their label distance to the anchor. In particular, we rank samples with respect their label distance to the anchor. We then contrast the anchor to the sample in the batch that is closest in the label space, which will be treated the positive sample. All other samples are farther from the anchor and should be negative samples. We also contrast the anchor to the second closest sample in the batch. In this case, only samples in the batch that are farther from the positive sample (i.e.. samples whose ranks is three or higher) can act as negative samples. We continue this process for higher rank samples (the third closest, fourth closest, etc.), and all anchors in a batch.
context. To exploit
Figure 2(a) shows an example batch, and Figure 2(b) shows two positive pairs and their corresponding negative pair(s). Specifically, for any anchor sample i, any other sample j in the same batch can be used to create positive pair with corresponding negative samples set to all samples in the batch whose labels differ from i's label by more than the label j. In other words, LsppCR aims to force the feature similarity between i and j greater than the feature similarity between i and any other sample in the batch if the label distance between and & is greater than and j. Therefore, optimizing LsSuCRR will make the feature embedding ordered according to the order in the label space. The design of LsupCR sufficiently leverages the ordered relationships between samples reflected in the continuous label space of regression problems, which is also the intrinsic difference between regression problems and classification problems.
3.2 THEORETICAL ANALYSIS
In this section, we analytically show that optimizing Lupcr will make the feature embedding ordered according to the order in the label space. To show this, we first derive lower bound of LSupCR. and show LSppCR can be arbitrarily close to it. Next, we formalize the concept of 8-ordering which refers to the feature embeddings being ordered according to the order in the label space, and show that as CSupCR gets sufficiently close to the lower bound, the feature embeddings will be d-ordered. All proofs are in Appendix A.
Notations. Let 8i, := sim (u,, 0;)/T, Vi,j E [2N] dr, == d ÙT). Vi.j E [2N], DA.1 < Di.2 < ... < Di.M. be the sorted label distances starting from the i-th sample, i.e., cor(ffi, E [2N])( Vi E [2N], and ni,m := kii | di, = Dim, jE [2N](()) be the number of samples whose distance from the i-th sample equals Di,, Vie [2N], E [M.].
First, we show that L == 2N(2N-1) 2NN M.,, ni.m log ni.m is a lower bound of LuppRR.
Theorem 1 (Lower bound of LsupCk). L" is lower bound of LSupcks i.e., LSupCR L*.
Next, we show that SapRR can be arbitrarily close to its lower bound T
4

---

Under review as a conference paper at ICLR 2023
Theorem 2 (Lower bound tightness). For any >> 0, there exists set of feature embeddings such that SuppR < L +
Then, we define property called ò-ordered for the feature embeddings (Vi Definition 1 (ő-ordered feature embedddings). For any 0 < 6 < 1, the feature embeddings {1Tp) are d-ordered if E [2N], j, E [2N]][t
Sú.j > 8i,6 + if d
i di,
8
18i.
8i.k|
< ô if di.j di
1
SLj < $1,6
if ds,j > d,,t
8
It says that d-ordered set of feature embeddings satisfies the following properties: First, for all and such that dı,5 = di, the gap between St.j and 8t,k should be smaller than 8; Second, for all j and such that dr,j < di, St,j should be larger than si,k by at least Notice that 1 > which means the feature similarity gap between sample pair with different label distance to the anchor is always larger than the feature similarity gap between a sample pair with equal label distance to the anchor.
Finally, show that for any 0 < 8 < 1, when LSupER is close enough its lower bound L*, the feature embeddings will be 5-ordered.
Theorem 3 (Main theorem). For any 0 < 6 < 1, there exist > 0, such that if LSupeR < L + e, then the feature embeddings are d-ordered.
4 EXPERIMENTS
In this section, we evaluate our proposed method empirically. We first benchmark our method and compare with the state-of-the-ard regression baselines. Then, we evaluate the desirable properties of our learned representations, including the robustness to data corruptions, the resilience to reduced training data, the performance on transfer learning, and the generalization ability to unseen targets. Finally, analyze and evaluate the design variants for method. Refer to Appendix E for additional experiments and analysis.
Datasets. We benchmark our method on five regression datasets for common real-world tasks in computer vision, human-computer interaction, and healthcare. Ethics statements of the datasets are included in Appendix B.
1) AgeDB (Moschoglou et al., 2017) is used for predicting age from face images. It contains 16,488 in-the-wild images of celebrities and the corresponding age labels. The age range is between 0 and 101. It is split into a 12,208-image training set, a 21 2140-image validation set and a 21 2140-image test set.
2) TUAB (Obeid Picone, 2016; Engemann al., 2022) is used for brain-age estimation from EEG resting-state signals. The dataset comes from EEG exams at the Temple University Hospital in Philadelphia. Following Engemann et al. (2022), we use only the non-pathological subjects, so that we may consider their chronological age as their brain-age label. The dataset includes 1,385 21 21-channel EEG signals sampled at 200Hz from individuals whose age ranges from 95. is split into a 1,246 2u4ject training set and a 139-subject test set.
3) MMIIIPaceGaaa (Zhang et al., 2017a;b) is used for gaze direction estimation from face images. contains 213,659 face images collected from 15 participants during their natural everyday laptop use. We subsample and split into 33,000-image training set, 6,000-image validation set and a 6000- image test set with no overlapping participants. The gaze direction is described vector with the pitch angle the first dimension and the yaw angle in the second dimension. The range of the pitch angle is -40 to 10 and the range of the yaw angle is -450 45
2-dimensional
4) SkyFinder (Mihail et al., 2016; Chu et al., 2018) is used for temperature prediction from outdoor webcam images. It contains 35,417 images captured by 44 cameras around I Iam each day under a wide range of weather and illumination conditions. The temperature range is -20 % to 49 CC is split into 28,373-image training set, 3,522-image validation set and 3,522-image test set.
5

---

Under review as a conference paper at ICLR 2023
Table 1: Evaluation results on AgeDB.
Method
MAE
R-
Method
MAE
R
L1 MSE HUBER
6.63
0.828
SUPCR(LI)
6.14 (10.49)
0.850 (-00023)
6.57
0.828
SUPCR(MSE)
6.19 (40.38)
0.849 (-00021)
6.54
0.828
SUPCR(HUBER)
6.15 (+0.39)
0.850 (-0.022)
DEX (Rothe et al., 2015)
7.29
0.787
SUPCR(DEX)
6.43 (+0.56)
0.836 (+0.44)
DLDL-V2 (Gao al., 2018)
6.60
0.827
(LLLR(D
6.32 (+0.28)
0.844 (+0.017)
OR (Niu et al., 2016)
6,40
0.830
SUPCR(OR)
6.34 (+0.06)
0.843 (+0.013)
CORN (Shi et al., 2021)
6.72
0.811
SUPCR(CORN)
6.44 (+0.28)
0,838 (-0.027)
Table 2: Evaluation results on TUAB.
Method
MAE
R²T
Method
MAE
R²7
LI MSE HUBER
7.96
0.599
SUPCR(LI)
6.97 (+0.99)
0.697 (-0.098
8.06
0.585
SUPCR(MSE)
7.05 (+1.01)
0.692
(+0.107)
7.59
0.637
(UPRR((URER)
6.99 (+0.60)
0.696
(-0.859)
DEX (Rothe et al., 2015)
8.01
0.537
SUPCR(DEX)
7.23 (40.78)
0.646 (-0.109)
DLDL-V2 (Gao et al., 2018)
7.91
0.560
(DLLL())
6.91 (+1.00)
0.697 (-0.137)
OR (Niu et al., 2016)
7.36
0.646
SUPCR(OR)
7.01 (+0.35)
0.688 (-0.042)
CORN (Shi et al., 2021)
8.11
0.597
SUPCR(CORN)
7.22 (+0.89)
0.663 (-0.066)
Table 3: Evaluation results on MPIIFaceGaze.
Method
Angular+
22
Method
Angulart
R21
5.97
0.744
SUPCR(LI)
5.27 (-0.7))
0.815 (+0.071)
MSE
6.02
0.747
SUPCR(MSE)
5.35 (-0.67)
0.802 (+0.055)
HUBER
6.34
0.709
SUPER(HUDER)
5.15 (+1.19)
0.830 (+0.121)
DEX (Rothe et al., 2015)
5.72
0.776
SUPCR(DEX)
5.14 (-0.50)
0.805 (+0.029)
DLDL-v2 (Gao et al., 2018)
5.47
0.799
SUPCR(DLDL-V2)
5.16 (-0.31)
0,802 (+0.003)
OR (Niu al., 2016)
5.86
0.770
SUPCR(OR)
5.13 (+0.73)
0.825 (+0.055)
CORN (Shi et al., 2021)
5.88
0.762
SUPCR(CORN)
5.18 (+0.70)
0.820 (+0.05))
Table 4: Evaluation results on SkyFinder.
Method
MAE
R
Method
MAE
R²
L1
2.95
0.860
IUPCR(LI)
2.86 (s0.09)
0.869 (-0.009)
MSE HUBER
3.08
0.851
DUPCR(MSB)
2.86 (10.22)
0.869
(-0.018)
2.92
0.860
SUPCR
2,86 (10.06)
0.869
(-0.009)
DEX (Rothe al., 2015)
3.58
0.778
SUPCR
2.88 (+0.70)
0.865
(+0.887)
DLDL-v2 (Gao al.. 2018)
2.99
0.856
(ULLR(DLDL-V))
2.85 (+0.14)
0.869 (+0.013)
OR (Niu el al., 2016)
2.92
0.861
SUPCR(OR)
2.86 (+0.06)
0.867 (+0.006)
CORN (Shi al., 2021)
3.24
0.819
SUPCR(CORN)
2.89 (40.35)
0.862 (+0.043)
5) IMDB-WIKI (Rothe et al., 2015) a large dataset for predicting age from face images, which contains 523,05 celebrities images and the corresponding age labels. The age range is between and 186 (some images are mislabeled). We use this dataset to test our method's resilience to reduced training data, performance
transfer learning. and ability to generalize unseen targets. We subsample the dataset to create a variable size training set, and keep the validation set and test set unchanged with 11,022 images in each.
Metries. We report kinds of metrics: the prediction error and coefficient of determination (R²). Prediction errors have practical meaning and are convenient for interpretation, while R² quantifies how much the model outperform a dummy regressor that always predicts the mean value of the training labels. For age, brain-age, and temperature, the mean absolute error (MAE) is reported as the prediction error. For gaze direction, the angular error reported as the prediction error.
We implemented seven typical regression methods as baselines. L1, MSE and HUBEr have the model directly predict the target value and train the model with an error- bused loss function, where L. uses the mean absolute error, MSE uses the mean squared error and HUBER uses MSE
Baselines.

---

Under review as a conference paper at ICLR 2023
term when the error is below threshold and an LI term otherwise. DEX (Rothe et al., 2015) and DV (Gao et al., 2018) divide the regression range of each label dimension into several bins and learn the probability distribution over the bins. DEX (Rothe et al., 2015) optimizes cross-entropy loss between the predicted distribution and the one-hot ground-truth labels, while DL.DI-V2 (Gao al., 2018) jointly optimizes KI. loss between the predicted distribution and normal distribution centered at the ground-truw truth value, as well as an LI loss between the expectation of the predicted distribution and the ground-truth value. During inference, they output the expectation of the predicted distribution for each label dimension. OR (Niu et al., 2016) and CORN (Shi et al., 2021) design multiple ordered threshold for each label dimension, and learn a binary classifier for each threshold. OR (Niu et al., 2016) optimizes a binary cross-entropy loss for each binary classifier to learn whether the target value larger than each threshold, while CORN (Shi et al., 2021) learns whether the target value is larger than each threshold conditioning on it is larger than the previous threshold. During inference, they aggregate all binary classification results to produce the final results.
Experiment Settings. We use ResNet-18 (He al.. 2016) as the backbone model for AgeDB, IMDB-WIKI, MP MPIIFaceGaze and SkyFinder datasets. For TUAB, we use 24-layer ID ResNet (He et al., 2016) as the backbone model to process the EEG signals. We use the linear regressor as the predictor. We use the SGD optimizer and cosine learning rate annealing (Loshchilov & Hutter, 2016) for training. The batch size is set to 256. We pick the best learning rates and weight decays by grid search for both our method and the baselines. We train all the baselines and the encoder of our method for 400 epochs, and the linear regressor of our method for 100 epochs. The same data augmentations are adopted for all baselines and our method: random crop and resize (with random flip), color distortions for AgeDB, IMDB-WIKI, SkyFinder: inder; random crop and resize (without random flip), color distortions for MPIIFaceGaze; random crop for TUAB. Appendix C shows augmentation examples for each dataset. Negative L2 norm, i.c., sim V. v₁) = -1 1yl12 is used as the feature similarity measure in SupCR. LI distance used the label distance measure in SupPR for AgeDB, IMDB-WIKI, SkyFinder and TUAB, while angular distance is used as the label distance measure for MPIIFaceGaze. The temperature is set to 2.0. Refer to Appendix for more details.
4.1
MAIN RESULTS
As explained earlier, our model learns a gggression-suitabbee representation that can be used by any of the baselines. Thus, in our comparison, we first train the encoder with the proposed LSupCR, and then freeze the encoder and train a predictor on top of it using each the baseline methods. We then compare the original baseline without our representation, to the same baseline with our representation. For example, L1 means training the encoder and predictor end-to-end with L1 loss and SUPCR(LI) means training the encoder with LSpceR and then training the predictor with L1 loss.
Table 1, Table 2, Table 3 and Table 4 show the evaluation results on AgeDB, TUAB, MPIIFaceGaze and SkyFinder, respectively. Green numbers highlight the performance gains brought by using representation and the best numbers are shown in bold. As all tables indicate, SUPCR consistently achieves the best performance both metrics across all datasets. Moreover, incorporating SUPCR to learn the representation consistently reduces the prediction error of all baselines by 5.8%, 10.1 1,, 11.7% and 7.0% on average the AgeDB, TUAB, MPIIFaceGaza and SkyFinder, respectively.
4.2 ROBUSTNESS TO DATA CORRUPTIONS
12
SupCR(L1)
11.75
Mean Mean Absolute Error (MAE)
11
It is well known that deep neural networks are highly vulnerable to out of distribution data and different forms of corruptions (Hendrycks & Dietterich, 2019).
10
9.86
9.60
such as noise, blur and color distortions. Using the corruption generation process in ImageNet-C bench-
9
8.60
8.78
8
7.47
7.78
mark (Hendrycks & Dietterich, 2019), we corrupt the AgeDB test set with 19 diverse corruption types under different severity levels. In Figure 3, we compare SLPCR(L1) with L1 by testing the model (trained on clean ABeDB training set) corrupted test set and reporting the average mean absolute error over all types of corruptions for each severity level. The re-
7
6.63
6.85
14
2
3
Corruption Severity Level
Figure SUPCR is more robust data cor- ruptions. Figure shows MAE+ as a function of corruption severity.
7

---

Under review as a conference paper at ICLR 2023
sults show that the representation learned by SUPCR is more robust to unforeseen data corruptions,
and SUPCR suffers from lesser performance degradation as corruption severity increases.
20
4.3 RESILIENCE TO REDUCED TRAINING DATA
L1
SupCR(L1)
18.60
(MAE) (MAE)
Much of the success of modern deep learning is due
Mean Absolute Emoo
16
15.09
to the availability of massive training datasets. How- ever, in many real-worls settings, collecting large training set is often infeasible due to the time and cost of labeling. Thus, is desirable to improve model resilience to reduced training data. In Figure 4, we subsample IMDB-WIKI to produce varying size train-
14
13.00
12
10.74
11.94
10
8.89
10.23
8.83
9.36
ing sets, and compare SUPCR(L1) 1) with L1. The results show that SUPCR is more resilient to reduced training data, and exhibits lesser performance degra- dation as the number of training samples decreases.
32k
16k
8k
4k
2k
Number Training Samples
Figure 4: SUPCR is more resilient to reduced training data. Figure shows MAE as func- tion of the number of training samples.
4.4 TRANSFER LEARNING
We evaluate the transfer learning performance by first pre-training the feature encoder on large dataset and then using either linear probing (fixed encoder) fine-tuning to learn predictor on a small dataset. We investigate two scenarios: transferring from AgeDB which contains ~ 112k samples to a subsampled IMDB-WIKI of 2k sam- ples and transferring from another subsampled IMDB-WIKI of 32k samples to AgeDB. As shown in Table SUPCR(L1) outperforms L1 both the linear probing and fine-tume tinettunin settings in both two scenarios.
Table 5: Transfer learning results.
R²
R-T
AgeDD IMDO (subsampled,
Linear Probing:
Fine-tuning:
L11
12.25
0.496
L1
11.57
0.528 11.09
SUPCR(LI)
11.12
SUPCROLI)
(41.3)
(+0.060)
(40.45)
(40,018)
INDS-VIKI (subsampled, 32k)
AgoDB:
Linzar Probing:
Pine-tniig:
LI SUPCRILI)
7,36
0
LI
6.36
0.848
7.06
SUPCR(LI)
0.850
(+0.30)
(+0,011)
(+0.23)
(+0.002)
4.5 GENERALIZATION TO UNSEEN TARGETS
In real-world regression tasks, it is common that some targets are unseen during training. As curate two subset of
Table 6: Generalization performance to unseen tar- gets on two curated subsets of IMDB-WIKA MAE+ used as the metric.
in Yang et al. (2021), IMDB-WIKI that contain unseen targets. The test set is uniform distribution across the whole target range. Table shows the label distribu- tions of the two training set, where pink shad- ing indicates regions of unseen targets, and the blue shading represents the distribution of seen targets. The first training set consists of 3,781 training samples with a bi-modal Gaussian dis- tribution over the target space, and the second training set consists of 3,662 training samples with tri-modal Gaussian distribution over the target space. We report the prediction error on the seen and unseen targets separately. The results show that SUPCR (L1) outperforms L.1 by larger margin the unseen targets without sacrificing the performance on the seen targets.
Label Distribution
Method
All
Seen
Unseen
LI
12.53
10.82
18.40
SUPCR(LI)
11.69
10.46
(+0.84)
(+0.36)
(+2.48)
11.94
10.43
14.98
SUPCR(LI)
10.88
9.78
13.08
(+1.06)
(+0.64)
(+1.90)
4.6 DESIGN VARIANTS
Similarity Measure and Projection Head. We explore potential variants of our method to pick the best design. Table 7 compares the performance of our method (CSupCR with similarity measure using the negative L2 norm) with alternative designs that use a different similarity measure and potentially support projection head. The results are on the AgeDB dataset and using SUPCR(LI).
Specifically, the table shows the performance using different feature similarity measures sirn(;;)) :.) We see that among cosine similarity, i.c., sim(v, v;))
mi Te
negative LI norm, ie..
8

---

Under review as a conference paper at ICLR 2023
Table Comparison potential variants of SuprR and their performance on AgeDB.
R²T
Loss Term
Similarity Measure
Projection Head
MAE
SsinLL
cosine
9.59
0.666
LSimCLR
cosine
X
10.67
0.600
LsupCon
cosine
/
8.13
0.753
LsupCon
cosine
X
7.89
0.760
SupeR
cosine
V
7.34
0.793
CupRR
cosine
X
6.51
0,836
SupRR
negative L1 norm
X
6,25
0.842
SSpCR
negative 1.2 norm
X
6.14
0.850
sim(ns wz) = - -1 æill and negative 1.2 norm, i.e., sim(n; 05) = - -1| 1yl12, negative 1.2 norm delivers the best performance.
The table also shows the performance with and without projection head. To train the encoder network f(-) - Rd,, one can involve projection head h(-) R2d - Rd during training to calculate the loss on (N(f(";)))==eeen) and then discard the projection head to use f(x;) for inference. One can also directly calculate the loss on (J(i;) [E(NN] and then use f((;;) for inference. SIMCLR (Chen et al., 2020a) and SUPCON (Khosla et al., 2020) both use the former approach.
For LSICLR the results show that adding the projection head can benefit the regression performance. This is because LSICLR aims to extract features that are invariant to data augmentations and can remove information that may be useful for the downstream regression task. For LSupCon and LSupCR. however, it better to train the encoder without the projection head, since both LsupCon and LspeE leverage the label information to extract features that directly target the downstream task.
Moreover, we see that using LsupCon or LupCR as the loss function delivers better performance than using LSICLLR verifying that it is helpful to utilize label information during encoder training. We also see that LupCR outperforms CsppCon by a large margin, highlighting the superiority of L which explicitly leverages the ordered relationships between samples for regression problems.
Training Scheme. There are usu- ally three schemes to train the fea- ture encoder: (1) Linear probing: first trains the feature encoder using the representation learning loss, then freezes the encoder and trains linear regressor on top of using a regres- sion loss. (2) Fine-tuning: first trains
Table 8: Performance of alternative training schemes for the feature encoder on AgeDB.
Scheme
Linear Probing
Fine-tuning
Regularization
MAE! R²T
6.14
6.36
6.42
6.63
0.850
0.844
0.833
0.828
feature encoder using the representation learning loss, then fine-tunes whole model using a regression loss. (3) Regularization: trains the whole model while jointly optimizing the representation learning loss and the regression loss.
Table 8 shows the performance AgeDB for the three schemes using LSupCR the representation learning loss and L1 as the regression loss. We see that all of the three schemes can improve performance
using the regression loss L1 alone train model. Further, unlike classification problems where fine-tuning often delivers the best performance, freezing the feature encoder performs the best. This is because the case of regression, back- pack-propagating the Ll loss the representation can destroy the order in the embedding space learned by LupcCR which leads to poorer performance.
CONCLUSION
In this paper, propose Supervised Contrastive Regression (SupCR), framework that learns regression-awarr representation by contrasting samples against each other according to their target distance. The proposed framework is orthogonal to existing regression models, and can be used in combination with such models to improve performance. It achieves the ofate-of-the-a performance and consistently improves prior regression baselines across different datasets, tasks, and input modalities. It also improves the robustness to data corruptions, resilience to reduced training data, performance transfer learning, and generalization unseen targets.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ahmed A Abdelrahman, Thorsten Hempel, Aly Khalifa, and Ayoub Al-Hamadi. L2cs-net: Fine- grained gaze estimation in unconstrained environments. arXiv preprint wwXXi:2203.033399 2022.
Wenzhi Cao, Vahid Mirjalili, and Sebastian Raschka. Rank consistent ordinal regression for neural networks with application to age estimation. Pattern Recognition Letters, 140:325-331, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020b.
Wei-Ta Chu, Kai-Chia Ho, and Ali Borji. Visual weather temperature prediction. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 234-241. IEEE, 2018.
Ching- Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. De- biased contrastive learning. Advances in neural information processing systems, 33:8765-8775, 2020.
Benoit Dufumier, Pietro Gori, Julie Victor, Antoine Grigis, and Edouard Duchesnay. Conditional aligmment and uniformity for contrastive learning with continuous proxy labels. arXiv preprint arXiv:2111.05643 2021a.
Benoit Dufumier, Pietro Gori, Julie Victor, Antoine Grigis, Michele Wessa, Paolo Brambilla, Pauline Favre, Mircea Polosan, Colm Medonald, Camille Marie Piguet, et Contrastive learning with continuous proxy meta-data for 3d mri classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 58-68. Springer, 2021b.
Denis A Engemann, Apolline Mellot, Richard Höchenberger, Hubert Banville, David Sabbagh, Lukas Gemein, Tonio Ball, and Alexandre Gramfort. reusable benchmark of brain-age prediction from m/eeg resting-state signals. Neurolmage, 262:1 19521, 2022.
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2002-2011 2018.
Bin-Bin Gao, Chao Xing, Chen-Wer Xie, Jianxin Wu, and Xin Geng. Deep label distribution learning with label ambiguity. IEEE Transactions Image Processing, 26(6):2825-288) 2838, 2017.
Bin-Bin Gao, Hong- Yu Zhou, Jianxin Wu, and Xin Geng. Age estimation using expectation of label distribution learning. In IJCAI, pp. 712-718, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729 9738,, 2020.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corrup- tions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Peter Huber. Robust estimation of a location parameter. In Breakthroughs in statistics, pp. 492-518. Springer, 1992.
10

---

Under review as a conference paper at ICLR 2023
Bingyi Kang. Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference on Learning Representations, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sama,, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33: 15116661-16777 2020.
Shikun Li, Xiaobo Xia, Shiming Ge, and Tongliang Liu. Selective-supervised contrastive learning with noisy labels. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 316-325, 2022a.
Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6918-6923 2022b.
llya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:/608.0399 2016.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 1rXiv:1802.03426, 2018.
Radu P Mihail, Scott Workman, Zach Bessinger, and Nathan Jacobs. Sky segmentation in the wild: An empirical study. 2016 JEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1-6. IEEE, 2016.
Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshop, volume 2, pp. 2017.
Zhenxing NIu, Mo Zhou, Le Wang. Xinbo Gao, and Gang Hua. Ordinal regression with multiple output enn for age estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4920 4928, 2016.
Iyad Obeid and Joseph Picone. The temple university hospital eeg data corpus. Frontiers in neuroscience, 10:196, 2016
Hongyu Pan, Hu Han, Shiguang Shan, and Xilin Chen. from face. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5285-5294, 2018.
loss for deep age estimation
Rasmus Rothe, Radu Timofte, and Luc Van Gool. Dex: Deep expectation of apparent age from single image. In Proceedings of the IEEE international conference on computer vision workshops, pp. 10-15, 2015.
Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioral and neural analysis. arXiv preprint arXiv:2204.0005
2022.
Fabian Schrumpf, Patrick Frenzel, Christoph Aust, Georg Osterhoff, and Mirco Fuchs. Assessment of non-invasive blood pressure prediction from ppg and rppg signals using deep learning. Sensors, 21(18):6022, 2021.
Xintong Shi, Wenzhi Cao, and Sebastian Raschka. Deep neural networks for rank-consistent ordinal regression based conditional probabilities, 2021.
Yaoming Wang, Yangzhou Jiang, Jin Li, Bingbing Ni, Wenrui Dai. Chenglin Li, Hongkai Xiong, and Teng Li. Contrastive regression for domain adaptation gaze estimation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19376-19385, 2022.
Yuzhe Yang, Kaiwen Zha, Yingcong Chen, Hao Wang, and Dina Katabi. Delving into deep imbal- anced regression. In International Conference on Machine Learning, pp. 11851. PMLR, 2021.
11

---

Under review as a conference paper at ICLR 2023
Xumin Yu, Yongming Rao, Wenliang Zhao, Jiwen Lu, and Jie Zhou. Group contrastive regression for action quality assessment. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7919-7928, 2021.
Zhiyuan Zeng, Keqing He. Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. Modeling discriminative representations for out-of-domain detection with supervised contrastive learning. Proceedings the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 1 Ith International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 870-878, 2021.
Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. Mpïigaze: Real-world dataset and deep appearance-basee gaze estimation. IEEE transactions pattern analysis and machine intelligence, 11(1):162-175, 2017a.
Xucong Zhang, Yusuke Sugano, Mario Fritz, and Andreas Bulling. It's written all over your face: Full- face appearance-basee gaze estimation. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on, pp.
IEEE, 2017b.
12

---

Under review as a conference paper at ICLR 2023
A
PROOFS
Theorem (Lower bound of EsppCR). L is a lower bound of SupprRs i.e., LSupeR L..
Proof.
LSuPER
=-
1
Σ
Σ
log Σ exp(si.A)
exp($;,)
2N
1)
i=1 JE(NNT((
RE(2N]N(O); 2d,,
2N
Σ
M,
Σ
log
exp(81.)
2N
exp(stA)
(2)( di.
AE(2N]R(E): 4,,22
1
ΣΣ
Σ
Σ
log E
exp(3;,)
2N(2N - JE day D
exp(sia)
RE(2N]((6). di Dimm
E
exp(six 85,5)
M.
Σ
Σ
log 14 KE(2N/)(1), d...D.m
2N (C))))(() day Dam
Σ
exp(81A 8i.))
kE(2N/ dea=Di.
2N Mr
exp(s1,)
Σ
log E
2N(2N -
exp(86.)
JE(2MNL(E) D..
kE(2NT dr,=D,,,
(2)
Vi [2N]. E [M., from Jensen's Inequality have
exp(sig)
Σ
log Σ
exp(six)
jE( diy=Diin
RE(2N]M()), d,, Dem
(3)
exp(si.s)
-m log
Σ
n4m log Ti.
Diam
Σ
exp(si,k)
JE(2N]N()) di.j=Di.m kE(pN])(s), dea D.m
Thus, by plugging Equation 3 into Equation 2, we have
22
1
SupCR
Σ
nim log fi.m L".
(4)
2N(2N
E=
Theorem 2 (Lower bound tightness). For any >> O, there exists set of feature embeddings such that LupPCR < L + e.
Proof. We will show Ve 0, there is set of feature embeddings where
si.j > Si,k + if dijj < di,b
$1 = 3i,k if di.j = di,k
and ;= log
2N
Vi E j,k E [2N],((,, such that LSupeR < L* +e.
<E[aN] m M.)
For such a set of feature embeddings, Vi E [2N], E [M.,j e (2NN()) di.j Di.m).
exp(*t.))
llog
log n.,m
(5)
Σ
exp();,))
kE(2M]
D..=D.m
13

---

Under review as a conference paper at ICLR 2023
since Si,k 8i, for all ls such that di Dim di.s and
Σ
exp(8i,k 8t.5)
1 ME(2N]u(), de Dim
log
exp(8i,k 8(.5) AE|2NTT(()) dia=Dimm
(6)
2N exp( (0((-)))
2N exp(-7) S
< log
1
ni,m
n4.m
since $i,k $t, < -y for all & such that di > Dim di, and 8 81,5 = 0 for all & such that
di,k Di,m di
From Equation we have
CsupeR 2N(2N
2N
M
Σ
log
exp(s.,))
1)
exp($1,E)
jE ME) Dt..
AC(2NTM(s), di, Dim
Σ
exp(81,6 - -8,1)
2N
Σ
log
KEIZN(( deL
2N(2N -
Σ
exp($s,t - 85,)
JE Di.m
d,,,
By plugging Equation 5 and into Equation 7 we have
2N
LSppr < 2N(2N 1)
1
Σ
Σ
nim log ni. + L* +e
(8)
1=
m=1
Theorem 3 (Main theorem). For any 6 < there exist 0, such that if LupeR then the feature embeddings are 5-ordered.
L +
Proof. We will show v0 < 6 < 1, there a
1 min min (e(NN].me(M.
log [1+ ni,m exp(6 + }) 2lggg 1 exp(o) 8) 0,
2N(2N -
such that when CSupeR L e, the feature embeddings are 8-ordered.
We first show that |si.j si,A| < 8 if = di. vi E [2N], j. k E [2N] when CSapeR < +e+e. From Equation we have
M.
exp(sij)
SSucCR 2N(2N 1
Σ
log
exp(5t,6) (9)
jE(2N] diy Di,
kE(2N] -
Let Pi.m
min JE(2)]((), dD,==
8i.j Ii.
arg max
Si.o Si. := Si.p.. mi
VE(N]V)() dD.
8i.gi.m we have
Si.pr.
Vi [2N], m E [M.]), by splitting out the maximum term and the minimum term
exp(Sim)
LSupCR
log
2N(2N 1)
Σ
exp(8i,6)
kE(NN](() dr.a=Diom
exp
Σ
81.i
+ log exp(Sim + Tmm)
jE[2N]] Pi Fon Di.m
exp(six) + log
kE(2N]{(1), di.a=D.m.
Σ
exp(si,))
sc(2 d,.
(10)
14

---

Under review as a conference paper at ICLR 2023
Let fa,m=
1
Σ
exp(si. ((,m), we have
ng,m
jE|2N] (1 (.. m.d.
exp(Si,m)
log
log(1 + exp(nim) + (14mm 2)0mm)
(11)
Σ
exp(si,N)
ke(2N]M((), di,4=D,m
and
exp(Si,m + (1mm)
log
log(1 + exp(x).) + (Nimm 2)0m.m) 71mm-
(12)
Σ
exp(81,K)
KE(ONT(() de.=Dam
Then, from Jensen's inequality we know
exp
Σ
S
Σ
exp(s1.)
JE(2N]) (iPe.mil.m =
m1,m
(EP.. = Dr.m
(13)
thus
log
16 Se R.l Pr.m
2) log(1 + exp(Ni.m + (Nam 2)0 (Mr,m - 2) log(O...) (m..m)
exp(en.))
VE/22TV1 PPu
(14)
By plugging Equation 11, 12 and 14 into Equation 10, we have
LSupER 2N(2N 1) Σ> Σ (nim log(1 + exp(n).) + (As,m 2)0mm) fe.m (Na,m 2) log(B,,m))
(15)
Let h(() nimm log(1 coxpp (Mi,m -2)) -hmm (n 2) log(0). From derivative analysis
we know h(0) decreases monotonically when 0 E 1. 1-exp(n and increases monotonically when 0 [+exp(mmm) eex((mmm) thus
h(0) 2 h 1 (1+ exp(N.m)))
+ exp(ni.m) nmm
ni.m log ns.m + +-21e 22g
(16)
By plugging Equation 16 into Equation 15, we have
CSupeR 2N(2N
1
Σ
2N
Σ
ni,m log nim
2log
+ exp(Ni.m)
7a.m
1)
2
s=1
m=1
(17)
2N
M
1 exp(()mm)
I 2N - 1)
2
17,,m
Then, since 7i.m 2 0, we have 2oog 1eexp(y...)
Th.m 2 Thus, Vi [2N], E [Mc].
2
1
1 ++eeppni exp(T).m) 7)
SupcR > I* + 2N(2N -1)
22log' 2
(18)
SSpCR L*+
22log Ltexp(d)
6). then
2N(2N-1
+ exp(1k.m) T. 2
exp(6)
2 log
(19)
2
2
Since y(z) = 2og 1+exp(z) x increases monotonically when x > 0, we have ni,m < 6. Hence,
Vi [2N], j. [2)] if d d Da 18t. 88,, S 1a < 8.
15

---

Under review as a conference paper at ICLR 2023
Next, we show $..j > Si,k + 8 if di < di when SupeR L +
From Equation we have
CsupER
2N
M
Σ
log
exp( 86,3)
2N(2N
1)
exp(s1,x)
JC(N))))) di,; Dam
k di, ;...
Σ
exp(8i,t - 85,5)
2N
M;
Σ
log 1 ACIANT())) de.
2N
E
exp(31,1 84,)
JE Di.m
kEjZN] doa=Di.
(20)
and combining it with Equation 3 we have
Σ
exp(SL# 81.5)
2N M.
CSupER L
1
Σ
Σ
Σ
KE|2N Vo de.. log 1+
2N(2N
Σ
exp(st, $1.)
=
m=1
ME)) Di.=Di,m
ke(2N]](6, dD.=Dim
exp(sEA 81,2)
> +
log
1+ Σ
2N(2N - 1)
exp(s1. - 85,5)
(E(NNT()): d.
(21)
vi [2N],
{E (2 de.5 d.
When Lupce < L +c, we already have |su st,l < 6, Add. d which derives 8t - 8i, <
and thus exp(st, -S1,) exp((). By putting this into Equation 20, we have vi [2N], j [2N])((), E {k [2N]((0) dig < di,),
log 1 exp(81, - 56.5) Mirri cxp(6)
LSppCR > I 2N(2N -
(22)
where rıj [Mi] is the index such that Dirri di.j-
Further, given LSupCR L +
2N(2N log(1 + marr, exp(o+ ) we have
log(1 + exp(8i, -8t.)) < log(1 Dirri exp(6 4 +)
(23)
Riry, exp(o)
which derives 8.j > 8i,k J. Vi [2N], j [2N]] (k E [2N]][+] dis < dr,)).
Finally, Vi E [2N]]. j, E [2N] 8ú, < 88 da,j > de, directly follows from 8i.j > 81.6 + 8 if dij < di
B ETHICS STATEMENTS
All of the datasets used in the paper are public datasets. The ethics statements for each dataset listed below:
This dataset is collected from archival records at Temple University Hospital (TUH). All work was performed in accordance with the Declaration of Helsinki and with the full approval of the Temple University IRB. All personnel in contact with privileged patient in- formation were fully trained on patient privacy and were certified by the Temple IRB (Obeid & Picone, 2016).
TUAB:
MPIIFaceGaze: The authors of Zhang et al. (2017b) confirmed that the informed consent has been obtained from all of the participants.
AgeDB, IMDB-WIKS and SkyFinder: Those datasets collected without including any interaction or intervention with human subjects and do not contain any private information (Moschoglou et al., 2017; Rothe al., 2015; Mihail et al., 2016).
16

---

Under review as a conference paper at ICLR 2023
Original
Augmented
AgeOB
www. Mwm
TUAB
MPIlFaceGaze
SkyFinder
Figure 5: Visualizations of original and augmented data samples each dataset.
C VISUALIZATION OF DATA AUGMENTATION
Figure shows examples of original and augmented data samples on each dataset. The data augmen- tation used on each dataset are listed below:
For AgeDB and SkyFinder, random crop and resize (with random horizontal flip), color distortions are used as data augmentation;
For TUAB, random crop is used data augmentation;
For MPIIFaceGaze, random crop and resize (without random horizontal flip), color distor- tions are used as data augmentation.
D DETAILED EXPERIMENT SETTINGS
For the encoder training of our method and regression learning baselines, we selected the best learning rates and weight decays for each dataset by grid search, with grid of learning rates from (0.01, 0.05, 0.1, 0.2, 0.5, 1.0] and weight decays from {10-6, 6, 10 10-3 10 10-3), For the predictor training of our method, we adopted the same search setting above except for adding no weight decay the search choices of weight decays. For temperature parameter т. we searched from (0.1, 0.2, 0.5, 1.0, 2.0, 5.0) and selected the best, which is 2.0.
For the classification-uue based baselines, we divided the regression range into small bins. For AgeDB, the target range is ~ 101 and the bin size is set to 1; for TUAB, the target range is 0 ~ 95 and the bin size is set to I; for MPIIFaceGaze, the target range is -40 ~ 10 for the pitch angle and -45 ~ 45 for the yaw angle, and the bin size is set to 0.5 for the pitch angle and is set to for the yaw angle. For SkyFinder, the target range -20 ~ 49 and the bin size is set to 1. For the implementation of SupCon (Khosla al., 2020) regression datasets, the regression range divided into small bins in the same way and each bin is regarded as a class. Samples from the same class, i.c., sharing the same target bin the anchor, considered positive samples.
17

---

Under review as a conference paper at ICLR 2023
E ADDITIONAL EXPERIMENTS AND ANALYSIS
E IMPACT OF MODEL ARCHITECTURES
In the main paper, we use ResNet- 18 as the encoder backbone for three image datasets (AgeDB, MPIIFaceCaze and SkyFinders In this section, we study the impact of backbone architectures on the experiment results. As Table 9 reports, the results of using ResNet-50 as the encoder backbone are consistent with the results with ResNet-18 18 in Table 1, Table 3 and Table 4, showing that our method is compatiable with different model architectures.
Table 9: Evaluation results using ResNet-50 as the encoder backbone.
Dataset
AgeDB
MPIIFaceGaze
SkyFing SkyFinder
Metrics
MAE+
R²1
Angular
RaD
MAE
R²7
L1
6.49
0.830
5.74
0.748
2.88
0.863
SUPCR(L.1)
6.10
0.851
5.16
0.819
2.78
0.877
GAINS
+0.39
+0.021
+0.58
+0.071
+0.10
+0.014
E.2 COMPARISON TO EXISTING SOTA METHODS ON EACH DATASET
In this section, we compare the performance of our method with the current state-of-the-art (SOTA) methods on each dataset. To the best of our knowledge, the SOTA on each dataset are (Yang et al., 2021) for AgeDB, (Engemann et al., 2022) for TUAB, (AbdeIrahman et al., 2022) for MPIIFaceGaze, and (Chu et al., 2018) for SkyFinder. However, They are either using different backbone archi- tectures and optimizers, or using different dataset splits and number of training samples from ours. Therefore, the results reported in their paper not directly comparable to our results. We implement their method under our training and evaluation protocols and summarize the results in Table 10. Results show that our method surpasses the state-of-the-ar methods all datasets.
Table 10: Comparison to exsiting SOTA methods on each dataset.
Dataset
AgeDB
TUAB
MPIIFaceCaze
SkyFinder
Metrics
MAE
MAE
R2T
Angular
MAE
RMSE
SOTA SUPCR
6.58
7.96
0.599
5.76
2.97
4.05
6.14
6.91
0.697
5.13
2.85
3.84
GAINS
+0.44
+1.05
+0.098
+0.63
+0.12
+0.21
E.3 COMPARISON TO MORE SELF-SUPERVISED LEARNING METHODS
In Section 4.6, we compare our method with SIMCLR (Chen et al., 2020a) and SUPCON (Khosla et al., 2020) on AgeDB dataset. In this section, we investigate more SSL methods under various training settings compare with our method on all datasets. Specifically, we evaluate the encoder pretrained using SIMCLR and DINO (Caron et al., 2021) on ImageNet and the regression datasets respectively. Besides, we also consider the setting where we use SIMCLR loss as regularization along with I.1 loss to train on the regression datasets. The results are shown in Table 11. We use ResNet-50 as the encoder backbone for image datasets. The ResNet-50 mererret-reerrained encoders 800-epoch checkpoints and their ImageNet linear evaluation accuracies are 69. 1 % and 75.3% respectively. For DINO-related methods, apart from linear evaluation, we also report K-NN evaluation results (k- 20).
The results show that SUPCR outperforms all these baselines on all datasets, and pre-training using SIMCLR DINO, using SIMCLR as regularizer all performs worse than the vanilla L1 baseline. This further verifies that the performance gain of our method stems from our proposed SUPCR loss rather than the pre-training scheme.
18

---

Under review as a conference paper at ICLR 2023
Table II: Comparison to SSL methods applied under different settings on each dataset. MAE- is used as the metric for AgeDB, TUAB and SkyFinder, and Angular Errort for MPIIFaceGaze.
Dataset
AgeDB
TUAB
MPIIFaceGaze
SkyFinder
Evaluation Protocal (for pre training scheme)
Linear
N-NN
Linear
N-NN
Linear
N-NN
Linear
NNNN
LI
6.49
7.96
5.74
2.88
LI SIMCLR ERGULARIZAIION
8.02
2.93
SIMCLR (PRE-TRAINED ON IMAGENET)
15.32
16.11
5.03
DINO (PRE-TRAINED IMAGENET)
9.14
11.82
12.44
12.84
4.62
3.64
SIMCLR TRAENTRAINED ON REGRESSION DATASET)
9.38
11.01
9.13
4.55
DINO (PRE-TRAINED ON REGRESSION DATASET)
9,91
12.03
11.62
12.32
11.90
12.50
5.00
4.20
SUPCR(LII)
6.10
6.97
5.16
2.78
E.4 STANDARD DEVIATIONS OF RESULTS
In this section, we study the standard deviations of the best results on each dataset with different random seeds. Table 12 shows their average prediction errors and standard deviations. These results are aligned with the results we reported in the main paper.
Table 12: Average prediction errors and standard deviations of the best results each dataset.
AgeDB: SUPCR(LI)
TUAB: SUPCR(DLDL-V2)
MP1IFaceGaze: SUPCR(OR)
SkyFinder: SUPCR(DLDL-V2)
6.19 + 0.08
7.00 + 0.10
5.24 + 0.13
2.87 + 0.04
E.5 ABLATION ON NUMBER OF POSITIVES
In CSupCR, all samples in the batch will be treated as the positive for each anchor. Here, we conduct an ablation study on only considering the first K closest samples to the anchor as positive. Table 13 shows the MAE on AgeDB for different Kss (using L.1 loss in the linear probing), where K = 511 means considering all samples as positive since we use a batch size = 256 and
Table 13: Ablation number of positives considered.
K
128
256
384
511
MAE
6.46
6.43
6.29
6.14
thus the number of all samples other than the anchor equal to 2NN 1 511. These experiments show that the larger K, the better the performance.
This phenomenon is aligned with the design intuition of SUPCR loss since each contrastive term ensures a group of orders related to the positive sample, i.e., it ensures all samples that have larger label distance from the anchor than the positive sample to be farther from the anchor than the positive in the feature embedding space. Only when all of the samples are considered as positive and their related groups of orders ensured, the order in the feature space be fully guaranteed.
E.6 TRAINING EFFICIENCY
We compute the average wall-clock running time (in seconds) per training epoch 8 NVIDIA TITAN RTX GPUs for SUPCR and compare it with SUPCON (Khosla et al., 2020) on all four datasets, as shown in Table 14. Results show the training efficiency of SUPCR is comparable to SUPCON.
Table 14: Average wall-clock running time (in seconds) per training epoch on NVIDIA TITAN
RTX GPUs for SUPCR and SUPCON (Khosla et al., 2020) each dataset.
Dataset
AgeDB
TUAB
MPIIFaceGaze
SkyFinder
SUpCON
23.1
25.3
69.1
55.6
SUPCR
26.2
27.3
75.4
61.8
RATIO
1.13
1.08
1.09
1.11
19


Reviews:
[{"review_id": "mtbU7iKwpk7", "reviewer": "Reviewer_7jug", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method for regression problems based on Supervised Contrastive Learning approach(SupCon) designed for classification tasks. The paper argues that existing regression-based approaches, don't learn embeddings suited for regression tasks, whereby the distance in embedding space is equivalent to the distance in the label space. They empirically show this claim by a UMAP plot for a regression task, whereby the embeddings learned by baseline L1 approaches do not capture the intrinsic ordered relationship between the samples. The paper argues that learning a regression-suited embedding space, will allow the models to be robust to corruption and also generalize to unseen targets.\nThus, the authors propose an approach similar to SupCon for regression tasks which they call SupCR(Supervised Contrastive Regression). In SupCon the positive samples for contrastive learning are not only the augmented version of the query but also those that belong to the same class as the query. In SupCR, the authors first consider the closest sample as the positive, and all other samples whose label distance is larger than the distance between the query and positive are treated as the negative. This is then iterated for the second closest, third closest samples and so on. Once the embeddings are learned in the contrastive way, the authors either fine-tune the entire network or only train a linear layer using either L1/MSE/Huber loss. \n\nThe authors show that their proposed approach achieves state-of-the-art results over several different datasets. Additionally, the proposed method shows robustness to corruption, better transfer learning abilities, and generalization to unseen targets.\nThe authors also provide a theoretical justification for how their proposed optimization will make the feature embeddings ordered according to order in the label space. ", "summary_of_the_review": "While the paper does extensive study and provides a good justification of the proposed approach via state-of-the-art results, the novelty of the work is a bit limited as it is mostly just adapting SupCon for regression tasks. I also have a few more concerns regarding the further justification of the proposed approach and the baselines. \nI am leaning towards weak accept but am willing to update my ratings if my concerns are addressed.", "strength_and_weaknesses": "Strengths - \n1. The paper is well written and easy to follow along.\n2. The authors do an extensive study across different approaches and settings, such as which similarity function to use, whether to use projection head or now and others.\n3. Along with better task performance, Improved robustness to corruption and generalization to unseen targets shows the goodness of the approach.\n\nWeakness - \n\n1. For the contrastive learning part, the paper iteratively considers the first closest sample, then the second closest and so on for their positive samples. It is unclear if we really need to consider this iteration till 2N steps. What happens if we only consider the first closest of the first K closest samples as positive. This ablation study would be important to justify the proposed approach, but is missing from the paper.\n\n2. It's unclear how is SupCon used as a baseline in the paper, when the task is a regression task. How are the authors constructing positive samples for the SupCon approach?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SKASMo9sYf7", "reviewer": "Reviewer_dbgX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new variant performing supervised contrastive learning with continuous labels. Improvements on four image-based and one EEG regression tasks are demonstrated when using the learned feature space instead of training various baseline algorithms from scratch. The paper also shows improvements in terms of robustness and transfer learning performance, as well as ablations on the impact of a projection head on the regression performance.", "summary_of_the_review": "The paper introduces a new loss function for supervised contrastive learning with continuous labels. The experimental setup is interesting and well evaluated, and the paper is well written. The main limitation is the insufficient discussion of prior work on contrastive learning with continuous labels and positioning in the literature, along with possible improvements in the experiments. I made specific suggestions on how to improve these weaknesses above, and am willing to adapt my scores.\n\n---\n\n## Post Rebuttal comments (for now)\n\nThe authors did a good job at addressing my concerns. I raised the *Correctness* from 3 to 4, especially because a lot of insightful baseline experiments have been added. I also appreciate that the authors now better position the work in the existing literature, and refined both the paper title and the discussion of prior work.\n\nI raised my overall score from 5 to 6 (pending discussion with other reviewers).", "strength_and_weaknesses": "**Strengths:**\n\n- The method is simple to implement, well evaluated and the results are laid out clearly.\n- The authors demonstrate improvements of their method both in terms of accuracy and robustness to image perturbations on a range of different evaluation tasks.\n- The method is supported by theory. *Disclaimer: While the theoretical results are plausible, I did not check the proofs yet. I might update this line after verifying them.*\n\n**Major Weaknesses:**\n\n- The models used for the regression tasks are relatively small. Do the results still hold when a more powerful backbone architecture (e.g., a ResNet50, or even more recent architectures) are used? What was the motivation for using a ResNet18?\n- The evaluation against the baselines is thorough, but I am missing a comparison to conceptually different methods. Even if the proposed method does not outperform them, it is good to provide this as a reference. Can the authors comment on the SOTA models for each of the five benchmarks? E.g. for Table 3, I find a reference result of [around 3-5 angular error](https://paperswithcode.com/sota/gaze-estimation-on-mpii-gaze), while authors report >5. I would be interested in learning about the state of the art of each of the benchmarks (I am not experienced with these datasets, so maybe the tables already contain the applicable SOTA).\n- Since all results are for visual data and the method uses augmentations for training the embedding, I am missing a comparison to state of the art representation learning methods like DINO (Caron et al., 2021) or also older methods such as SimCLR (Chen et al., 2020). Depending on the results, these do not necessarily need to go into the main paper, but it would be good to confirm that the boost in performance can actually be attributed to the supervised contrastive learning scheme, rather than the augmentations. The comparison could be executed within the proposed evaluation protocol (train the model on the data using the respective augmentations, then evaluate - this has been partially done in Table 7 for one of the datasets) and beyond it using the \"extra\" ImageNet-pretraining data (use the pre-trained checkpoint, evaluate the feature space). I am happy to further discuss the specifics of this experiment before the authors execute it to ensure that the protocol makes sense.\n- The positioning in the literature is not accurate, and the paper title is too broad. The authors already cite two papers (and I propose three additional ones below) that perform contrastive learning on continuous labels for the purpose of regresson tasks. Please see below (\"Novelty\").\n- The paper lacks an ethics statement given the use of data involving human subjects, and the nature of the tasks (e.g., age prediction). More details on the used datasets should be added. I did not flag the paper for ethics review yet because I assume that the original datasets included such a statement.\n\n**Minor Weaknesses**\n\n- Tables 1-4 (and also others) are lacking standard deviations / confidence intervals. For readers less familar with the datasets, it is hard to judge whether the performance improvement is signficant in all cases. If the authors want to save compute, at least running the best results for 3-5 seeds would be great.\n- Details on the grid search are missing (search range, number of samples per hyperparameter, ...)\n\n**Questions:**\n\n- Are all datasets taken from an established benchmark, or is the particular experimental protocol used in the paper a novel contribution? What was the rationale for using these specific prediction tasks for evaluation?\n- How did you find the optimal temperature parameter?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DQV7cV9tPXc", "reviewer": "Reviewer_hEGc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper attempts to incorporate Contrastive Learning into regression problems. The format of contrastive loss is not changed compared with SupCon, except for the definition of positive/negative pairs within the dataset. Namely, an anchor is first defined as a landmark and \n the sample with the closest label to the anchor's label is then defined as the corresponding positive sample and other remaining samples in a mini-batch are negative samples. optimizing LSupCR will make the feature embedding ordered according to the order in the label space. This definition of positivity/negativity is used to make the feature embedding ordered according to the order in the label space. \n", "summary_of_the_review": "Here are some concerns that might need clarification:\n1. How do authors choose the anchors in a batch? Is the anchor defined randomly, or every sample has to be enumerated?\n2. How to obtain the last inequality from the second last inequality of Eq. (2)? Namely, how to derive Eq (3)?\n3. How to obtain Eq. (4)? There are many long equations that lack detailed explanation, which makes it hard to parse, although the proof is placed in the Appendix. Please add the necessary explanation of the derivation step by step. \n4. In SupCon and other related works, the feature embeddings are often normalized of length 1. In this case, the maximum distance between two samples is 2, which seems to enforce the delta in Def 1 to be greater than 0.5. This restriction may affect the feasible range of epsilon. I am wondering if the authors thought about this.\n5. In the experiments, the performance suggests that SUPCR is more robust to unforeseen data corruption. I am curious if that's because the data augmentation is used as much contrastive learning did? Would the authors want to discuss this?\n6. To obtain the smoothness and continuity of Figure 1 c, do the authors have to choose many anchors so that the embeddings can be less discrete like how SupCon shows in Figure b? So how is the training efficiency of the model?", "strength_and_weaknesses": "The paper is written clearly and organized well. Contrastive learning for regression is an interesting topic where deep learning does not well addressed fully. Experiments show noticeable improvement over the baseline in terms of accuracy/loss.\nWeakness: 1. Please clarify how to determine the anchors in a batch; or the authors just enumerate all samples and assign each sample as an anchor?\n2. The proof needs more explanation step by step in the Appendix. The theorem guarantees that the order of similarity is consistent with the order of the similarity of labels. But it is not necessary that the order can preserve the continuity and the specific distance between labels. I am curious if the authors can make some more investigation into this concern.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9h6TQCIfw4S", "reviewer": "Reviewer_ZitN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a supervised contrastive regression loss for regression tasks, different from the supervised contrastive loss used for classification. The loss term is the same as NT-Xent where in their proposed approach, positive samples of an augmented view must fall within a radius or distance in the (continuous) label space, and negatives fall outside of this distance. Subsequently, they iteratively apply this term for the first closest positive sample, second, third and so on until the N-1 farthest sample. They use this to pre-train networks, which are further fine-tuned by freezing the model and training a single linear head, as in self-supervised methods.", "summary_of_the_review": "Overall, the applicability of this approach on other datasets is not clear and although the results do seem promising it's hard to assess whether the gain is simply coming from pre-training. ", "strength_and_weaknesses": "The overall idea somewhat makes sense, but some parts are lacking.\n- Most datasets selected are image datasets, though often regression tasks are on time-series/tabular data or other data modalities where creating \"augmented\" views drastically distorts the input where the label is no longer a valid one. It needs to be clarified how this would work for such datasets, where the relationship between augmenting/transforming the input and its correspondence on the target.\n- on the TUAB dataset, what augmentations were used? How is random-crop/random flip related to age? Is the same age used for both samples? In that case, augmented views of the sample will always have label distance 0, even though the augmentation may completely distort the input. \n- What happens if the target is cyclical? This hierarchical setup for contrasting iteratively will fail for such cases unless I misunderstand parts of the algorithm.\n- What happens to the contrastive term on the furthest sample? Is this dropped? I'm assuming it is.\n- Alternate baselines that pre-train or use an SSL loss as a regularizer along with the supervised L1 loss term should be added. For example, why is this method is compared with CORN?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject