ConR: Contrastive Regularizer for Deep Imbalanced Regression
Authors: Mahsa Keramati
Lili Meng
R. David Evans
Venue: ICLR 2024 poster (2024)
Keywords: Deep imbalanced regression
Contrastive learning
Representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We proposed a contrastive regularizer to address feature collapse in deep imbalanced regression.

Abstract: Imbalanced distributions are ubiquitous in real-world data. They create constraints on Deep Neural Networks to represent the minority labels and avoid bias towards majority labels. The extensive body of imbalanced approaches address categorical label spaces but fail to effectively extend to regression problems where the label space is continuous. Local and global correlations among continuous labels provide valuable insights towards effectively modelling relationships in feature space. In this work, we propose ConR, a contrastive regularizer that models global and local label similarities in feature space and prevents the features of minority samples from being collapsed into their majority neighbours. ConR discerns the disagreements between the label space and feature space, and imposes
a penalty on these disagreements. ConR minds the continuous nature of label space with two main strategies in a contrastive manner: incorrect proximities are penalized proportionate to the label similarities and the correct ones are encouraged to model local similarities. ConR consolidates essential considerations into a generic, easy-to-integrate, and efficient method that effectively addresses deep imbalanced regression. Moreover, ConR is orthogonal to existing approaches and smoothly extends to uni- and multi-dimensional label spaces. Our comprehensive experiments show that ConR significantly boosts the performance of all the state-of-the-art methods on four large-scale deep imbalanced regression benchmarks.

Published as a conference paper at ICLR 2024
CONR: CONTRASTIVE REGULARIZER FOR DEEP IM- BALANCED REGRESSION
Mahsa Keramati' 22; Lili Meng', R. David Evans!
Borealis AI, School of Computing Science, Simon Fraser University
mkeramat@sffu ca, 1ili meng@ ccom dave. www.seeorraaalsaa..........
ABSTRACT
Imbalanced distributions are ubiquitous in real-world data. They create constraints Deep Neural Networks to represent the minority labels and avoid bias towards majority labels. The extensive body of imbalanced approaches address categor- ical label spaces but fail to effectively extend to regression problems where the label space is continuous. Local and global correlations among continuous labels provide valuable insights towards effectively modelling relationships in feature space. In this work, we propose ConR, a contrastive regularizer that models global and local label similarities in feature space and prevents the features of minority samples from being collapsed into their majority neighbours. ConR discerns the disagreements between the label space and feature space, and imposes a penalty on these disagreements. ConR addresses the continuous nature of label space with two main strategies in contrastive manner: incorrect proximities are penal- ized proportionate to the label similarities and the correct ones are encouraged to model local similarities. ConR consolidates essential considerations into generic, easy-to-integrate, and efficient method that effectively addresses deep imbalanced regression. Moreover, ConR is orthogonal to existing approaches and smoothly ex- tends to uni- and multi-dimensiin show that ConR significantly boosts the performance of all the state-of-the-ae methods four large-scale deep imbalanced regression benchmarks. Our code is publicly available in httpss:/githuub.com http:///ttuub.ooo//orraaalAAA// Borcalls
label spaces. Our comprehensive experiments
INTRODUCTION
Imbalanced data distributions, which are common in real-world contexts, introduce substantial challenges in generalizing conventional models due to variance across minority labels and bias to the majority ones (Wang et al., 202Tb; Gong et al. 2022, Buda al. 2018). Although there are numerous works on learning from imbalanced data (Chawla et al. 2002 Cui et al. 2022 Jiang et al.) 202T), these studies mainly focus on categorical labels. Continuous labels are potentially infinite, high-dimensional and hard to bin semantically (Ren et al. 2022). These characteristics impede the performance of imbalanced classification approaches on Deep Imbalanced Regression (DIR) (Yang
al. 202I).
Continuous labels result in underlying local and global correlations, which yields valuable perspec- tives towards effective representation learning of imbalanced data (Gong et al. 2022 Shin et al.) 2022). For instance, regression training on imbalanced data fails to model appropriate relation- ships for minority labels in the feature space (Yang et al., 2021). Yang al. (202T) established an empirical example of this phenomenon on the age estimation task where the learned features of under-represenned samples collapse majority features. Several approaches tackle this issue by en- couraging local dependencies (Yang et al. Steininge et al.) 202I). However, these methods fail exploit global relationships and are biased toward only learning representative features for majority samples, especially when minority examples do not have majority neighbours. RankSim (Gong et al.) 2022] leverages global and local dependencies by exploiting label similarity orders in feature space. Yet, RankSim does not generalize to all regression tasks, as not all continuous label spaces convey order relationships. For example, for mepth-map estimation from scene pictures, the complicated Work done during an intersship Borealis AL.

---

Published as a conference paper at ICLR 2024
relationships in the high-dimensional label space not trivially convertible to a linearly ordered feature space. Given the importance of the correspondence between the label space and feature effectively transfer inter-label relationships, regardless
space for imbalanced regression, of their complexity, to the feature space?
We propose method to enforce this correspon- dence: ConR is a novel Contrastive Regularizer which is based on infoNCE loss (Oord et al., 2018) but adapted for multi-positive pairs. While similar extensions are performed for clas- sification tasks (Khosla et al. 2020b). ConR addresses continuous label space and penalizes minority sample features from collapsing into the majority ones. Contrastive approaches for imbalanced classification are mainly based predefined anchors, decision boundary-basee negative pair selection and imposing discrimina- tive feature space (Cui et al. 2021] Wang et al.,
a) Without ConR
input Isbel Pred. Age
26
Feature Space
Lower weight for majority anchor 25
repuised
negallive
atlrected
2. Strength based on degree of prediction
Strondly lsboled & pushtdd more
c) With ConR
2021a Li al., 2022). These are not feasibly extendible to continuous label spaces. ConR introduces continuity contrastive learning for regression tasks. Fig I Tillustrates an intuitive ex- ample for ConR from the task of age estimation. There are images of individuals of varying ages, including 21, 25, and 80 years. Age 1 and 80 are the minority examples, reflecting the limited number of images available for these age groups within the datasets. While 21 is a majority example, given the abundance of images around this age within the datasets. Without using ConR, similarities in the feature space are not aligned with the relationships in the label space. Thus, the minority samples' features collapse ones that mimic the majority sample (A.. ConR regularizes the feature space by simultaneously encouraging locality via pulling together positive pairs and preserving global correlations by pushing negative pairs. The 21-year-old sample coexists within a region in the feature space alongside 1-year-old and 80-year-old samples. Thus, ConR considers the 21-year-old sample an anchor, and, 2) pushes negative pairs for minority anchors harder than majority examples to provide better separation in feature space. Furthermore, ConR 3) increases pushing power based on how heavily mislabelled an example is (Fig Db.1 1 to Fig [b.3). We demonstrate that ConR effectively translates label relationships to the feature space, and boosts the regression performance on minority samples (Fig De). Refer to A.1 for the empirical analysis on the motivation of ConR.
Pred.Age
Figure Key insights of ConR. a) Without ConR, it is common have minority examples mixed with majority examples. b) ConR selects the sam- ple with confusion around it an anchor and ad- justs the feature space with relative contrastive learning. c) Reduced prediction error.
the majority sample, leading to inaccurate predictions for minority
ConR implicitly models the local and global correlations feature space by three main contributions to the contrastive objective: 1) Dynamic anchor selection: Throughout the training, considering the learned proximity correspondences, ConR selects samples with the most collapses on the feature manifold as anchors. 2) Negative Pairs selection: Negative pairs are sampled to quantify the deviation they introduce to the correspondence of similarities in feature and label space, thereby compensating for under-represented samples. 3) Relative pushing: Negative pairs are pushed away proportional to their label similarity and the density of the anchor's 's label.
ConR is orthogonal to other imbalanced learning techniques and performs seamlessly high- dimensional label spaces. Our comprehensive experiments on large-scale DIR benchmarks for facial age, depth and gaze estimation show that ConR strikes balance between efficiency and performance, especially on depth estimation, which has complicated and high nigh-dimensional label space.
2 RELATED WORK
Imbalanced classification. The main focus of existing imbalanced approaches is on the classification task, while imbalanced regression is under-explored. The imbalanced classification methods are either
or model-driven. Resampling (Chawla et al. 2002 Chu et al., 20200 Byrd &
Lipton 2019 Han al. 2005 Jiang et al. 2021) is data-driven technique that balances the input
2

---

Published as a conference paper at ICLR 2024
data by either aver-sampiing (Chawla et al., 2002: Byrd & Lipton 2019) the minority classes or under-sampling the majority ones (Han et al. 2005). Aware (MAK) (Jiang et al.)
2021) over-samples tail classes using external sampling pool. Another branch of data-driven approaches is augmentation- based methods. Mixup-based approaches linearly interpolate data either in input space (Zhang et al. 2017) or feature space (Verma et al. 2019) for vicinal risk minimization. RISDA (Chen et al. (022) shares statistics between classes considering a confusion-based knowledge graph and implicitly augments minority classes. Model-driven methods such as focal loss (Lin et al. 2017) and bogit-adjustman
(Tian et al., (020) are cost-sensitive approaches that regulate the loss functions regarding the class labels. To encourage learning an unbiased feature space, several training strategies including two-stage training (Kang et al., 2020), and transfer learning (Yin et al. (019) are employed. As discussed further, though there has been much success in this area, there are many issues in converting imbalanced classification approaches to regression.
Imbalanced regression. Unlike classification tasks with the objective of learning discriminative
representation, effective imbalanced learning in continuous label space is in lieu of modelling the label relationships in the feature space (Yang et al. 202T: Gong et al. 2022). Therefore, imbalanced
classification approaches do not feasibly extend to the continuous label space. DenseLoss (Steininger
jet al. 202I) and LDS (Yang et al.] (222)) encourage local similarities by applying kernel smoothing
in label space. Feature distribution smoothing (FDS) (Yang et al. (02T) extends the idea of kernel smoothing to the feature space. Ranksim (Gong et al.) (022) proposes to exploit both local and global dependencies by encouraging a correspondence of similarity order between labels and features. Balanced MSE (Ren et al. 2022) prevents Mcan Squared Error (MSE) from carrying imbalance to the prediction phase by restoring a balanced prediction distribution. The empirical observations in (Yang) et al. )02T) and highlighted in VIR (Wang & Wang 2023), demonstrate that using empirical label distribution does not accurately reflect the real label density in regression tasks, unlike classification tasks. Thus, traditional re- weighttt techniques in regression tasks face limitations. Consequently, LDS (Yang et al. 202T) and the concurrent work, VIR propose to estimate effective label distribution. Despite their valuable success, two main issues arise, specifically for complicated label spaces. These approaches rely on binning the label space to share local statistics. However, while effective in capturing local label relationships, they disregard global correspondences. Furthermore, in complex and nigh-dimensional label spaces, achieving effective binning and statistics sharing demands in- depth domain knowledge. Even with this knowledge, the task becomes intricate and may not easily extend to different label spaces.
Contrastive learning. Contrastive Learning approaches are pairwise representation learning tech-
niques that push away semantically divergent samples and pull together similar ones (He et al. 2020 Chen al. 2020a| Khosla et al., 2020aa Kang et al., 202 Momentum Contrast (Moco) (He
et al.) 2020) is an unsupervised contrastive learning approach that provides large set of negative samples via introducing a dynamic queue and a moving-
encoder; while SupCon (Khosla et al. 2020b) incorporates label information to the contrastive learning. Kang et al. (2021a) la argue that in case of imbalanced data, SupCon is subject to learning a feature space biased to majority samples and proposed k-positive contrastive loss (KCL) to choose the same number of positive samples for each instance to alleviate this bias. Contrastive long-tailed classification methods train parametric learnable class centres (Cui et al. 2021 Wang et al., 2021a) la) encourage learning regular simplex (Li et al., 2022). However, these approaches cannot be used for regression tasks where handling label-wise prototypes is potentially complex. Contrastive Regression (CR) (Wang) eet al. 2022) adds a contrastive loss to Mean Squared Error (MAE) to improve domain adaptation for the gaze estimation task. CR assumes there a correspondence between similarities in label space and the feature space. Regardless, when learning from the imbalanced data, the correspondence can't 't be assumed. Instead of the assumption of correspondence, ConR translates the relative similarities from the label space to the feature space. |Barbano et al. (202)) proposed a relative pushing for the task of brain age prediction using MRI scans. This relative pushing doesn't consider imbalanced distribution, and the weights are learnable kernels that can introduce complexities to complex label spaces. Rank-N-Contral regarding their relative rankings to impose continuous representations for regression tasks. Like RankSim Gong et al. (2022). Lank-N-Contren is designed based on the order assumption and cannot be used in all regression tasks.
(Zhaa et al., 2023) ranks samples and contrasts them against each other
3

---

Published as a conference paper at ICLR 2024
a) Pair Selection
b) Relative Pushing
Yes
postive pair
No. of samples
close lebels?
polential pair
Label
Yes
negative pair
close prediction?
label
regression
No
pair-up
encodar
function
X
postive
anchor
space
Figure 2: The framework ConR to translate label similarities to the feature space. a) Per each augmented sample, ConR selects positive and negative pairs with regard to the label similarities and prediction similarities. b) ConR pulls positive pairs together while pushing away negative pairs regarding their label similarities and label distribution for the anchor. In this way, the minority anchor pushes negative samples harder. The pushing weight is inversely relative to the label similarities.
METHOD
3.1
PROBLEM DEFINITION
Consider training dataset consisting N examples, which denote {(2,, o where x, Rd an example input, and y, E RRd is its corresponding label. d and d' are the dimensions of the input and label, respectively. We additionally enforce that the distribution of the labels Dg deviates significantly from uniform distribution. Given model that consists of a feature encoder E-), and regression function R.).,, the objective is to train regression model R-E(-)), such that the model output i; R(E((:)() (x:)) is similar to the true label yi-
3.2 IMBALANCED CONTRASTIVE REGRESSION
In regression tasks, inter-label relationships unveil meaningful associations in feature space. However, learning from imbalanced data harms this correspondence since the features learned for minority samples share statistics with the majority samples, despite dissimilar labels (Yang et al. (02T). By incorporating label and prediction relationships into contrastive learning, ConR enforces appropriate feature-level similarities for modelling balanced feature space.
ConR is a continuous variation infoNCE. Initially, for creating diversity in examples, we perform mroblem-ppccfiie augmentations each input x, to produce two augmented samples. We define the set of augmented examples from the input to be {(0). y3)} ? where 2i is an augmented input.
As illustrated in Fig. ConR is collaboration of pair selection and relative pushing. For each augmented sample, ConR first selects pairs. Next, in the case of at least one negative pairing, the sample is considered an anchor and contributes the regularizing process of ConR by pulling together positive pairs and relatively repelling negative pairs.
3.2.1 PAIR SELECTION
Given a pair of examples (a?,y; y/: and (x),,y) from the augmented inputs (labels and samples, Fig. a), each example is passed to the feature encoder E-) produce feature vectors z; and zj. and then to the regression function R(-) for predictions D: and ÿ (sample through regression function, Fig. a). The values of the predicted and actual labels are used to determine the examples should be positive pair, negative pair, unpaired (righthand side, Fig. a).
To measure similarity between labels (or predictions) of augmented examples, ConR defines a similarity threshold Given similarity function Sim(-,-) R (e.g., inverse square error), define two labels (or two predictions) y; and as similar if Sim (v,, y;) 2 w. We denote this as y: yj- Iff two examples have similar labels y; yj. then they are treated as positive pair. The
4

---

Published as a conference paper at ICLR 2024
examples have dissimilar labels, but similar predictions / = Dy. then they treated as a negative pair. Otherwise, examples are umpaired.
Anchor selection. For each example j. (19 v)) We denote the sets of positive and negative pairs
for this example, and their feature vectors as KJ = {<-p)r and K = { respectively, where NJ. is the number of positive examples and 5 is the number of negative samples for example j. If j > O, (z)) is selected as an anchor and contributes to the regularization process of ConR.
3.2.2 CONTRASTIVE REGULARIZER
For each example j. ConR introduces loss function LConRj. example j is not selected as an anchor, LCon R, 0. Otherwise, LCmn pulls together positive pairs while pushing away negative pairs proportional to the similarity of their labels. As shown Fig. b.. ConR pushes away negative samples with less label similarity to yj harder than negative samples with labels closer to the anchor Vj:
LConty log Σ Σ _ NJ
exp(zz 2//T)
ExyeK, exp(2) pp/T) + Ee,EK Sg exp(2) 2a/T) K,EK
where T a temperature hyperparameter and Snnn is a pushing weight for each negative pair:
Sja= - Ss(1 Sim My)),
and By is the label of F.g where 24 = Exrg)) 17, is a pushing power for each sample (a33 Mj) that depends on label distribution D, to boost the pushing power for minority samples (Fig. 2):: :: ny x wj. where Wy is density-based weight for input j derived from the empirical label distribution (e.g.. inverse frequency). The function Js computes the Sj.9 to be proportionate f and inversely related to Sim(y,, 3q). Please refer to Appendix A.4 for the definition fs.
Finally, LCoRR is the ConR's s regularizer value for the augmented example set:
2N
LCooR
Σ
CConRj
2NN
1=0
To prevent the representations of minority labels from collapsing to the majority ones in deep imbalanced regression, Lsum is optimized. Leam is weighed sum of LoouR and regression loss LR (c.g., mean absolute eтror) as below:
Tyum aLR + BL
3.3 THEBRRETCALL INSIGITS
We theoretically justify the effectiveness of ConR in Appendix B We derive the upper bound LCOR + the probability of incorrect labelling of minority samples to be:
2N
log S S),p((Y((z))) S CConst + te, No00
(5)
4
E=
where the set of anchors and x, is negative sample. p(Ý, (2,,)) the likelihood of sample z with an incorrect prediction E Ý = (0, w. Ú, + w). új a prediction that is mistakenly similar to its negative pair Jg. We refer to p( (()(,,))) the probability of collapse for Tq. The Left-Hand Side (LHS) contains the probability of all collapses for all negative samples, which is bounded above by LCooR. Minimizing the loss, consequently minimizes the LHS, which causes either 1) the number of anchors decreases or 2) the degree of collapses is reduced. Each p(Y, [zg) is weighted by Sy.g. which penalizes incorrect predictions proportional to severity. Thus, optimizing ConR decreases the probability of mislabelling proportional the degree, and emphasizes minority samples.
5

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
4.1 MAIN RESULTS
Datasets and baselines. We use three datasets curated by Yang et al. (2021) for the deep im- balanced regression problem: AgeDB-DIR is a facial age estimation benchmark, created based on AgeDB (Moschoglou et al., 2017).
is an age estimation dataset originated from IMDB-WIKI (Rothe et al. 2018 NYUD2-DIR is created based NYU Depth Dataset V2 (Silberman et al. 2012) to predict the depth maps from RGB indoor scenes. Moreover, we create MPIIGaze-DIR based on MPIIGaze, which is an appearance- gaze estimation benchmark. Refer to Appendix A.2 and A.4 for more dataset and implementation details. Please refer to Appendix A.3 for baseline details.
Evaluation process and metrics. Following the standard procedure for imbalanced learning (Yagg et al. [202T] Gong al.] (022). we report the results for four shots: All, median and many. The whole test data is denoted by All. Based on the number of samples in the training dataset, few, median and many that have less than 20, between 20 and 100, and more than 100 training samples per label,
respectively.
For AgeDB-DIR and IMDB-W WIKI-DIR, the metries are Alson-Absouute-Err (MAE), and Geometric Mean (GM). For NYUD2-DIR we use Root Mean Squared Error (RMSE) and Threshold accuracy (81) as metries in (Yang et al./ of d; that satisfies max
2021 Ren et al., 2022). Threshold accuracy 6; the percentage 1.25, where for each pixel, 91 is the ground truth depth value and d1 is the predicted depth value. For MPIIGaze-DIR, we use Mean Angle Error (degrees). To calculate relative improvement of ConR, we compare the performance of each combination of methods including ConR, against the same combination without ConR. Each "Ours vs. " entry shows the average of these improvements (e.g. "Ours vs. LDS" is the average of the improvements of adding ConR to each combination of baselines that has LDS).
di 41
gi di
Main results for age estimation. Table and Table show the results on AgeDB-DIR and IMDB- WIKI-DIR benchmarks, respectively. We compare the performance of DIR methods: FDS, LDS and RankSim with their regularized version by ConR. All results are the averages of 5 random runs. We observe that the performances of all the baselines are considcrably boosted when they are regularized with ConR. In addition, ConR results in the best performance across both metrics and all shots for the AgeDB-DIR benchmark with leading MAE results of 6.81 and 9.21 on the overall test set and ConR achieves the highest performance
few-shot region, respectively. For IMDB-WIKI-DE out of 8 shot/metric combinations with the best MAE results of 7.29 and 21.32 on the overall test set and and shot region, respectively.
Main results for depth estimation. To assess the effectiveness of ConR in more challenging settings where the label space is high-dimensional with non-linear inter-label relationships. we compare its performance with baselines: LDS, FDS and Balanced MSE on NYUD2-DIR depth estimation benchmark. For this task, the label similarity is measured by the difference between the average depth value of two samples. As shown in table ConR alleviates the bias of the baseline toward the majority samples. ConR significantly outperiorms LDS, FDS and Balanced MSE across all shots and both metrics with leading RMSE results of 1. 1.265 on the overall test set and 1.667 on the few-shot region. Notably, RankSim cannot be used on the depth estimation task; however, ConR can smoothly be extended to high nigh-dimensional label space with high performance. The reason is that order relationships are not semantically meaningful for all regression tasks with high-dimensional label space, such depth estimation.
Main results for gaze estimation. Table compares the performance of ConR with three baseline methods: LDS, FDS and Balanced MSE the MPIIGaze-DII benchmark. As shown in table 4 ConR consistently improve the performance of all basclines for all shots and achieves the best Mcan angular error of 5.63 (degrees) on the overall test set and 5.21 (degrees) on the few-shot region.
Error reduction. In Fig. B we show the comparison between three strong baselines (LDS, FDS and Balanced MSE) and by adding ConR for NYUD2-DIR benchmark. It shows consistent and notable error reduction effect by adding our ConR to the deep imbalanced learning. For more results and analysis other datasets, please refer Appendix A.5

---

Published as a conference paper at ICLR 2024
Table Main results for AgeDB-DIR benchmark. Results are reported for the whole test data (all) and three other shots: many, median, and few. Each section compares a baseline with its regularized version of ConR. At the bottom of the table, the average improvements with respect to corresponding baselines without ConR are reported in Green, The best result of either baseline or its regularized
version by ConR bold (in column) and Red (across column), Baselines: (Gong et al. 2022).
Metries
MAEL
GML
MethodsSShots
All
Many
Median
Few
All
Many
Median
Few
ConR only (Ours)
7.20
6.50
8.04
9.73
4.59
3.94
4.83
6.39
LDS
7.42
6.83
8.21
10.79
4.83
4.39
5.80
7.03
LDS ConR (Ours)
7.16
7.97
9.62
4.51
4.21
4.92
FDS
7.53
6.99
8.40
10.48
4.82
4.49
3.47
6.58
FDS ConR (Ours)
7.08
6.46
7.89
9,80
4.31
4.01
5.25
6.92
RankSim
6.91
6.34
7.79
9.89
4.28
3.92
4.88
6.89
RankSim ConR (Ours)
6.84
6.31
7.65
9.72
4.21
3,95
4.75
6.28
LDS + FDS
7.37
6.82
8.25
10.16
4.72
4.33
5.50
6.98
LDS + FDS + ConR (Ours)
7,21
6.88
7.63
9.59
4.63
4.45
5.18
5.91
LDS RankSim
6.99
6.38
7.88
10.23
4.40
3.97
5.30
LDS RankSim ConR (Ours)
6.88
6.35
7.69
9.99
4.43
3.87
4.70
6.51
FDS RankSim
7.02
6.49
7.84
9.68
4,53
4.13
5.37
6.89
FDS RankSim+ ConR (Ours)
6.97
6.33
7.71
9.43
4.19
3.92
4.67
6.14
LDS FDS RankSim
7.03
6.54
7.68
9.92
4.45
4.07
5.23
6.35
LDS FDS RankSim + ConR (Ours)
6.81
6.32
7.45
9.21
4.39
3.81
5.01
6.02
Ours LDS
2.58%
L.11%
3.97%
6,35%
2.39%
2.43%
9.15%
10.78%
Ours vx. FDS
3.07%
3.14%
4,57%
5.47%
5.34%
4,85%
6,78 %
6.56%
Ours vs. RankSim
1.62%
1.70%
2.24%
3.49%
2.45%
3.31%
8.14%
7.79%
Table 2: Main results on IMDB-WIKI-DIR benchmark.
Metrics
MAEJ
GM.
Methods/Shols
All
Many
Median
Few
All
Many
Median
Few
ConR-onny (Ours)
7.33
6.75
11.99
22.22
4.02
3.79
6.98
12.95
LDS
12.43
4.42
4.19
7.00
13.94
LDS ConR (Ours)
7.43
6.84
12.38
21.98
4.06
3,94
6,83
12.89
FDS
7,83
7.23
12.60
22.37
4.42
4,20
6.93
13.48
FDS ConR (Ours)
6.90
12.01
21.72
4.02
3.83
6.71
12.59
RankSim
6.84
12.12
22.13
4.10
3.87
6.74
12.78
RankSim + ConR (Ours)
7.33
6.69
11.87
21.53
3,99
3,81
6.66
12.62
LDS + FDS
7.78
7.20
12.61
22.19
4.37
4.12
7.39
12.61
LDS FDS ConR (Ours)
12.31
21.81
4.07
3.96
12.86
LDS RankSim
7.57
7.00
12.16
22.44
4.23
4.00
6.81
13.23
LDS RankSim ConR (Ours)
7.48
6.79
12.03
22.31
4.04
3,86
6.77
12.80
FDS RankSim
7,50
6.93
12.09
21.68
4.19
3.97
6.65
13.28
RankSim+ ConR (Ours)
11.79
4.05
3.88
6.53
12.67
LDS FDS RankSim
7.69
7.13
12.30
21.43
4.34
4.13
6.72
12.48
LDS FDS RankSim + ConR (Ours)
7.46
6.98
12.25
21.39
4.19
4.01
6.75
12.54
Ours LDS
3.67%
3.54%
1.07%
1.21%
5.75%
4.07%
2.37%
2.08%
Ours vs. HDS
4.00%
2.91%
2,49%
1.625
5.68%
4,47%
2,86 16
2.18%
Ours vs. RankSim
1.55%
2.37%
1.51%
1.29%
3.50%
2.56%
0,79%
2.16%
Table 3: Main results on NYUD2-DIR benchmark.
Metrics
RMSET
0.1
Methods/Shots
All
Many
Median
Few
All
Many
Median
Few
ConR-only (Ours)
1.304
0.682
0.889
1,885
0.675
0.699
0.753
0.648
0.615
0.940
2,059
0.681
0.760
FDS + Coon (Ours)
1.299
0.613
0.836
1.825
0.696
0.800
0.819
0.701
LDS
1.387
0.671
0.913
1.954
0.672
0.701
0.706
0.630
LDS ConR (Ours)
1,323
0.786
0.823
1.852
0.700
0.632
0.827
0.702
RankSim
Balanced MSE (BNI)
0.787
0.870
1,736
0.694
0.622
Balanced MSE (BNI) ConR (Ours)
1,265
0.772
0.809
1.689
0.705
0.631
0.832
0.698
Balanced MSE (BNI) LDS
1.319
0.810
0.920
1.820
0.681
0.601
0.695
0.648
Balanced MSE (BNI) + 1.DS ConR (Ours)
0.893
0.699
0.652
0.752
Ours vs. LDS
4.13%
4.25%
10.41%
6.81%
3.41%
0.07%
13.31%
13.74%
Ours FDS
9.92%
0.03%
11.06%
11.37%
2.20%
5.27%
17.84%
17.62%
Ours Vs. Balanced MSE (BNI)
2.52%
6.33%
8.99%
18.425
2.11%
4.97%
6.36%
6.30%
Time consumption Analysis. Table 5lolvide the time consumption of ConR in comparison to other baselines and VANILLA for the age estimation and depth estimation tasks. VANILLA is a regression model with no technique for imbalanced learning. The reported time consumptions
7

---

Published as a conference paper at ICLR 2024
Table 4: Main results of on MPIIGaze-DIR benchmark.
Metric
Mean angular error (degrees).
Method/Shot
All
Many
Median
Few
ConR (Ours)
6.16
5.73
6.85
6.17
LDS
6.48
LDS ConR (Ours)
6.08
5.76
6.81
5,98
FDS
6.71
6.01
7.50
6.64
FDS ConR (Ours)
6.32
5.69
6.56
5.65
Balunced MSE (BNI)
5.73
6.34
6.41
5.36
Balanced MSE (BND+ ConR (Ours)
5.63
6.02
6.17
5.21
Ours vs. LDS
6.17
2.87%
6,46%
12,45 %
Ours vs. FDS
5.81 %
5,32%
12.53%
14.91 %
Ours vs. Balanced MSE
1.75 %
5.05%
3.75%
2.80 %
2004
1.9
LDS Conk
LDS
TDS
FDS
Conft
Delenced MSC Balanced MSE Conft
1.50
0.1
6.75
0:4
0.2
Many
Manyy
Nedion
Few
Al1
Manw
Median
Rew
Figure 3: comparison on RMSE results by adding ConR on top of the basclines for NYUD2-DIR benchmark.
expressed in seconds for AgeDB-DIR and in minutes for NYUD2-DIR, representing the average forward pass and training time, and were measured using four NVIDIA GeForce GTX 1080 Ti GPUs. 3]s findings demonstrate that even with high-dimensions time is considerably lower than FDS while remaining comparable to the time complexity of LDS, RankSim, and Balanced MSE. This indicates that ConR is efficient alternative to FDS without compromising efficiency compared to other well-established methods. This result highlights ConR's ability to handle complex tasks without introducing significant computational overhead.
label space, ConR's 's training
Table 5: Time consumption for AgeDB-DIR and NYU2D-DIR benchmarks.
Benchmark
AgeDB-DIR
NYUD2-DIR
Method/Metric
Forward pass (S)
Training time (8)
Forward pass (m)
Training time (m)
VANILLA
12.2
31.5
7.5
28.3
LDS
12.7
34.3
30.2
FDS
38.4
60.5
10.8
66.3
RankSim
16.8
38.8
Balanced MSE
16.2
36.3
7.9
29.4
ConR (Ours)
15.7
33.4
8.1
30.4
Feature visualizations. We evaluate ConR by comparing its learned representations with VANILLA, FDS, and LDS. Using t-SNE visualization, we map ResNet-50's features to a 2D space for the AgeDB DIR dataset. Fig. 4 demonstrates the feature-label semantic correspondence exploited by ConR, compared to VANILLA, FDS and LDS. VANILLA fails to effectively model the feature space regarding three key observations: a) high occurrences of collapses: features of minority samples are considerably collapsed the majority ones. b) low relative spread: contradicting the linearity in the age estimation task's label space, the learned representation exhibits low feature variance across the label spread (across the colour spectrum) compared to the variance across a single label (same colour). Noticeable gaps within the feature space: contradicts the intended continuity in regression tasks. Compared to VANILLA, FDS and LDS slightly alleviate the semantic confusion in feature space. However, shown in Fig. 4d ConR learns considerably more effective feature space with fewer collapses, higher relative spread and semantic continuity.
4.2 ABLATION STUDIES ON DESIGN MODULES
Negative sampling, pushing weight and pushing power analysis. Here we assess the significance of our method's contributions through the evaluation of several variations of ConR and investigating the impact of different choices of the hyperparameters in ConR. We define four versions of ConR: Contrastive-Cont contrastive regularizer where negative peers selected only based label
8

---

Published as a conference paper at ICLR 2024
(a) VANILLA
(b) FDS
(c) LDS
(d) ConR
Figure 4: Feature visualization AgeDB-Dil DIR for (a) VANILLA, (b) FDS, (c) LDS and (d) ConR.
ConR with no pushing power assigned to the negative pairs. ConR-q ConR with pushing powers that are not proportionate to the instance weights. ConR-Sim: ConR with pushing powers that are not proportionate to the label similarities. Table 6 describes the comparison between these three versions on the AgeDB-DIR benchmark and shows the crucial role of each component in deep imbalanced regression. Contrastive-Con of SupCon (Khosla et al., 2020b) that is biased to majority samples (Kang et al., (021b). Thus, Contrastive- ConR shows better results many shot that is due to the higher performance for majority samples, while it degrades the performance for minority ones. However, ConR results in a more balanced performance with significant improvement for the minoirty shots.
similarities. ConR-S:
is the continuous version
Table Ablation results on AgeDB-DIR benchmark.
Metrics
MAE
Methods/Shots
Many
Median
Few
Contrastive-Conk
7.69
6.12
8.73
12.42
ConR-S
7.51
6.49
8.23
10.86
ConR-Sim
7.54
6.43
8.19
10.69
ConR-r
7.52
6.55
8.11
10.51
ConR
7.48
6.53
8.03
10.42
Similarity threshold analysis. We investigate the choice of similarity threshold w by exploring the learned features and model performance employing different values for the similarity threshold. Fig. 5a and Fig- 5b compare the feature space learnt with similarity threshold = 2 and w = 1 for ConR on the AgeDB- B-DIR benchmark. ConR implicitly enforces feature smoothness and linearity in feature space. A high threshold (w 2) is prone encouraging feature smoothing in limited label range and bias towards majority samples. As illustrated in Fig. 5b choosing a lower similarity threshold leads to smoother and more linear feature space. Fig. demonntratss the ablation results for the similarity threshold on the Age-DB-DIR dataset. For AgeDB-DIR, w = produces the best performance. An intuitive explanation could be the higher thresholds will impose sharing feature statistics to an extent that is not in correspondence with the heterogeneous dynamics in the feature space. For example, in the task of facial age estimation, the ageing patterns of teenagers are different from people in their 20s. For more details on this study please refer to Appendix A.6
MAX
3
0.5
(a) Visualization for + = 0.5 (b) Visualization for = (c) Similarity threshold analysis.
Figure 5: Ablation study on the similarity threshold w. (a) and (b) compares the learnt feature space for similarity threshold of and respectively. (c) Comparison of different choices of w in terms of MAE
5 CONCLUSION
In this work, we propose ConR, a novel regularizer for DIR that incorporates continuity to contrastive learning and implicitly encourages preserving local and global semantic relations in the feature space without assumptions about inter-label dependencies. The novel anchor selection proposed in this work consistently derives balanced training focus across the imbalanced training distribution. ConR is orthogonal to all regression models. Our experiments on uni- and duutti-dimensional DIR benchmarks show that regularizing regression model with ConR considerably lifts its performance, especially for the represeeede samples and high- high-dimensional label spaces. ConR opens a new perspective to contrastive regression imbalanced data.

---

Published as a conference paper at ICLR 2024
REFERENCES
Carlo Alberto Barbano, Benoit Dufumier, Edouard Duchesnay, Marco Grangetto, and Pietro Gori. Contrastive learning for regression in multi-site brain age prediction. In 2023 IEEE 20th International Symposium Biomedical Imaging (ISBI, pp. 1-4. IEEE, 2023.
Mateusz Buda, Atsuto Maki, and Macicj A Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks, 106:249-259, 2018.
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In International conference machine learning, pp. 872-881. PMLR, 2019.
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and W Philip Kegelmeyer, Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for con- trastive learning of visual representations. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020a.
Xiaohua Chen, Yucan Zhou, Dayan Wu, Wanqian Zhang, Yu Zhou, Bo Li, and Weiping Wang. Imagine by reasoning: A reasoning- based implicit semantic data augmentation for long-tailed classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 356-364, 2022.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020b.
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling, Feature space augmentation for long-tailed data. In Computer ViCV 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings. Part XXIX 16, pp. 694-710. Springer, 2020.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 715-724, 2021.
Yu Gong, Greg Mori, and Frederick Tung. Ranksim: Ranking similarity regularization for deep imbalanced regression. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 7634-7649. PMLR, 2022.
Hui Han, Wen- Wen-Yuan Wang, and Bing-Huan Mao. Borderline-smote: a over-sampling method in imbalanced data sets learning. In Advances in Intelligent Computing: International Conference on Intelligent Computing, ICIC 2005, Hefei, China, August 23-26, 2005, Proceedings, Part I I, pp. 878-887. Springer, 2005.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729 9738, 2020.
Junjie Hu, Mete Ozay, Yan Zhang, and Takayuki Okatani. Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries. In 2019 IEEE winter conference on applications of computer vision (WACV), pp. 1043-1051. IEEE, 2019.
Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Improving contrastive learning on imbalanced data via open-world sampling. Advances in Neural Information Processing Systems, 34:5997-6009, 2021.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2020.
Bingyi Kang. Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference on Learning Representations, 2021a.
10

---

Published as a conference paper at ICLR 2024
Bingyi Kang. Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. International Conference on Learning Representations, 2021b.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishman. Supervised contrastive learning. Advances in neural information processing systems. 33:18661-18673, 2020a.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. H. Larochelle, M. Ran- zato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 18661-18673. Curran Associates, Inc., 2020b.
Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6918-6928, 2022.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2999-3007, 2017. doi: 10. 10.110//CC22201..2244
Edward James McShane. Jensen's 's inequality. 1937.
Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 51-59, 2017.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint XrXiv: 0074488 2018.
Jiawei Ren, Mingyuan Zhang, Cunjun Yu, and Ziwei Liu. Balanced mse for imbalanced visual regres- sion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7926 7935, 2022.
Rasmus Rothe, Radu Timofte, and Lue Van Gool. Deep expectation of real and apparent age from a single image without facial landmarks. International Journal of Computer Vision, 126(2-4): 144-157, 2018.
Nyeong- Ho Shin, Seon-Ho Ho Lee, and Chang-Su Kim. Moving window regression: novel approach to ordinal regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18760-18769, 2022.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part 12, pp- 746-760. Springer, 2012.
Michael Steininger, Konstantin Kobs, Padraig Davidson, Anna Krause, and Andreas Hotho. Density- based weighting for imbalanced regression. Machine Learning, 110:2187-2211 2021.
Junjiao Tian, Yen-Cheng Liu, Nathaniel Glaser, Yen-Chang Hsu, and Zsolt Kira. Posterior re- calibration for imbalanced datasets. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 8101-8113 Curran Associates, Inc., 2020.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International conference on machine learning, pp. 6438-644 6447. PMLR, 2019.
Peng Wang, Kai Han, Xiu-Shen Wei, Lei Zhang, and Lei Wang. Contrastive learning based hybrid networks for loigetailed image classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 943- 2021a.
11

---

Published as a conference paper at ICLR 2024
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. Long-tailed recognition by rout- ing diverse distribution-awaan aware experts. In International Conference on Learning Representations. 2021b. URL BRRlhttpps///peneev.ees net/ wot.ee/eooorrrii-11111333333.3..
Yaoming Wang, Yangzhou Jiang, Jin Li, Bingbing Ni, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Teng Li, Contrastive regression for domain adaptation on gaze estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19376-19385, 2022.
Ziyan Wang and Hao Wang. Variational imbalanced regression: Fair uncertainty quantification via probabilistic smoothing. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Yuzhe Yang. Kaiwen Zha, Yingcong Chen, Hao Wang, and Dina Katabi. Delving into deep imbal- anced regression. In International Conference Machine Learning, pp. 11842-11851 11851. PMLR, 2021.
Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learning for face recognition with under-represented data. Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp.
2019.
Kaiwen Zha, Peng Cao, Jeany Son, Yuzhe Yang. and Dina Katabi. continuous representations for regression. Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Learning
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 17X0:01710 2017.
12

---

Published as a conference paper at ICLR 2024
A EXPERIMENTS
A.1 EMPIRICAL ANALYSIS OF THE MOTIVATION
The key motivation of the design of ConR is that features of minority samples tend collapse to their majority neighbours. ConR highlights these situations by defining a penalty based on the misalignments between prediction similarities and label similarities. Further, ConR regularizes regression model by minimizing the defined penalty in a contrastive manner.
Here we first define the penalty P(y) for each prediction value y E Dy. P(y) denotes the av- erage of the regression errors of the samples with the similar prediction value of y: P(y) = N, Ei=o ( Vs), where (x)()))) = y. Vi a y and Ny is the number of the samples collapsed to the feature space of samples labelled with y.
To empirically confirm the motivation behind ConR, we investigate the importance of the penalty term in regression tasks. In addition, we show that regularizing a regression model with ConR consistently alleviate the penalty term defined by ConR and this optimization considerably contributes to imbalanced regression. Fig. 6a and Fig. 6b demonstratee the comparison between LD$ and ConR in terms of the training loss curve and validation loss curve, respectively. Moreover, Fig. Fog [o trend of the expected value of P(y) over the label space, throughout the training. Comparing Fig. 6cc with Fig. [6 Fig. [bb] P(y) follows the same decreasing pattern as training loss and validation loss. This observation show that penalizing the penalty term is highly coupled with the learning process. In addition, Fig. Tig.Gsshoe shows that ConR outperform LDS with a considerable gap, particularly in terms of P(y): showing that ConR regularizer consistently alleviates the penalty and significantly contributes the imbalanced regression.
shows the
MAE 20
MAE
8,(9)
ConR
15
CanR
22
10
10
epoch
epoch
epoch
80
80
(a) Training loss
(b) Validation loss
(c) Expected value of penalty
Figure 6: Comparison of the performance of LDS and ConR on AgeDB-DIR benchmark in terms of
(a)training loss, (b)validation loss, and (c)regularizer penalty.
Fig. 7a shows the training label distribution and Fig. 7b depicts the difference of P(y) across the label space between ConR and LDS. It empirically shows the considerable improvement of ConR over LDS in terms of the defined penalty. More improvement over the majority of samples is intuitive because due to the imbalanced distribution, most of the collapses in feature space happen in the majority areas and decreasing the penalty these areas contributes the most to the imbalanced regression. Finally, Fig. [cccooeares the regression error of ConR and LDS and we observe ConR results in a considerable improvement over LDS, especially for minority samples.
A.2 DATASET DRTAILS
Age estimation. We evaluated our method on two DIR benchmarks for age estimation curated by Yang et al. (202T):
and AgeDB-DIR. IMDB-WIK' (Rothe et al. (018)) has 191.5K images for training, and 11.0K images for validation and testing, respectively. To structure IMDB-WIKI WIDB-WIKIDDIR, Yang et al. (2021) bin the label space with bin length of 1 year, where the minimum age is 0 and the maximum age is 186. The bin density varies between 1 and 7.149. The AgeDB dataset (Moschoglou et al. 2017) has 16,488 samples Yang et al. (202I) constructed AgeDB- DIR in a similar manner as IMBB WIKI-DIR, where the minimum age is O and the maximum age is 101. The maximum number of samples per bin is 353 images and the minimum bin density is 1.
13

---

Published as a conference paper at ICLR 2024
of Trainno samplee
diference
MAE diference
20
50
80
40
102
Target Valua
Target Value
Target Value
(a)
Training loss
Validation loss
(c) Expected value of penalty
Figure 7: Qualitative analysis of ConR in terms of regularizer penalty on AgeDB-DIR benchmark. (a) training label distribution. (b) Penalty difference (L.DS minus ConR) on balanced test data. (c) MAE/ difference (LDS minus ConR) on balanced test data.
There are 12,208 training samples and the validation set and test set are made balanced with 2,140 samples each.
Depth estimation. Yang et al. 2021 created NYUD2-DIR based the NYU Depth Dataset V2 Silbbrran et al. (2012). NYU Depth Dataset V2 has images and corresponding depth maps for different indoor scenes and the task is to predict the depth maps from the RGB scene images. The upper bound of the depth maps is 10 meters and the lower bound is 0.7 meters. Following standard practices. There are 50K images for training and 654 images for testing. Yang et al.) (2021) use the bin length of 0. I-meter bin density varies between 1.13 x 106 and 1.46 x 108. For a balanced test set, Yang et al. [202T randomly select 9.357 test pixels for each bin from 654 test images with total of 8.70 X 105 test pixels in the NYUD2-DIR test set.
Gaze estimation. We used a subset of the MPIIGaze dataset comprising 45,000 training samples from 15 individuals, with 3,000 samples per person. The dataset is naturally imbalanced over the 2D training label distribution.
Here we provide the details of deriving the proposed MPIIGaze-DIR benchmark from the MPIIGaze dataset. the label space of MPIIGaze is 2-dimensional with one dimension ranging from -0.39 0.08 and the other from -0.72 to 0.67. Fig. [a shows the imbalanced distribution across each dimension separately and Fig. [b] the joint distribution of the label space with grid size of 10. Considering the joint distribution, we define thresholds on the bin densities to curate shots (i.e. many, median, and few) as follows: 1300 or more for the many-shot, 700 to 1 1300 for the median-shot and less than 700 for the sho-shot. Next, with the code snippet provided Fig. We assign the samples to their corresponding shots.
2009
(a) Individual distribution
(b) Joint distribution
Figure 8: Distribution of the 2-dimensional label space of MPIIGaze benchmark.
For training, we follow a leave-one-out scheme where each time person is dedicated validation. one person for testing and 13 people for training. The reported results are in terms of average test results among all the 15 people. To create balanced test set, we take 200 samples from each shot similar to the methods used in the FDS work. Follow the baselines (Gong et al. 2022), only the training data for these tasks is imbalanced; the test dataset is balanced.
14

---

Published as a conference paper at ICLR 2024
gaze__ htt.sss-p...iisi2digaaas.. shot_gt th
bins-18)
'redian': (788,
'few'; [0,700],
'seny"; 11388,flosts inf
shot. dict =()
for shot, in shot_ thttthhtee ():
indices no.wherel (gaze_hist (8] (0]) (gaze_ gaaze_hittion (1)))
shot_ shot_dict(s hnntitsntt--pppooo cowwooosttteeeeeeeeeeeeeeeo _um_stackl (indices 1 Indices (1]))
Figure 9: Code snippet for MPIIGaze-DIR creation.
A.3 BASELINES
ConR is orthogonal to state-of-the-art imbalanced learning methods, thus we examine the improve- ment from ConR when added on top of existing methods, which we refer as baselines for our technique: Label- and Ceature-Distributii
Smoothing (LDS and FDS) encourage local similarities in label and feature space (Yang et al., 202I). RankSim imposes a matching between the order of similarities in label space with these similarities in feature space (Gong al., 2022). Balanced MSE encourages a balanced prediction distribution (Ren et al.] 2022). To investigate the effect of contrastive learning
also regularize using inFoNCE (Ordd et al.) 2018) and contrastive architecture of MoCo (He et al. (020), MoCo V2 (Chen et al. 2020b). Refer to Appendix A.6 for contrastive analysis. Results for RankSim on depth estimation and gaze estimation are omitted as RankSim is not suitable for these tasks.
deep imbalanced regression,
A.4 IMPLEMENTATION DETAILS
We four NVIDIA GeForce GTX 1080 Ti GPU train all models. For fair comparison, follow (Yang et al. 2021) for all standard train/val/test splits. The rest of this section provides the implementation details and choices of hyperparameters for all three datasets.
Age estimation. For AgeDB-DIR benchmark and WID--WIKIDDIR benchmark, we use Resnet50 for encoder E(-) and a one-layer fully connected network for the regression module R((-). The batch size is 64 and the learning rate 2.5 * 10- and decreases by 10x epoch 60 and epoch 80. We use the Adam optimizer with momentum of 0.9 and weight decay of le-4. Following the baselines (Yang et al.| (202T) the loss function for regression LR Mean Absolute Error(MAE). All the models are trained for 90 epochs. The augmentations in the age estimation task are random crop and random horizontal flip.
7 (0.01)w; and the similarity function Sim(-.-) is inverse Mean Absolute Error(MAE). To resolve divide by zero and infinite numbers, pair of samples with MAE distance < 1 are considered similar. Further, the pushing weight S3.q is defined as: Sj.g fs( MAE(U, y M MAM(y yg)- Finally, the similarity threshold 1, = 0.2, a = 1, and 4.
Depth estimation. For NYUD2-DIR benchmark, we use ResNet-50-based encoder-decoder archi- tecture (Hu al. 2019). The output size is 114 x 152. The batch size is 32 and the learning rate * 10 + All models are trained for 90 epochs with an Adam optimizer. The momentum of the optimizer is 0.9 and its weight decay is 1e 4. Following the baselines (Yang al., 2021 Hu et al. 2019) the loss function for regression Ln is root-mean-square((e -qusee The augmentations in the depth estimation task random rotate, color jitter, and random horizontal flip.
To measure the similarity in label space, first, for each sample (z,, Bj), we take the average value of the depth map, denoted as Tj. Then, for each pair of samples, use the (ot-mean-square(RMSII quantify the similarity of this pair in the label space. If the RMSE( Dj) less than _., sample and sample j are considered similar and otherwise dissimilar. In addition, the pushing weight Sj.q is defined as: Sj = (1) RMSE = 7i RMSE(y, yn). The similarity threshold w is 5,
0.2, 0.7 and fi = (0.2)
15

---

Published as a conference paper at ICLR 2024
Gaze estimation. We reported our results a balanced test containing 600 samples in total and 200 samples per each "many," "median." and "few" shots. Our results were averaged over five random ensure statistical significance. Each run in the evaluation incorporated leave-one-out scheme, where we performed 15 runs with a single individual as the designated test set. The final results the Mean Angle Error (in degrees) for all the individuals. The backhone is LeNet, 1, w = 1. f) = (0.01)uu w and $ = fs( MA = AEM Mq). The batch size and the base learning rate are 32 and 0.01, respectively. The augmentations in the gaze estimation task are random crop, random resize, and colour jitter.
0.4,
A.5 PERFORMANCE ANALYSIS
All the experimental results are reported as the averages random runs.
More results for age estimation. For more extensive empirical confrimation that ConR is orthogonal to DIR baselines, Table 7 shows the performance improvements when RRT Yang et al. (2021) Focal-R Yang al. (202I) I and Balanced MSE are regularized by ConR.
Table 7: MAE results of ConR on AgeDB-DIR Benchmark and HIDB-WIKI-DIR benchmark.
Metrie
MAE
Benchmark
AgeDD-DIR
IMDB-WIKI-DIR
Methods/Shots
All
Many
Median
Few
All
Many
Median
Few
RRT
7.74
6.98
8.79
11.99
7.81
7.07
14,06
25.13
RRT ConR (Ours)
7.53
6,79
7.60
10.30
7.41
6.89
13.20
23.30
Focal-R
7.64
13.00
7.97
7.12
15.14
Focal-R ConR (Ours)
7.23
8.30
11.89
7.85
7.01
14.31
25.23
Balabeed MSE (GAT)
7.57
7.46
8,40
10.93
8.12
7.58
12.27
23.05
Balahced MSE (GAI) + ConR (Ours)
7.22
6.71
7.99
9.88
7.84
7.20
12.09
22,20
Ours vs. RRT
2.71 %
2.72%
13.54%
14.10 96
5.12
2.55%
6.12%
7.28 %6
Ours vs. Focal-R
5.37 96
0.75%
9.98%
8.54 96
1,51 96
1.54%
5.48%
6,42 %
Ours vs. Balanced MSE (GAD)
4.62 96
10.05%
9,61 %
5.01%
1,47%
Error Reduction. Here we show the comparison of the Error reduction resulting from adding ConR to the deep imbalanced regression baselines (LDS, FDS and RankSim) for age estimation benchmarks(e.g. AgeDB-DIR and IMDB BEDD-WIKII-B WIKI-DIR) and (LDS, FDS and Balanced MSE) for gaze estimation benchmark. Fig. mance consistency ConR introduces to DIR for Age DIR, IMDB-WIKI-DIR and MPIIGaze-DIR benchmarks, respectively.
I0 Fig. I and Fig. [2] empirically confirm significant perfor-
LDS LDS
RankSim
FDS
Ranksii ConRt
FDS ConRt
A1
Mary
Mudian
Fow
Median
A01
Marry
Median
Fee
Figure 10: comparison on MAE results by adding ConR on top of the baselines for AgeDB-DIR benchmark.
compares the learned representations by RankSim and Balanced MSE. RanKSim by imposing order relationships encourage high relative spread while Balanced MSE suffer from low relative spread. Both RankSim and Balanced MSE have high occurrences of collapses and noticeable gaps in their feature space. Comparing Fig. with Fig. 13 shows that ConR learns the most effective representations.
Feature visualization. Fig. 13
16

---

Published as a conference paper at ICLR 2024
BankSim RankSim
LDS ConR
Median
A1
Many
Median
Few
All
Many
Median
Few
Figure 11: Comparison of the performance gain by regularizing the DIR baselines (LDS, FDS, RankSim) with ConR on
benchmark.
Balancad
(degroe)
LDS ConR.
FDS CanRt
angun enor (dearea)
Balanced MSE ConR
Moan angulll (or ereer
A4
Many
Median
Fert
Al1
Many
Median
Fur
A1
Many
Medien
Fee
Figure 12: Comparison of the performance gain by regularizing the DIR baselines (LDS, FDS, Balanced MSE) with ConR on MPIlGaze-DIR benchmark.
A.6 ABLATION STUDY
Negative Sampling, Pushing Weight and Power Analysis. Table 8 table 9 and table TO show the significance of the main contributions of ConR for IMDB-WIKE HIDB-WIKI-DIR NYUD2-DIR and MPIIGaze- DIR benchmarks, respectively.
Table 8: Ablation results of the design modules of ConR on IMDB-WIKI-DIR benchmark.
Metric
MAET
Method/Shot
All
Many
Median
Few
Contrastive-Conk
8.10
6.79
15.87
26.51
ConR-S
7.79
6.99
14.61
25.64
ConR-Sim
7.83
6.87
14.30
25.59
ConR-7
7.76
7.10
14.25
25.33
ConR (Ours)
7.84
7.09
14.16
25.15
Table 9: Ablation results of the design modules of ConR on NYUD2-DIR benchmark.
Metrie
RMSEJ
Method/Shot
All
Many
Median
Few
Contrastive-ConR
1.518
0.586
1.124
2.412
ConR-S
1.410
0.670
0.941
1.954
ConR-Sim
1.383
0.667
0.935
1.929
ConR-n
1.318
0.693
0.892
1.910
ConR (Ours)
1.304
0.682
0.889
1.885
Similarity Threshold Selection. Fig. shows the ablation study the similarity threshold for IMDB-WIKL WIKI-DIR and NYUD2-DIR benchmarks. w and w are the best similarity threshold choices for
and NYUD2-DIR benchmarks, respectively.
17

---

Published as a conference paper at ICLR 2024
30
(b)
Balanced
(a) RankSim
MSE
Figure 13: Feature visualization on AgeDB-DIn dataset for (a) RankSim, (b) Balanced MSE.
Table 10: Ablation results of the design modules of ConR on MPIIGaze-DIR benchmark.
Metric
Mean Angle Error (degrees)..
Method/Shot
All
Many
Median
Few
Contrastive-Conk
7.11
5.00
7.73
9.89
ConR-S
6.47
5,94
6.91
6.71
ConR-Sim
6.39
5.69
7.09
6.48
ConR-7j
6.24
5.87
6.96
6.27
ConR (Ours)
6.16
5.73
6.85
6.17
All Many - Median Fow
All - Many
Median
Few
MAE
RMSE
1.5
0.5
0.5
(a) BMDB-WIKI-DIN
(b) NYUD2-DIR
All
Many
Median
Few
Degree
6.5
5.5
0.5
2
4
(c) NYUD2-DIR
Figure 14: Comparison of different choices of w (a) in terms of MAE for IMDB-WIKI-DIR bench- mark, (b) in terms of RMSE for NYUD2-DIR and (c) in terms of Mean angular error (degrees) for MPIIGaze-DIR.
Hyperparameter selection. Here we present the selection process of hyperparameters a, B and 7) for all the benchmarks. Table 11 Table 12| Table 13 and Table 14 show the ablation study of ConR on and 3 in Eq. for AgeDB- benchmarks, respectively. In these tables, hyperparameter is set to the values mentioned in A.4 Additionally, Table Table 16 Table 17 and Table 18 show the ablation study of ConR on 7j in Eq 2 for AgeDB-DIR, WIDD--WII--DRR. NYUD2-DIR and MPI[Gaze-DIR benchmarks, respectively. In
IMDB WIKI-DIR, NY NYUD2-DIR and MPIIGaze-DD
these tables, hyperparameter v) set to the values mentioned A.4
Contrastive Regression. To evaluate the impact of contrastive learning on deep imbalanced regression, we use two contrastive regularizers: MoCo: We regularize the baselines with infoNCE loss, using the architecture of with MoCo V1 (He al. 2020). MoCo V2 (Chen et al. 2020b), and ConR. Here we regularized a regression model with both Moco v1 and MoCo v2. Our experiments shows that MoCo v2 degrades the regression performance in some cases. Table 19 compares the performance a regression model on AgeDB-DDr DIR, IMDB-WIKI-DE WIKI-DIR and NYUD2-DIR benchmarks when it is regularized in contrastive manner with MoCo V1, MoCo V2, and ConR. The results
18

---

Published as a conference paper at ICLR 2024
Table 11: Ablation study on a and 3 in Eq. for AgeDB-DIR benchmark
MAE]
8
All
Many
Median
Few
0.5
7.31
6.53
8.08
10.51
0.5
7.30
6.59
8.10
10.46
7.31
6.55
8.11
10.48
2
7.35
6.58
8.07
10.46
7.28
6.53
8.03
10.42
5
7.25
6.47
8.12
10.54
Table 12: Ablation study on and ( in Eq. 4 for IMDB-WIKI-DIR benchmark
MAEJ
8
All
Many
Median
Few
0.5
7.98
7.11
14.26
25.19
0.5
7.90
7.17
14.34
25.21
7.94
7.29
14.28
25.19
7.80
7.21
14.23
25.17
4
7.84
7.09
14.16
25.15
5
7.98
7.25
14.24
25.13
Table 13: Ablation study on and [ in Eq. for NYUD2-DIR benchmark
RMSE
B
All
Many
Median
Few
0.5
1
1.344
0.722
0.897
1.918
1
0.5
1.324
0.712
0.891
1.945
1
0.2
1.304
0.682
0.889
1.885
0.4
1.316
0.673
0.909
1.905
Table 14: Ablation study on and Eq. for MPIIGaze-DIRR benchmark
Mean Angle Error (degrees)
All
Many
Median
Few
0.5
6.18
5.79
6.89
6.19
0.2
6.14
5,85
6.84
6.22
0.4
6.16
5.73
6.85
6.17
6.09
5.64
7.05
6.24
Table 15: Ablation study of (pushing power) for AgeDB-DIR benchmark.
MAET
1
All
Many
Median
Few
0.009
7.31
6.63
7.99
10.5
0.01
7.28
6.53
8.03
10.42
0.05
7.36
6.51
8.11
10.51
0.1
7.38
6.49
8.09
10.48
are reported in terms of MAE for AgeDB-DIR benchmark and IMDB-WIKI-DE benchmark and in terms of RMSE for NYUD2-DIR dataset. Moco considerably boost the performance VANILLA and shows that contrastive training significantly improves the regression performance, especially for minority samples. ConR incorporate unbiased supervision into the contrastive regression and significantly boost the performance majority samples. As shown in Fig. [5] Moco and ConR provide more consistent performance compared to the baseline. In addition, Moco is consistently outperformed by ConR and empirically
minority samples with no harm to the learning process for
19

---

Published as a conference paper at ICLR 2024
Table 16: Ablation study of (pushing power) for IMDB-WIKI-DIR benchmark.
MAET
7
All
Many
Median
Few
0.009
7.88
7.07
14.25
25.16
0.01
7.84
7.09
14.16
25.15
0.05
7.88
6.99
14.20
25.16
0.1
7.91
7.14
14.18
25.21
Table 17: Ablation study of 1) (pushing power) for NYUD2-DIR benchmark.
RMSE
All
Many
Median
Few
0.1
1.312
0.674
0.886
1.891
0.2
1,304
0.682
0.889
1.885
0.4
1.295
0.619
0.92
1.911
Table 18: Ablation study of 1 (pushing power) for MPIIGaze-DIR benchmark.
Mean angular error (degrees).
1
All
Many
Median
Few
0.5
6.15
5.77
6.83
6.21
1
6.16
5.73
6.85
6.17
2
6.18
5.74
6.89
6.23
confirms ConR improves the self-supervised contrastive regularizer by incorporating supervision in an unbiased manner.
Table 19: Results of contrastive learning analysis on AgeDB-DIR, IMDB-WIKI-DIR and NYUD2-DIR benchmarks. Results are reported for the whole test data (all) and three other shots: many, median. few. At the bottom of the table the improvements of ConR with respect to Moco reported in green for each benchmark, shot and metric. In each column, the best result is in bold.
Benchmark
DIRDB-DIR MAET
MDDEWIKL-DE
NYUDZDIK RMSE
Metric
MAE.
Shot
Method
All
Many
Median
Few
All
Many
Median
Few
All
Many
Median
Few
VANILLA
7.35
6.56
8.23
12.37
8.06
7.23
15.12
26.35
1.477
0.391
0.952
2.125
+ Moco V1
7.33
6.50
8.19
11.72
7.89
7.13
14.78
26.11
1.370
0.601
0.902
1.912
Moca V2
7.47
6.21
8.75
12.75
8.12
6.99
15.02
26.01
1.404
0.632
0.978
2,207
ConR
7.84
7,09
14.16
25.15
1.304
0.682
Maco
0.68%
0.46%
190%
10.10%
000399
0.56%
4,20%
3,68%
1.63%
1.44%
35
Moco
Corff
65
Median
Many
Fel
Mony
Modian
Fow
(a) AgeDB-DIR
(b) IMDB-WIKI-DIR
(c) NYUD2-DIR
Figure 15: Comparison of the performance gain of regularizing VANILLA regression model with con- trastive regularizers: Moco V1 and ConR (a) AgeDB-DIR, (b) MMDB-WIKI-DE anf (c) NYUD2- DIR benchmarks. shows contrastive regularizer consistently lifts the performance the baseline, particularly minority samples.
20

---

Published as a conference paper at ICLR 2024
B MORE THEORETICAL INSIGHTS
Here we theoretically justify the effectiveness of ConR by deriving a upper bound on the probability of incorrect labelling of minority samples. We show that minimizing CCont robustly minimizes this probability of mislabeling for minority samples and consequently improves the generalizability.
In the following, we'll derive the upper bound: following (Oord al. 2018
Wang al., 2022
we define the density ratio (() in InfoNCE Oord et (2018) to be P(u Wang et al.
(Ord et 2018
p())
2022) that is estimated by exp(2) 2 T) in Eq. I like other contrastive objective
functions (He et al. 2020 Chen et al. 2020a). p( (r) the desired prediction distribution and
Y, = (0 - w. DJ + w).
For each anchor zy the Conkt in Eq. iis is defined to be:
exp(z) (2/T)
CC - log
Σ z,K,
(6)
NJ.
Ex,eK, exp(e) 2p/T) Dz,eK, Si exple) 2//T)
Then, CCnnty can be rewritten as:
f(),,24)
ConRRj log N Ep= f(0j, xp) + En Syaf
Following Jensen's 's inequality we have:
f(0j.
LConR = - log
(8)
== p= [(0j kp) + E Sja Sj
McShane 1937
f(ū xT:)
log
(9)
KJ Ep [(0): mp)) + Exjo ()
==0
where Ty are the negative sample, and ūj is the prediction of positive samples. prediction of zq mistakenly fall the range of (1) + w). Further, we have:
kt
f(0g. xD:)
LooRR 2
log
(10)
N
NJ Ep f(0j, xp) + Exe Staf (l
i=
N
EnT Sig (V,; Rqg
log
(11)
((0
f(0g.
N
p (1ss)pp))
Eg- S (Xg
log
(12)
1=
Di ((1s.2z) 21:)
Nt
I Sj.gf ((y, 2gg)
log
(13)
NO
D= St Rgg)
Log
(14)
i=0
N
N log
log
Si (2,, )11))
f(ūs, x:)]
(15)
N
-=0
1=0 0
21

---

Published as a conference paper at ICLR 2024
Next, the CCont is:
1
Con'
Ron
(16)
2N
==0
2N
NJ
N+
1
1
log
E
Sjal (V), - log
(ú))
(17)
2N
N
j=0
9=0
2N
N
N
1
1
log Σ Sp Sf
log Σ f(0j
(18)
2.V
2N
j=
q=0
2N
NJ
Nj
log
5 (V),
Fig) - log _ f(j m;)]
(19)
4N2
1=0
9=0
i=0
2N
N
2N
log
Saas (Y,
Σ
log
[ū0 x.))
(20)
4N22
1=0
q=0
1=0
2N
N
2N
p p (z:)
log f xqg)
Σ
log
(21)
4N2
p(ū;)
1=
q=0
1=0
1=0
((6g1es)
<1
2N
N
2N
NJ
log Sg.Sf 55gf((y (gg)
log
Σ
1)
(22)
4N2
1=0
q=0
5=0
1=0
2N
N
log Σ
f (V x.)) - 2N log N2)).
(23)
4N2
)=
q 0
Further, we have:
2N
NJ
LConR
1
log Σ S ( 2N log N1t)
(24)
4N2
1=0
9=0
2N
1
log S ff (Y,,,zz) 2N log N1+)
(25)
4.22
1=0
q=0
2N
log Sj.4
2N log NT))
(26)
4
p(L,)
q=u
2N
1
2
log S Sj.PP( (Y,1o)) 2N log N1+)
(27)
4
0= 9=0
2N N7
1
log S)mp((Y)(e)) 2N log(2.N)).
(28)
4.22
/=0 0=0
Then:
2N N
log Sy.P((y (rm)) $ Lcont + log(2N)
(29)
4
2N
j= g=0
22

---

Published as a conference paper at ICLR 2024
As explained in 3 among the 2N augmented samples only the ones with the confusion around them are chosen as anchors. Assuming is the set of selected anchors that is a subset of the 2N augmented samples have:
2N Σ
log S (Y)p(( S CContt + C, N0000
(30)
4
E= q=
30 p 5 (zg) is the likelihood of sample Ti with incorrect prediction y/ E Yg. We refer to p( (1gg) ) as the probability of collapse for zg. The left-hand side presents this probability for all the
In Eq.
negative pairs.
Regarding the empirical study by Yang et al. (2021) when learning a regression function from the imbalanced data, the representations of minority samples tend to collapse to the majority ones. Since the definition LCon in Eq. 1 I show the collapsed minority samples, minimizing the left side of the inequality in Eq. 30 is the intended optimization in deep imbalanced regression.
The left-hand side the probability of collapses during the training and regarding Eq. 30 Convergence of LConR tightens the upper bound for it. Minimizing the left-hand side be explained with two non-disjoint scenarios: either the number of anchors or the degree of collapses is reduced. Here the degree of collapse for each negative sample refers to the quantified disagreement between the label similarity and prediction similarity as discussed section 3 In addition, each collapse probability is weighted with Sj.q. leading penalizing the incorrect predictions with regard to their severity. In other words, ConR penalizes the bias probabilities with re-weighting scheme, where the weights defined based the agreement of the predictions and labels.
C ALGORITHM OF CONR
Algorithm [ shows the pseudo-code of regularizing regression model using ConR. In this algorithm, Sj,4 is the pushing weights for selected anchor 2j and its negative pair 24 (Eq. 2 Kj and Kj are positive pairs of 2j. and negative pairs of 2,, respectively.
23

---

Published as a conference paper at ICLR 2024
Algorithm ConR: Contrastive regularizer for deep imbalanced regression
Require:
input samples {24, y.) feature encoder (.)
regression function (-)) for e in epochs do for b = { No
== in Batches do Angmentations {2 N.
,1 y3 17
{z,] - [((29))=) 12Nz)
23 feature embeddings of augmented input a..
{ūj fj=0 (((E((Z))3=0))
رن prediction of 2j-
{K5 } + {zz, My. Fo 05 1
for j E (0,2.N) do
if N = then
EomRR
Samples with negative pairs not selected as anchors.
else
( Nr Ss(Ui (ua)
Sjgi pushing weight (Fq.
LComtj + computeConR (2,, K5. KJ. {))
Eq
end if end for
LConR
Average
[Coon 12% 22.
>> Eq.
CR compptceegressiooooss(///// )
Lsuu aLR BL Lo bammbbacPPropgaae())
end for end for
Ensure: Trained regression function R(.) and trained feature encoder (.)
24


Reviews:
[{"review_id": "2H5qxq2OJ5", "reviewer": "Reviewer_viAu", "summary": "The paper introduces a contrastive learning approach to address the issue of imbalanced regression. This method is orthogonal to existing solutions and has demonstrated promising experimental results.", "questions": "Given the distinct advantages of data augmentation under the contrastive learning framework, are the baseline methods utilizing their typical data augmentation strategies, or those consistent with contrastive learning? If it's the former, a comparison showcasing results after aligning the augmentation techniques would be valuable.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Data imbalance and the fairness of machine learning algorithms are practical issues that warrant significant attention.\n2. The method is reasonably designed and has shown good experimental results.", "weaknesses": "1. While the paper presents a contrastive learning paradigm adapted for regression, it does not appear to directly address the issue of data imbalance. It would enhance the paper if the authors could clarify how the method specifically tackles this challenge or consider adapting the technique to more explicitly focus on imbalanced datasets.\n2. Comparisons should be made with other contrastive regression learning methods (e.g., [a]).\n3. The manuscript could be strengthened by providing some theoretical analysis and insights to support the empirical findings. \n4. It would be beneficial if the authors could provide the pseudocode for the algorithm to facilitate the readers' comprehension of the algorithm's details.\n\n[a] [Rank-N-Contrast: Learning Continuous Representations for Regression](https://arxiv.org/pdf/2210.01189.pdf)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t3YMUz95bc", "reviewer": "Reviewer_ECg6", "summary": "This paper addresses the imbalanced regression problems where the label space is continuous. The proposed contrastive regularizer is to model global and local label similarities in feature space and prevent the features of minority samples from being collapsed into their majority neighbours. The proposed ConR consolidates essential considerations into a generic, easy-to-integrate, and efficient method that effectively addresses deep imbalanced regression. The empirical study shows that ConR significantly boosts the performance of SoTA methods on four large-scale deep imbalanced regression benchmarks.", "questions": "1. In the selection of anchor, if an example without any negative example, it will not be chosen as an anchor. What is the main reason for this selection? Do you consider the contrastive learning method without negative pairs, such as BYOL and SimSiam?\n2. The empirical label distribution is needed to determine the pushing weight for the negative pair. In addition, the authors suggest using the inverse frequency to compute the pushing weight, so that the minority samples will obtain harder force to be repelled from the anchor. How can we determine the continuous weight with the discrete frequency? Do we need any kernel density estimation technique?\n3. Please explain the specific reason for combining the proposed loss with the ordinary regression loss in Eq. (4) with more details and evidence.\n4. I notice some failure case in Table 1, Table 2, and Table 3, when adding the ConR loss as the regularization term. It is strange that you can down-weight of $\\beta$ to avoid this situation. I understand that the main goal is to improve the performance on *Few* case, however, there are still some cases that the performance on *Few* case drops. Also, there are some cases that the performance on *All* case drops. Hence, please provide the explanation for this phenomenon and what is the specific metric for a good imbalanced regression in your paper. I think the *Few* case should be much more important.\n5. Why the results on the *Few* case of Balanced MSE in Figure 3 is inconsistent to that in Table 3?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to understand.\n2. The novelty of this paper is good. To my knowledge, applying contrastive learning to imbalance regression is novel.\n3. The authors provide a new dataset with 2-dimentional label space by using MPIIGaze, which could be useful to the imbalance regression community.", "weaknesses": "1. The proposed objective in Eq. (4) is relative hasty, without the reason of introducing the ordinary regression loss. Also, the definition of the introduced loss should be clearly provided.\n2. No deviation measure in the experimental results.\n3. No experimental result for the ConR-only case. All the results for the proposed ConR are with respect to the combination with existing methods as a regularized. Due to supervised information is already used in the loss of ConR, its preformation should be provided as a baseline.\n4. Some references are missing, e.g., Page 16.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EfRyM7PuIt", "reviewer": "Reviewer_M28F", "summary": "The paper addresses the problem of imbalanced problem in real-world data. To tackle the imbalance in continuous label spaces, the authors introduce a contrastive regularization technique named ConR, which is based on infoNCE. This technique simulates both global and local similarities between labels in the feature space, preventing the features of minority samples from being overshadowed by those of majority samples. ConR primarily focuses on discrepancies between the label space and the feature space and penalizes these differences. Indeed, this is also an augmentation method rather than re-weighted method in DIR.\n\nHowever, this work contains several weakness, which is discussed below.", "questions": "See Above. Overall, this work is novel, and interesting. The problem they try to solve is imporant, and results seems good. Nonetheless, I strongly recommend the authors to delve deeper by: \n\n(1) **theoretical insights or guarantees** supporting the efficacy of the proposed method, and \n\n(2) **thorough empirical analysis of the experimental outcomes**.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. ConR presents a novel auggmentation approach to handle the imbalanced problem in continuous label spaces, whose problem is important.\n\n2. The methods of regularizing process of **ConR** by pulling together positive pairs and relatively repelling negative pairs seems solid.\n\n3. The results seem to indicate that the proposed method can be seamlessly integrated with other models and exhibits improvements over existing baselines.", "weaknesses": "1. While the author claims that **ConR** reduces prediction error, are there any theoretical insights or guarantees supporting the idea that **ConR** can achieve a lower generalization bound? Relying solely on empirical results might not suffice to attest to the superiority of the proposed methods.\n\n2. I am curious about complexity. When performing augmentation on large-scale datasets, sampling might increase the complexity. This leads to a prevalent question: Why not leverage reweighting methods which can attain comparable (or potentially superior) results without the significant memory and time overhead associated with the augmentation step? [1,2] \n\n3. While AgeDB-DIR, IMDB-WIKI-DIR, and NYUD2-DIR are structured in DIR [3], and MPIIGaze-DIR is a creation of the authors, a comprehensive description of each dataset should be included in the Appendix. Moreover, given the variety of metrics in NYUD2-DIR, the rationale behind selecting only two needs clarification.\n\n4. The experiments did not surpass all baselines. For instance, in Table 2, the combination of **LDS + FDS + RankSim** posts the best results in terms of GM in few-shot case.\n\n5. For larger datasets like IMDB and NYUD2, **ConR** doesn't consistently outperform other models. However, it seems to excel with smaller datasets such as AgeDB and MPIIGaze.\n\n6. Examining Table 1, the reported results are as follows:\n\n| model | few |\n| :---------: | :------: |\n|FDS + RankSim|  9.68 |\n|FDS + RankSim + **ConR**| 9.43 |\n|LDS + FDS + RankSim| 9.92 |\n|LDS + FDS + RankSim + **ConR**| 9.21 |\n|||\n\nFurther analysis is warranted. For instance, why does **LDS + FDS+RankSim** underperform compared to **FDS + RankSim**? Yet, with the addition of **ConR**, it achieves superior results. This observation hints that **ConR** might enhance outcomes when paired with **LDS + FDS + RankSim**. However, this theory doesn't hold when assessed on IMDB-WIKI-DIR. A more detailed analysis of the experimental results is recommended.\n\n[1] Wang et al., Variational Imbalanced Regression: Fair Uncertainty Quantification via Probabilistic Smoothing. NeurIPS 2023\n\n[2] Ren et al., Balanced MSE for Imbalanced Visual Regression, CVPR 2022\n\n[3] Yang et al., Delving into Deep Imbalanced Regression. ICML 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s189PXizzW", "reviewer": "Reviewer_Yj3F", "summary": "For imbalanced regression, the authors propose ConR, which uses\nsupervised contrastive loss as part of the loss function.  They define\nSim(label1,label2) > omega, a threshold hyperparameter, as similar\nlabels.  For each of the input instances, it generates two augmented\ninstances.  If the actual labels of two augmented instances are\nsimilar, they form a positive pair.  If the predicted labels of two\ninstances are similar, but the actual labels are not, they form a\nnegative pair.  For an augmented instance, a set of positive samples\nand a set of negative samples are found.  Augmented instances with at\nleast one negative sample are called anchors, which participate in\ncontrastive loss.  For each anchor, the fraction in the regular\ncontrastive loss is summed over all positive samples.  For the\nnegative samples in the denominator, they have a \"pushing weight\" S,\nwhich is a function of the density-based weight of the anchor and the\nSim(anchor_label, negative_sample_label).  L_conR is an average of the\ncontrastive loss of up to 2N anchors. The overall loss is a weighted\nsum of the regular regression loss and L_conR.\n\nConR was evaluated on 4 datasets, one of which has\nmulti-dimensional labels, and compared with 4 recent techniques.\nEmpirical results indicate adding ConR generally improves performance.", "questions": "1. Eq 1, the outermost summation: Summing over i's implies j's with more\n   positive samples would have more contribution.  Is that desirable?  If so,\n   what is the main reason?\n\n2. L_R is on the original instances, while L_conR is on 2\n   augmentations of each of the original instances.  That is, the\n   original instances are not directly involved in L_conR--is that\n   correct?  If so, what is the main reason?\n\n3. Similarity threshold omega between labels dependent on\n   applications and range of the labels (1-100 vs 1-10^6), any\n   further insights?  It seems to be trial and error as a\n   hyperparameter.\n\n4. How does ConR alone, not inconjunction with another technique,\n   perform?  Including it in Tables 1-3, would be beneficial.\n\n5. How do the representations from Balanced MSE and RankSim, which are\n   more recent, compare with ConR.  Including them in Fig 4 would be\n   important.\n\n6. In Fig 5, why 1/omega, instead of omega, is used?  In the approach\n   section, 1/omega was not discussed. \"Fig. 5b, choosing a higher\n   similarity threshold\"--it seems similarity threshold omega is\n   smaller at 1 (1/omega is higher).\n\n7. p9: Could you further explain: \"sharing feature statistics to an\n   extent that is not in correspondence with the heterogeneous\n   dynamics in the feature space\"?\n\n8. Sec 3.2.1: how are the instances augmented?  It seems to be not\n   discussed in the approach or experiments.\n\nComments:\n\nSec 3.2.2: for completeness, f_S could have more description (it seems\nto be a simple product in the appendix)\n\nminor:\n\nEq 3: 1/2N assumes all 2N augmented instances are anchors, but some\nmight not be anchors (if I understand correctly).\n\nKang et al. 21 and Khosla et al. 20 have duplicated entries in\nReferences.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Imbalanced regression is an interesting problem.\nThe proposed technique adapts supervised contrastive loss from\nclassification to regression.  Particularly, they added a pushing\nfactor for the negative samples based on similarity in labels and density, \nwhich is interesting.  Also, negative samples have not just\ndifferent labels but also similar predictions.  Empirical results\nindicate adding ConR generally improves performance.", "weaknesses": "The proposed contrastive loss, Eq 1, could be further explained and\njustified.  Empirical evaluation could be improved by adding \"ConR\nonly\" and representations of RankSim and Balanced MSE, which are more\nrecent techniques.  The similarity threshold omega seems to be an\nimportant parameter, further insights could be explored.\n\nMore details are in questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)