Gradient Aligned Regression via Pairwise Losses
Authors: Dixian Zhu
Tianbao Yang
Livnat Jerby
Venue: ICML 2025 poster (2025)
Keywords: Regression
Pairwise Losses
Primary area: general_machine_learning->supervised_learning
TL;DR: Novel pairwise losses for regression.

Abstract: Regression is a fundamental task in machine learning that has garnered extensive attention over the past decades. The conventional approach for regression involves employing loss functions that primarily concentrate on aligning model prediction with the ground truth for each individual data sample. Recent research endeavors have introduced novel perspectives by incorporating label similarity into regression through the imposition of additional pairwise regularization or contrastive learning on the latent feature space, demonstrating their effectiveness. However, there are two drawbacks to these approaches: (i) their pairwise operations in the latent feature space are computationally more expensive than conventional regression losses; (ii) they lack theoretical insights behind these methods. In this work, we propose GAR (Gradient Aligned Regression) as a competitive alternative method in label space, which is constituted by a conventional regression loss and two pairwise label difference losses for gradient alignment including magnitude and direction. GAR enjoys: i) the same level efficiency as conventional regression loss because the quadratic complexity for the proposed pairwise losses can be reduced to linear complexity; ii) theoretical insights from learning the pairwise label difference to learning the gradient of the ground truth function. We limit our current scope as regression on the clean data setting without noises, outliers or distributional shifts, etc. We demonstrate the effectiveness of the proposed method practically on two synthetic datasets and on eight extensive real-world tasks from six benchmark datasets with other eight competitive baselines. Running time experiments demonstrate the superior efficiency of the proposed GAR compared to existing methods with pairwise regularization or contrastive learning in the latent feature space. Additionally, ablation studies confirm the effectiveness of each component of GAR. The code is open sourced at https://github.com/DixianZhu/GAR.

Gradient Aligned Regression via Pairwise Losses
Dixian Zhu Tianbao Yang²
Livnat Jerby'
Abstract
of the proposed GAR compared existing meth- ods with pairwise regularization or contrastive learning in the latent feature space. Additionally, ablation studies confirm the effectiveness of each component of GAR. The code is open sourced at com/ Dixi Sanzuhu/GAR
Regression is a fundamental task in machine Iearn- ing that has garnered extensive attention over the past decades. The conventional approach for re- gression involves employing loss functions that primarily concentrate aligning model predic- tion with the ground truth for each individual data sample. Recent research endeavors have introduced novel perspectives by incorporating label similarity into regression through the im- position of additional pairwise regularization or contrastive learning on the latent feature space, demonstrating their effectiveness. However, there are two drawbacks to these approaches: (i) their pairwise operations in the latent feature space are computationally more expensive than conven- tional regression losses; (ii) they lack theoretical insights behind these methods. In this work, we propose GAR (Gradient Aligned Regression) as competitive alternative method in label space, which is constituted by conventional regression loss and two pairwise label difference losses for gradient alignment including magnitude and direc- tion. GAR enjoys: i) the same level efficiency as conventional regression loss because the quadratic complexity for the proposed pairwise losses can be reduced to linear complexity; ii) theoretical insights from learning the pairwise label differ- ence to learning the gradient of the ground truth function. We limit our current scope as regression on the clean data setting without noises, outliers or distributional shifts, etc. We demonstrate the effectiveness of the proposed method practically synthetic datasets and eight extensive real-world tasks from six benchmark datasets with other eight competitive baselines. Running time experiments demonstrate the superior efficiency
1. Introduction
As one of the most fundamental tasks in Machine Learn- ing (ML) field, regression stands out as a powerful tool for understanding and modeling the relationships from com- plex data features to continuous data labels. Regression techniques have been widely utilized in many areas, such as computer vision (Moschoglou et al.) 2017: Niu et al. 2016), drug discovery (Dara et al., 2022). economics and finances (Benkraiem & Zopounidis 2021 environmental forecast Zhu al., 2018), material science (Stanev et al., 2018). For centuries, Mean Squared Error (MSE) and Mean Absolute Error (MAE) have been adopted to measure the distance from the ML model predictions to the ground truths. Consequently, researchers have naturally proposed to opti- mize the least squared error and least absolute error (Legen. dre 1806 Dodge 2008). Since the last century, to improve model stability and control model complexity, researchers have proposed various model regularizations based on MSE loss, resulting in multiple variants such as Ridge Regression, LASSO, and Elastic Net (Tikhonov et al. 1943 Tibshirani 1996; Zou & Hastie 2005). Along this line, the Huber loss was proposed trade off between MAE and MSE, aiming to mitigate the issue of MSE being sensitive to outliers (Hu- ber 1992). All these conventional loss functions and many of their variants focus minimizing the difference between model prediction and ground truth for each individual data sample pointwisely, but do not explicitly model and capture the relationships for multiple data samples.
Recent research has proposed incorporating label pairwise similarities into regression across different data samples, addressing part of the gap left by conventional loss func- tions that focus solely on individual prediction errors (Gong) et al. 2022] Keramati et al. 2023 Zha et al. 2023). Those latent- www-ssppaceeeeesee pairwise regression methods define an "anchor data point', which either acts as a reference point for calculating the relative ranks to the anchor for the other
'Department of Genetics, Stanford University, CA, USA DDepartment of Computer Science, Texas A&M University, TX, USA. Conrespondence to: Dixian Zhu <dixian zhu sttanford.edux. Livnat Jerby <ljerby @ sddu,
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Gradient Aligned Regression via Pairwise Losses
2022 or the ranks are consequently uti- lized to construct positive and negative pairs (Zha et al. 2023 Keramati et al. 2023). One of the concerns for those approaches is conceptually to see: the original pairwise label similarities are converted to ranks or positive and negative pairs. There is an approximation loss because the conver- sion is single-directional (and the original label similarities cannot be recovered from the coarser converted informa- tion). Moreover, they also require higher computational costs than conventional loss functions, as they need to com- pute pairwise similarities in the latent space, which requires quadratic time related to the training batch size.
data Gong et al.
2016 Rothe et 2013 Zhang al. 2023). When learning time-series data, researchers can apply advanced ML models, such as Recurrent Neural Network (RNN), Long short-term memory (LSTM) network or Attention model, for capturing the time-series pattern (Rumelhart et al.) 1986 Hochreiter & Schmidhuber 1997; Vaswani et al. 2017): or, apply consecutive regularization that enhances the sim- ilarities for the predictions from adjacent time slots (Zhu et al. 2018). Studies propose to minimize penalties for regression on multi-task learning when there are multiple tasks or targets from the dataset (Solnon et al. 2012). Work adapts regression to active learning when there is unlabeled data available for querying and labeling (Holzmüller et al.) 2023). Papers improve robustness for regression for online learning when the data is sequentially available (Pesme & Flammarion 2020). Work improves the robustness for re- gression when the data is imbalanced (Yang et al.) 2021 Ren et al. 2022). Research strengthens the robustness for regression when there are adversarial corruptions (Klivans et al. 2018).
Our contributions can be summarized in three main aspects: 1) We propose pairwise losses in label space to capture pair- wise label difference for regression task that can enhance rank preserving; moreover, we provide theoretical insights between learning the gradient for the ground truth function and optimizing the proposed pairwise losses. 2) We further propose equivalent and more efficient formulations for the proposed pairwise losses, which only require linear time rel- ative to the training batch size and are as fast as conventional losses, such as MAE and MSE. 3) We demonstrate the effec- tiveness of the proposed method, GAR (Gradient Aligned Regression), on two synthetic datasets and eight real-worke regression tasks. Extensive experiments and ablation stud- ies are conducted for supporting that GAR is as efficient as conventional regression loss and the effectiveness of each component for GAR.
Regression with Pairwise Similarity: recently, there are several methods are proposed to improve regression with pairwise label similarities (Gong et al.. 2022 Zha al.) 2023 Keramati et al.) 2023). The Rank method is pro- posed to convert the label pairwise similarities for other data as ranks for each anchor data; then enforce the predictions follow the same rank by adding a MSE loss on the prediction rank versus the label rank (Gong et al. 2022). The RNC (Rank-N-Contran method inspired by contrastive learn- ing, which follows the similar fashion SimCLR (Chen) ct al. 2020). but defines the positive and negative pairs by choosing different anchor data points with pairwise label similarities; RNC is proposed for pre- pre-training that requires fine tuning afterwards (Zha et al. 2023). ConR (Contrastive Regularizer) is also proposed in contrastive learning fash- ion, whose positive pairs are decided by the data pairs with high label similarities and negative pairs are decided by the data pairs with high label similarities but low prediction similarities (Keramati et al. 2023). All of them impose the label pairwise similarity in latent feature space. In contrast, we propose GAR, which: 1) Directly captures the pairwise label difference in label space, theoretically connecting it to learning the gradients of the ground truth function. 2) Reduces the quadratic pairwise computation to linear com- plexity because the proposed pairwise losses are equivalent to the variance of prediction errors and the negative Pear- son correlation coefficient between predictions and ground truths.
2. Related Work
Regression on Generic Setting: MSE, MAE and Huber loss are the most exposed conventional regression loss func- tions that work on generic regression settings (Legendre) 1806] Dodge 2008 Huber 1992), all of which focus on fitting the prediction to the truth for each individual data sample. Specifically, MAE and Huber loss are more robust to outliers. There are other less exposed losses for generic regression, such as Rooted Mean Squared Error (RMSE) loss and Root Mean Squared Logarithmic Error (RMSLE) loss (Ciampiconi et al. 2023), but they are also restricted to minimizing individual error. Different regularization such as 4n-nrmm 2n-norm penalties on model parameters have been studied as promising way to prevent model over-fitting and improving training stability (Tikhonov et al., 1943 Tib- shirani 1996 Zou & Hastie| 2005). Extensive research has been conducted to improve the regularization for regression by making the penalty weight adaptive or optimal under certain conditions (Zou) 2006 Wu & Xu 2020).
Pairwise Loss in Other Applications: pairwise loss functions have also been utilized in other fields, such as learning-to-rank, metric learning, and AUROC optimiza- tion. RankNet, for example, employs a logistic loss on sample pairs to learn the relative order of items in ranking
Regression on Extensive Settings: when the data label is discrete, ordinal regression techniques can be applied to convert the regression task to either multiple binary classifi- cation tasks one multi-class classification task (Niu et al.]
2

---

Gradient Aligned Regression via Pairwise Losses
function, which incurs quadratic time complexity with respect data size (Burges et al.
additional bias correction, f2(-) could easily achieve 0 training error.
2005). In metric learning,
pairwise contrastive loss (Hadsell et al. 2006) encourages embeddings of predefined similar samples to be close, while pushing dissimilar samples apart. For AUROC optimiza- tion, several pairwise loss functions have been proposed, typically computed on pairs of positive and negative sam- ples using binary label information (Gao & Zhou 2015). In contrast, our proposed pairwise losses are designed for regression with continuous labels, achieve linear time com- plexity, and do not depend on predefined similarity pairs or binary class labels for pair construction.
2) Prior studies have been proposed to enforce the model to maintain the relative rank regarding to label similari- ties (Gong et al.) 2022 Zha et al. 2023] Keramati et al.) 2023). Take a simple example for age prediction, there are people aged as 8/1 10. 32 20, ys = 70; the prior methods either impose regularization or adopt contrastive learning in the latent feature space with pairwise fashion to push prediction ((x2) to be closer to f'(x1) than f'(x)).
Motivated by the second point, we explicitly enforce the pairwise prediction difference fx,) - f(z)) to be close to the pairwise label difference 1/s - yj through two pairwise losses in the label space: the first directly matches the origi- nal label difference, while the second is a relaxed version with scaling factor. Next, we prove that: the first pair- wise loss is equivalent to the variance of prediction errors and the second is equivalent to negative Pearson correlation coefficient between predictions and ground truths, which addresses the concern in the first point. Moreover, the equiv- alent forms reduce the quadratic time complexity for the proposed pairwise losses to linear complexity. Last but not least, we provide theoretical insight between learning the pairwise label difference and learning the gradients of the ground truth function.
In this work, we focus on regression on the clean data setting without considering noises, outliers, distributional shifts, etc.
3. Method
Denote the underlying ground truth function for target y E R! on data x E Ed as y = (xx)): the training dataset as D = (1),,,1)..... (X., 3n)]. where y/1 = (xx.). which sampled from underlying data distribution D. The re- gression task is to learn fitting function ((x) that can approximate (xx). It is straightforward to extend the single- dimensional target case to multi dimensional target case. For the sake of simplicity, the presentation in this paper focuses on the single target form.
3.2. Pairwise Losses
The conventional regression task optimizes:
We propose the following pairwise loss of label difference for a regression task:
1 N Σ (((())..))
Le
Laift
Σ Σ =1
(3)
where e(.--) denote the individual loss function for each data sample. Denote the prediction error (difference between prediction and ground truth) for x under the model f(-) by: of ((x(x), Xxx)) = fxx) - ->(x). The conventional MAE and MSE loss can be written as:
N2
where the individual loss E(:,)) )) is defined similarly as con- ventional regression. Lait explicitly and directly enforces the pairwise prediction difference to close to the pairwise label difference.
N
MMAR__
1
Σ
C
MSE
Σ
(2)
N
Theorem 1 When choose individual loss E as 1/2 squared error function, l (aMSE(a, b) ] - b)2:
3.1. Motivations
Our work is motivated by the following two aspects:
N
N
(xx.) - ,(xx)) - Vxx,))
MSE diff
1) The conventional regression loss function only focuses on minimizing the magnitude of 6g. which can be blind on other evaluation perspectives. For example, there are two models f-) and fz() 4 training data samples with er-
N2
Var (
(4)
rors as: (6/) .... { (1, 1, -)), (6k)...,, :k: = 1, Conventional regression loss doesn't tt impose any preference over (-) or f2(.). However, fa()) enjoys smaller error variance (indicating more consistent predic- tion and better for rank preservation). Moreover, with
where Var(-) denote the empirical variance. The proof is included in Appendix A.
The proof is based on simple algebras that add and subtract the mean value of of denoted as of. The similar proof logic
3

---

Gradient Aligned Regression via Pairwise Losses
can be used to decouple MSE loss, CAME
1 N ( )::
Next, we provide the theoretical insight between learn- ing pairwise label difference and learning the gradients of ground truth function.
N
1
5/ + +
N
1=1
Theorem 4 For two K-order differentiable deterministic
N
1 166 82)2 + (81) Var(o + (8)) Σ
functions with open domain f()): R. + RK, (1)) IR +
N
R', the following holds: f(x1) - fx22) = }(xx)
y(x2), V V*)(x), V(x, V(x)) E D, = 21,
(x2,)(x)) E D, iff V =
MSE lacks the flexibility to trade off the mean and variance of errors. Simply summing them results in the mean of squared individual errors, which loses the scope of group patterns.
We include proof in the Appendix A.3
The proof mainly depends on Mean Value Theorem, Limits and L'Hôpital's rule. Besides of fitting the prediction f(x;) to be close to the ground truth yo. that is, f(x) ~ {xx), (xx,)((x) E D, we additionally make the model to explicitly capture the gradient (and the higher-order gradients) of the ground truth function, i.e.
Remark 1:
Corollary 2 When E is 2 squared error, the loss of pairwise label difference has the following simpler and more efficient empirical form:
N Σ -
1
MSE Lig
(1: -
(5)
V fx) = V*)((x, k = f1.2,.... with the proposed pairwise losses.
The corollary directly comes from Theorem 7 by the defini- tion of variance.
Remark 2: Although we assume that the function is de- terministic for simplicity, this theorem is also applicable to common stochastic functions in a certain form. For example, yxxx 2) = m(x) + z sxx,, where ~ N (0, 1) is stochas- tically sampled from the standard normal distribution (also works for any zero-mean distribution). Let m m(x) and s(x) represent the ground truth mean and standard deviation for the general heteroskedastic case. We can still prove that the model can capture the expected gradient in x, that is, Eg [V>)(xx 2)] if and only if the difference in the predic- tion pairs matches the difference in the truth pairs under expectation. We have the following extended corollary.
It is obvious that dill enjoys linear time complexity. Next, we propose a relaxed version for Lditt. We apply a relaxing scaling factor when match the pairwise differ- ences from predictions and ground truths, in order to allow mismatch on magnitude:
f(x:) - [(x)) = ()(x)) - -((x)]]
Denote the pairwise difference as dfej s(x:) f(x)) and d)i, = ((x,) - )(xx)) for the predic- tion and the ground truth, the vector notations for them as df = [dft,1, df1.2..... dff..N-..ffv.Nl dy' = 2..,...... dVN.N (dD),,,,d'i.,.........
N.V..v] We proposed the fol- lowing p-norm loss function for the normalized pairwise difference to capture ground truth function shape, where the scaling factor is removed by normalization:
Corollary 5 For K-order differentiable deterministic func- tion with open domain f(+) : IR + RR, and K-order dif- ferentiable stochastic function with open domain V(---) Rd+1 - RR. that can be written as )(x, 2) m(x) +
= =x((x), where z is randomly sampled from distri- bution with zero-mean and m(-) R + R, s(.)
df
dy
Caiffnorm =I
(6)
|d>'
Rd + R. The following holds: f(xi) - f(xx) =
E., 21) - Ez, (Xx2, 22)]. Vεl, 22 NO., 1);
Corollary 3 When choose p = 2 for Laighorm it equals to: 1- Pearson correlation coefficient:
V(XI, (xi,, 21),, (x2, x2)) E D. iff A V*f(x) =
Eg Vxx 2)], v(x.)(xx 2)) D; Ve N(O,1).
1
N ([(x))-).
(yi 5) 2
difiiorm
2N
[Var(S)
/ Var(y)
Proof sketch; E, (XX, =)] = m(x), E. V (x =
1=1 1
v*m(x). Our previous proof for Theorem 4 still goes through by replacing V(x) with m(x). Notice that the stochastic term s(x) can be canceled by taking the ex- pectation over due to zero-mean.
Cov(S.v)
1
(arl()Var(v)
11 p(f.v)
where Cov(-,-) denotes covariance and p(---) denotes Pear- son correlation coefficient. We include the proof in the Appendix A.2 The formulation also enjoys linear time com- plexity.
Remark 3: It should be noted that the proposed pairwise losses are designed to work with conventional pointwise re- gression loss (e.g. MAE), which complementarily captures pointwise function values.
4

---

Gradient Aligned Regression via Pairwise Losses
3.3. Efficient and Robust Aggregation for Different Losses
It is worth noting that:
M
Now that we have three losses that are important for the proposed GAR (Gradient Aligned Regression): Le, Lains and Leilfnorm: more specifically, we will focus on
exp (C (C1, LI.,.,, LMr:a)
=
M11 Σ
L vaya
(M 1=1 Li) tt
- t00.
C CMSE Lp=2 diff diffnorm in the rest of the presentation. Next, we propose a robust approach to aggregate them.
1 Σ Ci M
a=
(10)
arg max M
Li
a a+0.
Given the huge diversity of the dataset and model, it is time-consuming tune the three terms with a convex com- bination. Furthermore, the magnitudes for each term can be different, which adds more difficulty to balance them. For example, diffnorm, E [0,2] 2] but MA can be infinitely large; therefore, dilfnorm could be easily overwhelmed if simply combining other components with larger magnitudes by arithmetic mean, i.e. L = (C..AR + M + C 3/3.
which trade off from geometric mean, arithmetic mean and the maximal value for Li,...g LMs-
Remark: the proof is included in the Appendix A.4
Next, discuss how to take care of extreme case for LGAR- Notice that c!/a can have numerical issue when a - O, + +oo or a + +o0, + 0. The former case can cause computation overflow on forward propaga- tion and the later case can cause computation overflow on backward propagation. Denote Lms. = arg max C, and Lmin arg min L., We utilize the constant value (de- tached from back propagation) of Lmax Lmin to control CKL GAR
An intuitive way amplify the significance of difthorm is the geometric mean, that is, [ (CMME diff co 11/3
where the effects of different loss magnitudes are unified. However, it could raise another issue. Consider C(a, c) (abe))/3, @C(a.b.) (be)*/3
a-75-- The smaller component can get a larger gradient but may experience a numerical issue when a < be. As a consequence, the overall objective could focus solely on the smallest component.
It is easy to show LK), has the following equivalent empirical formulation:
a log (Mr Ei=
)1/a) + log fmuxx a < 1.
CKL =
Inspired by the previous two intuitive examples (the loss with smaller magnitude either gains too less attention or too much attention). we propose an efficient and robust aggrega- tion approach for GAR based on variant of Distributionally Robust Optimization (DRO), which can not only trade off the previous two cases, but also only requires one tuning hyperparameter (Zhu et al., (023).. For each loss, we ap- ply a logarithmic transformation reduce the magnitude effect. For the sake of generality, we denote the loss L; and the overall loss as LGAR(C1, (Mr) where the overall loss is composed with M sub-losses. Denote ((p) Ji ) as divergence measure from probability vector p over simplex AM to the uniform distribution 1. for GAR takes the balance from the averaged value to the maximal value:
a aooo Li=1
)1/a) + log Cmin a 2 1.
Lum
(11)
by which we can avoid the numerical issue for both for- ward and backward propagation. We present algorithm pseudocode in Alg
Algorithm 1 Gradient Aligned Regression (GAR)
Require: hyper-parameter a for balancing sub-losses, training dataset 1 {(x;, u:)
Initialize model f(-). for t= 1 to T do
37- The DRO formulation
Sample mini-batch of data ((x;, 31 )i),en,)) Compute MAE loss CMAE
Compute the losses for derivative: CMSE and CP=2 by E.q. 5 and E.q. 7
LoAR(Li,..,, = max PEAM
Σ
M p. log C, - (p
1
Compute GAR (KI.) loss GAR by E.q. 11
Utilize SGD or Adam optimizers to optimize model with gradient CKI.
M
(8)
end for
where a 2 is the robust hyper- paremeerr for GAR.
the computational complexity for A is O(TB), where T is iteration number and B is batch size.
Remark:
Theorem 6 When take D(-1-) KL-divergence, GAR has the following equivalent formulation:
4. Experiments
CKI. (., a logg
c
1/ay.
(9)
We demonstrate the effectiveness of GAR with two synthetic datasets, eight real-world tasks on five tabular benchmark
M
5

---

Gradient Aligned Regression via Pairwise Losses
datasets and one image benchmark dataset. Further analysis for running time of GAR, ablation studies on different com- ponents, sensitivity to nyper-parameter a and batch sizes are presented afterwards.
we uniformly randomly sample half of the data as the train- ing dataset. The total training epochs are set 300 and batch size is set as 128. The initial learning rate is tuned in [le-1, 1e-2, 1e-3, le-4) for Adam optimizer (Kingma & Ba 2014), which is stage-wised decreasing by 10 folds at the end of the 100-th and 200-th epoch. The weight decay is tuned in (le-3,1e-4,Ie- I.-..). RNC takes the first 100 epochs are pre-training and the remaining as fine-tuning with MAE loss; temperature for RNC is tuned in (1,2,4). For each baseline, we run 5 trials independently with the following 5 seeds: (1,2,3,4,5). We set the = 0.5 for GAR on all synthetic experiments.
4.1. Synthetic Experiments
We first demonstrate the effectiveness of GAR on two syn- thetic datasets as toy examples.
Datasets: 1) Sine: we take 2y from [-10T, 10x] with the interval 0.1 and yi = sin(x.). The data ground truth is presented as the grey solid line in Fig ! 2) Squared sine: take ã, from 1024 1024 with the interval 0.1, then calculate x; sign( (x4 v V which increases the density of data from the region far from the origin. Fi- nally, denote u² as the empirical mean value of a7. make y: = x2 sin(2;) which magnifies the target values that are far from the origin. The data ground truth is presented as the solid grey line in Fig 2
Results: the results on the sine dataset are summarized in Fig. D and the results on the squared sine dataset are summa- rized in Fig. [ The figures for mean and standard deviation of the predictions are shown on the right. Due to the lim- ited space, we include MSE results in Appendix [B.3] as it is similar with MAE. On both of the datasets, we can see there is a clear advantage for the proposed GAR over the conventional MSE/MAE loss. For the sine dataset, GAR captures approximately or more peaks than MSE/MAE loss. For the squared sine dataset, GAR can almost recover all the patterns (both shape and magnitude) of the ground truth; however, MSE/MAE loss is hard to capture the pattern of the ground truth except part of peaks with the largest magnitude. Compared with LMAR p and RNC, GAR also demonstrates clear ad vanttage on the both synthetic datasets for capturing more peaks on the sine dataset and capturing magnitudes better on the squared sine dataset. We also con- duct experiments the sine dataset with different number of layers for FFNN model in Appendix B.5 Fig 6 which again demonstrates similar results.
Baselines: we compare the proposed GAR with 4 ap- proaches: MAE, MSE, RNC and an extra heuristic fused loss (named (AE-Pearson) that combines MAE with nega- tive Pearson correlation coefficient linearly:
LMAE-P =3Co
MAE + (1 diffnomm
where recall that Cammoom = 1 - pLJ, y). and 33 is dynam- ically assigned as the constant value of p(f. y) on each iteration. Intuitively, when the Pearson correlation coeffi- cient high, the model focuses more on the MAE loss; and vice versa. It is worth noting that GAR can also be adapted to Symbolic Regression (SR) settings. For example, GAR loss could be defined as a fitness function for evolutionary algorithm based method or reward function for reinforce- ment learning based method (Cranmer 2023 Petersen et al. 2019). However, given that this work focuses on Numeric Regression (NR) setting, it is unfair to directly compare the proposed GAR under the NR setting with SR methods here. Because SR methods have placed the ground truth functions in their 'toolbox' (as search candidates) for these toy examples. To elaborate on this point, we extensively em- ploy a well-established SR method, PySıT on the synthetic datasets. We find PySR performs perfectly when trigono- metric functions are included in the search space. However, performs much worse when we exclude trigonometric functions. Results are presented in the Appendix B.4
4.2. Benchmark Experiments
We conduct experiments for comparing GAR with other eight competitive baselines on eight real-world tasks.
Datasets and Tasks: 1) Concrete Compressive Strength Ych 1998): predicting the compressive strength of high- performance concrete. 2) Wine Quality (Cortez et al. 2009): predicting wine quality based on physicochemical test val- ues (such as acidity, sugar, chlorides, etc). 3) Parkinson (Total) (Tsanas al. 2009): predicting clinician's 's total unified Parkinson's 's disease rating scale (UPDRS) score by biomedical voice measurements from 42 people with early- stage Parkinson's 's disease recruited to a six-month trial of a telemonitoring device for remote symptom progression monitoring. 4) Parkinson (Motor) Tsanas et al. 2009): predicting clinician's 's motor UPDRS score with the same previous data feature. 5) Super Conductivity (Hamidieh) 2018): predicting the critical temperature for super conduc- tors with 81 extracted material features. 6) IC50 (Garnet) et al. 2012]: predicting drugs' half-maximal inhibitory con-
Model: We utilize a 7-layer simple Feed Forward Neu- ral Network (FFNN) with 5 hidden layers where neuron number for each layer is set as 100 with ELU activation function (Clevert et al. 2015).
Experimental Settings: For both of the synthetic datasets,
https://astroauttont http://ssttrrtttttaaaaaaoo//g///
6

---

Gradient Aligned Regression via Pairwisc Losses
1.0
MAE
1.0
RNC
1.00
0.5
0.75
0.0
0.50
MAE-Pearson GAR
-0.5
-0.5
0.25
1.0
0.00
20
-0.25
0.50
GAR
-0.75
0.5
0.5-
-1.00
0.0
-30
20
-0.5
Figure I. sine: truth v.s. prediction.
-1.0
MAE-Pearson
-1.0
-20
20
-20
20
truth
1.0
MAE
RNC
1.5
MAE
1.0
Am
RNC
MAE-Pearson
0.0
0.5
GAR
-0.5
0.0
-1.0
-2
-20
-20
20
0055
-1.0
MAE-Pearson
-20
20
Figure 2. squared sine: truth v.s. prediction.
20
centration (IC50) for 15 drugs filtered from 130 total drugs with missing-value-comtee over 639 human tumour cell lines. We include more details about data preprocessing in the appendix. AgeDB (Scratch) (Moschoglou al. 2017): predicting age from face images. All baselines are trained from scratch. 8) AgeDB (RNC Linear Probe); pre- dicting age with the latent features from a pre-trained RNC model (Zha et al. 2023). The statistics of the datasets are summarized in Table in Appendix B.I
based SOTA regression method that combines conventional regression loss with a contrastive loss that pushes data latent features be closer if their labels are closer. 9) GAR: the proposed Gradient Aligned regression method via pairwise losses.
Models: for the tabular datasets with feature dimension less equal to 16, we utilize 6-layer-FFNN with hidden neurons as (16, 32, 16, 8) as the backbone model. For the tabular datasets with feature dimension larger to 16 and less equal to 128, we utilize a 6-layer-FFNN with hidden neurons as (128, 256, 128, 64) the backbone model; for IC50, we generate 15 linear heads for predicting the 15 targets the last prediction layer. All FFNN variants utilize ELU the activation in this work. For AgeDB (Scratch), we follow the previous work and employ ResNet18 8 as the backbone model (Zha et al. 2023). For AgeDB (RNC Linear Probe), we simply train a linear model based on the latent feature pre-trained with RNC.
Baselines: the baselines for comparison are: 1) MAE. 2) MSE. 3) Huber loss (Huber 1992). 4) Focal (MAE) (Yang et al. )021): variant for Focal loss with MAE loss, which put more learning weights on harder samples. 5) Focal (MSE) YYang et al. 2021: combine Focal loss with MSE loss similarly as Focal (MAE). 6) RankSim (Gong et al. 2022): a regression method that regularizes predictions to maintain similar order with the their labels. 7) RNC (Zhaa al., 2023): state-of-the-art (SOTA) contrastive learn- ing based pre-training method for regression that enhances the prior that the pairwise similarities from Iearned latent features should be close to pairwise similarities from data la- bels. 8) ConR (Kramatii et al. 2023): a contrastive learning
Experimental Settings: follow the standard procedure for data splitting, cross validation and hyper-parameter tun- ing. The full details are included in Appendix B.6

---

Gradient Aligned Regression via Pairwise Losses
Table 1. Tessting performance on eight different datasets/settings Mcan and standard deviation are reported. 'Gains' includes the percentages that GAR outperform the MAE and the best competitor (in parenthesis). The 'p-values' are calculated based on Student's s
t-test for the 'Gains". p-value < 0.05 usually means significant difference. 1 means the smaller the better: 1 means the larger the better.
Dataset
Method
MAE
MSE
Huber
Focal (MAB))
Focal (MSE)
RankSam
RNC
ConR
GAR
Gains
p valllss
MAE
1
4.95310.139 1999
4.69840
4973:00069)
439000
11.753(06614)
4.776001066
5.016601131)
4.60301775)
7.49
0.000445
Compieseiv
Concrete
RMSE
6.62400922)
2.51
6.52900226
6.623/0.011
6.54900237)
(4.503(0.710)
6.28300.06)
6.693(0155)
6.222/0.148)
6.28(3.69) 6M)
0.010.55
Stresgth
Pezrson 1
0.9190.0002)
0.918/0.008)
0.922(0.005)
0919000002
091900.000
0.67200000
0.923000022)
0.917100004)
0.929/0.004
1.140.61)
0.0:0.03)
Speannttt
0.917)0.004)
0:92300.03)
(.925)0.004)
0911)00004)
092100000
0703500059)
0.913)0)244)
0.931(0.003)
1.4600485 485
(.10..99)
MAB
0.500.0053
0.55/0.0041
0.536400004
0.11800004)
0.55210.08)
0.2100004)
0.54500.004
0.504)0.0055
0.494/0.084)
1.90100
0.110.11
Wine
RMSE 1
0.71500.00)
0.707(0.005)
0.693)0.005)
0666600007)
0.705000033
001
0.71000004)
0.712/0011
0.69).0044
3,53/ 46
0.000
Quality
Peurscu 7
0.573/0.01)
(.585
0.606)00005)
0.002
0. 0.59900.005)
0.583600111
0.56440.006
0
(613:00011)
7 05)
0.800.36)
Spearman
0.602/0.01)
0.6070.006)
0.62(00011
0622/000006
0.00500.005)
0.67700011
0.614(0UM07) 0.00700.005
0.35(0.007)
1.58(_11)
0.000.07
MAR
2.87500.662
3464(0.029)
2.442/0.379)
26930991)
(0,4(2
0615
515
2.400.1711
1.15(-8.14)
0.60.33)
Parkinson
RMSE 1
4.00300800)
1.175
4,596/03922 392)
3.7400.32
5.15700.4)
4 10.777(0022))
3.91
2.56).200))
11,3665.33
0
(Toual)
Pesrson 1
0.81000078)
0.87600.00 000)
D. (17)
0894200088)
0.877000077
0.2260.022
0 004)
0.862/00019)
(.92200009)
144(144)
0.1500211)
Speannist
0.8700044))
(56600005)
(2
087400007
0.85
0 011400051)
0.776000022)
0.842/01225
(.907(0.01)
4,24(1.36 16)
0.160.31
(84600464)
86700.99)
1O0N 283)
L5990.44)
2 36600.8811
6.91:0013)
2.20K0 (68)
2,88600 466)
65200.060
30
0,4600 34
Paekinson
RMSF 1
1336604491)
0670.077)
1,2360035 35)
2,93000 105)
4 116)
3612/0 136)
(1990-311)
2.680.1344
19.6010.45
0.000.01
(Motor)
Pesssou
0.91000229)
D. (655
0.310.005) 0.917)00119)
0.86300000)
0112500033
0.890000000
0.854000123
(.94500000)
3.71(1.48
Spearman
0902000032)
(5440.0044
0.912/0.19)
0.24000 006)
0.5560.009)
0, 01440.036)
0.8800000
0.64310.024
0.929/0.000
4,16(163)
0,500.01
3.650.007)
403
6.23800078
C000
Super
RMSE 1
11 13.6770006)
(2.6390.067)
1363000.039)
12.13:0 107)
17,059/0 408)
11 10 18)
L3 (360000))))
18.755.0.83))
24
0.00002)
Conductvvay
Pesrson 1
0.9120000
0.915000001)
O. 0.9300011
091820.0 04
0915:00011
0.8770.007)
054700.001)
0.018/0001)
0.949(0.002)
3.50V0.34)
0.000005
Spearmin
0.910000011
0.9240.0)
O 0013
0.2500.00
0.860.008)
0.9450.0011
6.917901001)
(.944/).002)
3.130.55)
08.0.18
1
1.66000022
0200000
36500001>
38600.011
13300000)
214
L.200255)
33500.000)
0.571-1-3)
00
IC50
Pearson 1
RMSE 1
0.85:00049)
1776000013 3)
0.0210
89000000000 008)
0.106:0.029) 0.
007)
69620 0033
1 7707:00014)
0.02500017))
0.012000266
1 18080 (3)
0.018/0.05
2:4540
1665/0.022)
0.200000033
0.302/0.008
1.70000))
252.9749..0 13)
0.50K-1 1.88)
0.0:0.0)
0.98/0.02)
Spearman
0.114:0.022)
0.06500039)
050470
0 033)
0064:0.04
0.3340006))
0.038:0.05)
18400.04)
0.2/0.024)
127.6440.500
0.000.01
MAA 1
6.5170.0093
6.3670.007)
6.552200855
0.286000044
2.201.3%)
(Scratch)
Ag:DB
Pearson
RMSE 1
0.91700.002
0,5090.008)
0.91400.00
(466
O. 0.915(0.001)
8,419(0.455)
S. 5.47600006))
0918:0.001 002)
0.14000001)
104)
8, 5.5960 188)
0.91600005)
0.916/0.002)
8.69/0.019)
(.921(0.001)
8.2680.05)
0:48:0.35)
2.83(179)
0.0:0.0)
0.0:0.0)
Spearman.1
0.91900.002)
0.917/0.001)
0.92/0.0)
092/0.002)
0.91600.000)
0.913000044
0,919H0L0033
0.925/0.002)
0.41(0288)
0.01/001L
0.836000003
0:8660002)
0838/0000 003)
0.32200,004)
0.833400007
0.837H00002)
0,8460.002)
(.150667)
0.0:0.01
AgeDD
(RNC Linese Probe)
Pearson
RMSE 4
MAB 1
6.124/0.009)
0.926/0.0)
5.077:0.026)
0.924/0.0)
6.130UUSS
8,399/0.04)
S. (.10000066)
0.924/0.0
6. 124000186)
6. AIEERO
0.9200000
S.073000220
6.139/0.233)
8.113:0.037)
0.92500.0)
8.054/0.014)
0.924/0.0)
6.66900802)
0,28:0223)
0.8900.55 55)
0:0:0.0)
0.286022)
1.0(1.0)
Speannan
0.927/0000
0:92710.1)
0.M2700UU)
0.2200000
0.926/0:30
0.927000000
(.00..0
1.01.00
R²1 7
0.855/0.001)
0.852)
0.852(00003
0.853/00001
0.35100.001)
0.854/0.0)
0.12(0.07)
0.2/0.35)
all the testing results are summarized in Table I GAR outperforms all the competitors from the perspective of Pearson Correlation Coefficient and Spearman's 's Correla- tion Coefficient over all the tasks, which indicates the pro- posed pairwise losses can capture the rank or order based in- formation better. GAR also enjoys competitive performance on MAE and RMSE evaluations. Finally, GAR achieves new SOTA results on AgeDB for both training from scratch and linear probe settings with R For the scratch set- ting, GAR achieves 6. 6.286(0.034) for MAE and 0.846(0.00) (0.002) for R² score, while the previous SOTA Is 6.40 for MAE and 0.830 for R² score; for the 2-stage linear probe on RNC pre-trained feature setting, GAR achieves 6. 6.069(0.022)) for MAE and 0.854(0.0) for R² score, while the previous SOTA is 6.14 for MAE and 0.850 for R² score (Zha et al. 2023).
Results:
conventional regression loss, such as MAE, MSE, etc. From the perspective of 'Loss Time', we find that GAR consis- tently takes slightly more time than conventional regression losses (but still in the same order/level). It is consistent with our theoretical result that GAR only takes linear time com- plexity. On the other hand, the pairwise latent feature based regression methods, such as RankSim, ConR and RNC, are clearly slower than GAR.
Ablation Studies for GAR: GAR constitutes three losses: CMAE LMSE dittnorm We conduct ablation studies on dif- ferent combination of the losses with the six real-world tasks on the tabular datasets. There are seven possible variants including GAR itself by different combinations of the three losses. Due to the limited space, we include the details in Appendix B.8 From the Tab. []] we find that the proposed GAR outperforms the other six variants. Besides, the pro- posed pairwise losses, L-fif and diffiorm can also improve the conventional loss CMAE even applied separately.
4.3. Further Analysis
/p=2
We provide practical running time comparison, ablation studies on different components, sensitivity analysis on a and batch size for GAR and difference with data standard- ization in this subsection.
Sensitivity on a: we conduct extensive hyper-paramen search for a in {0.1, 0.2, 0.25, 0.4, 0.5, 0.8, 1, 1.25, 2, 2.5, 4, 5, 10). The initial learning rate and weight decay are still searched in the same previous sets that are described in Appendix B.6 We include the full results in Appendix B.9 From the box-plot, GAR is little sensitive to a and is suggested to tune it in the 10.1, 10] range.
Running Time Comparison: we run all compared methods sequentially on an exclusive cluster node with AMD EPYC 7402 24-Core Processor 2.0 GHz. Each method runs 1500 epochs for the five tasks on the tabular datasets. The results are summarized in Table 2 From the perspective of 'All Time", we find that GAR is as efficient as the most efficient
Sensitivity on Batch Sizes: we additionally conduct ex-
8

---

Gradient Aligned Regression via Pairwise Losses
Table 2. Running time (seconds/epoch) five benchmark datasets. Repeated 1 1500 epochs, mean and standard deviation are reported. CCS": Concrete Compressive Strength. All Time": all the time consumed per epoch. 'Loss Time": : the time for loss calculation and loss
backward operation.
Dutasel
Concrete Compressive Strenyth
Wine Quality
Parkinson (Total)
Super Condactivity
IC50
Metric
All Time
Loss Time
All Time
Loss Time
All Time
Loss Time
All Time
Loss Time
All Time
Loss Time
MAE
0.089(0.013)
0.002(0.D)
0.169X0.03)
(.01
0.214(0.129)
0.045(0.113)
0.587(0.0388
0.162(0.001)
(.094(0.003)
0.0055(0.0)
MSE
0.0%00.014)
0.002(0.005)
0.168(0.03)
(.01(0.0)
0.210..24))
0.041(0.0)
0.566(0.002)
0.16(0.001)
0.095(0.003)
0.005(0.0)
Huber
0.09/0.015)
0.002(0.008)
0.168(0.03)
0.01(0.0)
0.21(0.046)
0.041(0.0)
0.15900.001) 0.581(0.038)
(.094(0.003)
0.005(00.0)
Focal (MAE)
0.09/0.013)
0.002(0.001)
0.177(0.33)
0.016(0.0)
0.214(0.024)
0.046(0.0)
0.603(0.037)
0.178(0.001)
0.096(0.041)
0.005(0.0)
Focal (MSE)
0.089/0.012)
0.002(0.0)
0.173(0.03)
0.015(0.0)
0.216(0.046
0.016(0.001)
0.665(0.061)
0.179(0.002)
0.094/0:003)
0.005(0.0)
RankSim
0.364(0.046)
0.275(0.045)
0.224(0.021)
0.06810.002)
(.982(0.033)
1.8(0.03)
6.876(1 (04)
6.42((0.977)
0.362(0.009)
0,271(0.006)
RNC
0.37](0.012)
0.275(0.01)
2.178(0.016)
2.014(0.015)
2.195(0.03)
2,02000299
0.275(0.004) 7.545(0.168)) 7,9890.2311)
0.178(0.003)
ConR
0.105(0.011)
0.014(0.004)
0,269r0.026
0,108(0.001)
(,300)))023)
0.13000.00))
0,943(0.057)
0,508(0.022)
0. 0.10000)002)
0.02(0.0)
GAR (ours)
0.086(0.012)
0.003(0.0)
0.177(0.026)
0.026(0.0)
0.22(0.024)
0.056(0.001)
0.64(0.039)
0.213(0.002)
0.09(0.003)
0.006(0.0)
periments on different batch sizes, f32, 64, 128, 256, 512), for GAR on Concrete Compressive Strength, Parkinson (To- tal) and Parkinson (Motor) datasets. Details and results are included in Appendix [B.10] GAR is also a little sensitive to batch size and suggested to be tuned in (32, 64, 128) in practice.
ground truth function. We demonstrate the effectiveness of GAR two synthetic datasets and eight real-world tasks five tabular datasets and one image dataset. Running time experiments are conducted to support that GAR is as efficient as the conventional regression loss. Experimental ablation studies are conducted to demonstrate the effective- ness separately for the two proposed pairwise losses.
Difference with Data Standardization: to emphasize the difference between Eq. 5 7 and data centering or standard- ization, we include extensive discussions and experiments in the Appendix [B.1 and Tab. 7 The proposed GAR joys similar superior performance on the datasets after data standardization as the main results in Tab. [
Acknowledgement
This research partially funded by NIH U01HG012069 and Allen Distinguished Investigator Award, a Paul G. Allen Frontiers Group advised grant of the Paul G. Allen Family Foundation.
5. Limitations
The scope for this work is limited improving learning label pairwise difference for regression clean data setting, without considering noises, outliers, or distributional shifts. Based on the Theorem 4 we relate GAR to capture the gradient of the ground truth function. However, GAR might be misled the training data does not reflect the underlying ground truth function. It would be an interesting future work to research the robustness for GAR under extensive settings.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Benkraiem, R. and Zopounidis, C. Preface: Regression methods based on or techniques and computational as- pects in management, economics and finance. Annals of Operations Research, 306:1-6, 2021.
6. Conclusion
In this work, we propose GAR (Gradient Aligned Regres- sion) via two novel pairwise losses for regression. Similar to the prior research, the proposed losses can help model preserve rank information from ground truths better. Dif- ferent with the prior studies, the proposed pairwise losses directly capture the pairwise label difference in the label space without any approximation, which additionally al- lows us to make transformations and reduce the quadratic complexity to linear complexity. We proved that the two proposed pairwise losses are equivalent to the variance of prediction errors and the negative Pearson correlation coef- ficient between the predictions and ground truths. Last but not least, provide theoretical insights for learning the pairwise label difference to capturing the gradients of the
Burges, C.,, Shaked, T., Renshaw, E.,, Lazier, A., Deeds, M., Hamilton, N., and Hullender, G. Learning to rank using gradient descent. In Proceedings the 22nd international conference on Machine learning, pp. 89-96, 2005.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Ciampiconi, L., Elwood, A., Leonardi, M., Mohamed, A., and Rozza, A. A survey and taxonomy of loss functions

---

Gradient Aligned Regression via Pairwisc Losses
in machine learning. arXiv preprini arXiv:2301.05579, 2023.
Huber, J. Robust estimation of location parameter. In Breakthroughs in statistics: Methodology and distribu- tion, pp. 492-518. Springer, 1992.
Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.0722 2015.
Keramati, M., Meng, L., and Evans, R. D. Conr: Con- trastive regularizer for deep imbalanced regression. arXiv preprint arXiv:2309.0651/ 2023.
Cortez, P., Cerdeira, A., Almeida, F., Matos, T., and Reis, Modeling wine preferences by data mining from physic- ochemical properties. Decision support systems, 47(4): 547-553, 2009.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980) 2014.
Klivans, A., Kothari, P. K., and Meka, R. Efficient algo- rithms for outlier-robust regression. In Conference On Learning Theory, pp. 1420-1430. PMLR, 2018.
Cranmer, M. Interpretable machine learning for science with pysr and symbolicregression, jl. arXiv preprint arXiv:2305.01582, 2023.
Legendre, A. M. Nouvelles méthodes pour la détermination des orbites des comètes: avec un supplément contenant divers perfectionnemens de ces méthodes et leur applica- aux deux comètes de 1805. Courcier, 1806.
Dara, S., Dhamercherla, S., Jadav, S. S., Babu, C. M., and Ahsan, M. J. Machine learning in drug discovery: review. Artificial Intelligence Review, 55(3):1947-1999, 2022.
Loshchilov, I. and Hutter, F. Sgdr: Stochastic gra- dient descent with warm restarts. arXiv preprint SSKiv:6608.0388.0 2016.
Dodge, Y. Least absolute deviation regression. The Concise Encyclopedia of Statistics, pp. 299-302, 2008.
Moschoglou, S., Papaicannou, A., Sagonas, C., Deng, J., Kotsia, I., and Zafeiriou, S. Agedb: the first manually collected, in-the-wild age database. In proceedings of the IEEE conference on computer vision and pattern recogni- tion workshops, pp. 51-59, 2017.
Gao, W. and Zhou, Z.-H. On the consistency of auc pairwise optimization. In Proceedings of the 24th International Conference on Artificial Intelligence, pp. 939-945, 2015.
Garnett, M. J., Edelman, E. J., Heidorn, S. J., Greenman, C. D., Dastur, A., Lau, K. W., Greninger, P., Thompson, R., Luo, X., Soares, J., et al. Systematic identification of genomic markers of drug sensitivity in cancer cells.
Niu, Z., Zhou, M., Wang, L., Gao, X., and Hua, G. Ordinal regression with multiple output for age estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4920-4928, 2016.
Nature, (83(7391):570-573) 2012.
Gong, Y., Mori, G., and Tung, F. Ranksim: Ranking simi- larity regularization for deep imbalanced regression. In International Conference Machine Learning, pp. 7634 7649. PMLR, 2022.
Pesme, S. and Flammarion, N. Online robust regression via sgd on the loss. Advances in Neural Information Processing Systems, 33:2540-225 2552, 2020.
Petersen, B. K., Landajuela, M., Mundhenk, T. N., Santi- ago, C. P., Kim, S. K., and Kim, J. T. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. arXiv preprint arXiv: MwXiv:191.004877 2019.
Hadsell, R., Chopra, S., and LeCun, Y. Dimensionality reduction by learning invariant mapping. In 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR 06), volume 2, pp. 1735- 1742. IEEE, 2006.
Ren, J., Zhang, M., Yu, C., and Liu, Z. Balanced mse for imbalanced visual regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7926-7935. 2022.
Hamidieh, K.
Superconductivty Data.
UCI DOI:
Machine Learning Repository, 2018. https///o..rg//0.4433//553
Hochreiter, S. and Schmidhuber. J. Long short-term memory. Neural computation, 98)):1335-1780 1997.
Rothe, R.. Timofte, R., and Van Gool, L. Dex: Deep expecta- tion of apparent age from single image. In Proceedings of the IEEE international conference on computer vision workshops, pp. 10-15, 2015.
Holzmüller, D., Zaverkin, V., Kästner, J., and Steinwart, I. A framework and benchmark for deep batch active learning for regression. Journal of Machine Learning Research, 24(164):1-88 2023.
Rudin, W. et al. Principles of mathematical analysis, vol- ume McGraw-hil New York, 1976.
10

---

Gradient Aligned Regression via Pairwisc Losses
Rumelhart, D. E., Hinton, G. E., and Williams, R. J. Learn- ing representations by back- pock-propggating errors. nature, :2366088) 1986.
Zhu, D., Ying, Y., and Yang, T. Label distributionally bust losses for multi-class classification: Consistency, robustness and adaptivity. In International Conference on Machine Learning, pp. 43289 433255. PMLR, 2023.
Solnon, M., Arlot, S., and Bach, F. Multi-task regression us- ing minimal penalties. The Journal of Machine Learning Research, 13(1):2773-2811 2012.
Zou, H. The adaptive lasso and its oracle properties. Journal of the American stafistical association, 101 1418- 1429, 2006.
Stanev, V., Oses. C.,, Kusne, A. G., Rodriguez, E., Paglione, J., Curtarolo, S., and Takeuchi, I. Machine learning mod- eling of superconducting critical temperature. upj Com- putational Materials, 4(1):29, 2018.
Zou, H. and Hastie, Regularization and variable selection via the clastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(2):301-320, 2005.
Tibshirani, R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267-285 288, 1996.
Tikhonov, A. N. et al. On the stability of inverse problems. In Dokl. akad. nauk sssr, volume 39, pp. 195-198, 1943.
Tsanas, A., Little, M., McSharry. P., and Ramig, L. Accurate telemonitoring of parkinson's 's disease progression by non- invasive speech tests. Nature Precedings, pp. 1-1, 2009.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, t., and Polosukhin, I. At- tention all you need. Advances in neural information processing systems, 30, 2017.
Wu, D. and Xu, On the optimal weighted lz regularization in overparameterized linear regression. Advances Neu- ral Information Processing Systems, 13:10112-10123, 2020.
Yang, Y., Zha, K., Chen, Y., Wang, H., and Katabi, D. Delv- ing into deep imbalanced regression. In International Conference Machine Learning, pp. 11842-11851. PMLR, 2021.
Yeh, I.CC. Modeling of strength of high-performance crete using artificial neural networks. Cement and Con- crete research, (8(12):11)) 1797-1808. 1998.
Zha, K., Cao, P., Son, J., Yang, Y., and Katabi, D. Rank-n- contrast: Learning continuous representations for regres- sion. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Zhang, S., Yang, L., Mi, M. B., Zheng, X., and Yao, A. Improving deep regression with ordinal entropy. arXiv preprint arXiv:2301.089155 2023.
Zhu, D., Cai, C., Yang. T., and Zhou, X. A machine learning approach for air quality prediction: Model regularization and optimization. Big data and cognitive computing. 2 (1):5, 2018

---

Gradient Aligned Regression via Pairwisc Losses
A. Proofs
A.1. Proof for Theorem
Proof
Recall that 8 1 ΣN 82 denotes the empirical mean.
CMSE diff
[f(x;) - -xxss) (f(x)) )((x,))1
N
N
N
Σ
Σ
-82
2N
i=
N
N
ΣΣ
(8% - 5% + -
2N
;=1==1
1
(0 -
N
(ar(
where the cross terms expanded out in the third equation can be canceled when iterate across all data samples.
A.2. Proof for Corollary 3
Proof
Recall that df f(xx) fxx)) and dVis V(xx)) )(x)), and df [df1,1, df1.2.
ay
[dV],, dV1,, 1I,NN-I,DNN...
df
dy
c==2 diffnorm
12
|drlla
|d
(by Theorem
df
dy
11
Var(F
22N2 (y)
1
f(x,) f(xx)-((x)) v(x,) - v(xx))
2N
Var(f)
Var(y)
N
N
[(x,) - -
V(xx)) (xxx)
+ 17 --
2N
Var(f)
Var(y)
1=1
(by algebras, there are 2N repeats)
2N
N
[xxx) (xx)) -13- -2(
- [xx; -J ((xx)) - -M/ + )(xx)) - 1/ 2
2N
Var(f)
Var(f)
Var(y)
Var(y)
N
(f(x;) - (yi 4)
2N
Var((f) VVr(yy
Cov(S.y) 1 - p(f.v).
Var(f))arl)
12

---

Gradient Aligned Regression via Pairwise Losses
A.3. Proof for Theorem 4
Proof
f(x1) f(x2) V(x1) - )(x2), (xx,,)(xx1)) (x2, (xx)) ED V*f(x) V*)(x,, v(x) e k= (1.
First, we prove that vx(x)= V)((), V(x, (x))) E D.
Take two neighbors X1, X2 that are infinitely close to an arbitrary x. By mean value theorem for multiple variables (Rudin
al. 1976]: f(x1) - f(x2) V)(()) (x1 X2), ((x)) - (xx)) = V)(() (x1 - X2), where 7 are two points
that are infinitely close x. Because f(xi) f(x2) v(xi) )'x2),
D, have
(E)) (x1 x2) V)( (x1 (2).
As X1, X2 are infinitely close, E, collapse to the same point x. Once X1, X2 are selected, define
c
where a is a unit length vector and we will decrease the scalar e to scale down the vector X2. As consequence,
-xx,x-xxxx--- V((E) (x1 1 x2) = Vx()) "limm->o e a)); limx, -x,xx-xx Vy(7) (x1 - X2) = Vy(() (lim,-0 e a).
Plug them back to the left and right side of the equation: fx()) (lim.-00 a) = ()(x) (lim, . a). by L'Hôpital's
rule vf(xx] T = v)(x)T a Him = v)(().
lim,, c v)(x)] a. Hence, V.(xx= v)(x) because can be any direction
constructed by X1, X2- Therefore, Vs(x)=
Next, we show V
V*)(x), V(x, E D, h = 3 by induction. Assume we have f*f(x) V*)(x), v(x, (x)) D, to show Vh+ f(x) V(x), ((x)) E D.
Denote i-th element from fxx as [x)(( Take two neighbors x1, X2 that are infinitely close to arbitrary x and apply mean value theorem for multiple variables solely on the i-th element: f(x))x, - [V* (x2)]; = v (x1
x2), [x))(x))]: ]]: [x²)(x))]: }], = +++))(()(x,
X2). where V(+) represents the higher order gradient taken for [V* fx)]s regarding to variables. It is worth noting that & can be different points for different which hamper the general
mean value theorem for vector-valued functions in the literature (Rudin et al. 1976) but doesn't harm the specific proof here.
Similarly, because (V* [xi)]; V [xx2)]: | yxx)); (x))
x2) i, VX1, X2, we have v [(E)((x x2)
(()())())xx X2))
As X1, X2 are infinitely close, Ei, 7/s collapse to the same point x, Vi. Utilize the same logic with L' L'Hôpital's rule in the previous first order gradient case, we can cancel the effects of the magnitude for x1 X2 closing to but preserve the direction information. Therefore, v f(x) = v Vi; thereby vk+1 f(x) = vx+l)(()).
Next, we prove the other direction.
f*x()) V*)((x, (x)) E D, k (1,2,..)
= f(x1) s(x2) -V(x))
)'(x2), (xI, (x2, D. fxx(x) = V*)(x),
E D = vf(x) = v)(x). v(x, M(x) E D. We take infinitely small but
infinitely many steps e,, i = I,.... 0o that constitute a path from x1 x2. By mean value theorem for multiple variables,
f(x) + E))())) - s(x + 1-1 ei) vs(xi + E TT(()))), )'xi + Σi )(xi + Sk-1 ei) = v) +
G () (cx).
Therefore, Σx= f(x] + E 6) - f(x + 4) = Em V(x + + k Et) - -x((1 + E et); and by telescoping sum, f(x2) (x1)= = V(x2) V(x)).
A.4. Proof for Theorem 6
Proof
13

---

Gradient Aligned Regression via Pairwisc Losses
M
CKI.
max PEAM Σ
log Li - a
pi
M
By Lagrangian multiplier:
M
M
CKL GAR (L1, (M:a) max min
Σ a20,8
p. log Ci (KL(p)
1
+
Σ
a:P:
b
Pi 1).
(12)
M
1=1
By stationary condition of p:
log Li -
P
exp
1).
M
Take the optimal p* back to E.q. 12
M
1
log Ci + az -
CKL
(I,, LM;a) min b +
exp
1).
(13)
a20,b
M
a, can be minimized as 0, Vi, because a 2 Further take the stationary condition for b:
log L; b*
EXp
1)
M
a
M
log Li 1).
exp(b"/a) = Σ
exp
M
1=1
Take the optimal solution 6* back to E.q. 13
CKL (LI,.. LM; = a log
Σ
exp
log Li
1)
+
(14)
M
a
M
log Li
= a log
Σ
exp
M
1=1
M
a log(
1
cl/a)
M
1=1
Therefore,
exp (I,,,, = -
1
cl
caya
When 1, we recover the Arithmetic Mean. Next we discuss the cases that - too and -> 0. By L'Hôpital's rule:
log(
Eiz Cl/ag
KL G (Li, 400
lim
LM;))
lim t-0+00
M7
1/a
Σ C./a log Li
lim
too
c./a
Σ M
log Ci
M
Hence, lima exp (CKL = ,
L.) By the same logic, lim exp (CKL
maxM Li.
14

---

Gradient Aligned Regression via Pairwisc Losses
1.0
MSE
MSE
0.5
0.5
0.0
0.0
-0.5
-0.5
-1.0
-20
20
-20
0
20
Figure 3, MSE predictions on synthetic datasets.
B. More Experiments
B.1. Dataset Statistics
The statistics for benchmark datasets are summarized as following. The AgeDB is image dataset where the image size is not identical but with fixed center crop size. The of feature is measured after data pre-processing model input.
Table 3. Dataset Statistics
Dataset
of instance
of feature
of targets
Conerete Compressive Strength
1030
8
Wine Quality
4898
11
Parkinson
5875
19
2
Super Conductivity
21263
81
1
IC50
429
966
15
AgeDB
16488
224x224
B.2, Data Pre-processing for IC50
For IC50 (Garnett al., 2012), there are some missing values for all the 130 drugs across all tumour cell lines, which make the model training and evaluation inconvenient. We filter out the drugs with more than 28% missing values, which gives us targets for 15 drugs. Then, we select the data samples (cell lines) that don't 't have any missing values for the 15 drugs, which end with 429 samples. We utilize dummy variables to encode data categorical features (tissue type, cancer type, genetic information).
B.3. MSE on Synthetic Datasets
Due to the limited space and high similarity between MAE and MSE, we include the predictions for MSE in Fig. 3 As mentioned in the main content and Fig. MSE is similar with MAE, which performs worse on capturing complex data rank pattern.
B.4. Symbolic Regression on the Synthetic Datasets
We adopt PySR (Cranmer 2023) on the synthetic experiments. We present the results for without trigonometric functions and with trigonometric functions prior knowledge in Fig and Fig. for Sine and Squared Sine datasets. The predictions (dashed line) is the mean prediction values based on 5 independent runs of PySR.
15

---

Gradient Aligned Regression via Pairwisc Losses
1.0
1.0
truth
PySR (no tri. func.)
0.5
0.5
truth
0.0
0.0
PySR (with tri. func.)
-0.5
-0.5
-1.0
-1.0
-20
20
-20
0
20
Figure 4. PySR on Sine dataset.
truth
truth
PySR (no tri. func.)
1
PySR (with tri. func.)
0
0
-1
-1
-20
20
-20
0
20
Figure 5. PySR on Squared Sine dataset.
B.5. Backbone Model Complexity Variation on Sine Dataset
To demonstrate the effectiveness for the proposed GAR. We varies the backbone model complexity from (2,4,6,8,10 10) hidden layers for the previous FFNN backbone model, besides of the previous hidden layers FENN results main content. o for GAR is tuned 10.5,1,2). All of other settings are kept the same as previously described in the main content. The results are presented in the Fig 6 Based the results, have two observations: increasing model complexity can generally improve the learning performance but may saturate at certain level (e.g. for MAE, 8 hidden layers model is better than 10 hidden layers model; for MSE, 6 hidden layers model is better 8 and 10 hidden layers model). 2) GAR consistently demonstrates its superiority over other baseline methods for comparison.
B.6. Experimental Setting on Real World Datasets
For tabular datasets, we uniformly randomly split 20% data as testing; the remaining 80% as training and validation, where we conduct pooold-coos-validdatee
with random seed set as 123. The total training epochs is set as 100 and batch size set as 256. The weight decay for each method is tuned (1e-3, le-4, le-5): we utilize SGD with momentum (set as 0.9) optimizer and tune the initial learning rate for baseline method in { (le-1, le-2, Ie-3, le-4, le-5), which stage-wised decreased by 10 folds at the end of 50-th and 75-th epoch. The switching hyper-paramett hyper-parameter (9 for Focal (MAE) or Focal (MSE) loss are tuned in (0.25, 4). The interpolation hyper-parameter fixed as 100 suggested by their sensitivity
for Huber loss and the scaling
for RankSim is tuned in (0.5, 1, 2) and the balancing hyper- pareererrrmtterr study in their Appendix C.4 (Gong et al., (022). The temperature hyper-parameter for RNC is tuned in (1,2,4); ): the first 50 epochs are used for RNC pre-training and the remaining 50 epochs are used for fine-tuning with MAE loss. The linear combination hyper-parametert a is fixed as 1, tuned in (0.2, 1, 4) for ConR, as suggested by the ablation studies in their Appendix A.5 (Keramati et al. 2023). The robust reconciliation hyper-parameter a for GAR tuned in (0.1, 1, 10). The model performance evaluated with the following 4 metries: MAE, RMSE, Pearson correlation coefficient,
16

---

Gradient Aligned Regression via Pairwisc Losses
MAE
MSE
RNC
MAE INE-Pearson
GAR
1.0
1.2
0.5
0.5-
0.5
0.3
0.0
0.0
0.0
-0.5
-1.0
0-0. 579,
0.438
2-0.282, Prx 9.355
-1.0
Jin 209. 0.255
-1.0
J- $-0.669. 0..455
-1.0
2- 0-5.005 0.522
-20
-20
-20
20
Hidden Layers
MAE
MSE
RNC
MAE-Pearson
GAR
1.0
1.0
A-0.325 0.362
5-0.505 0.644
2-0.676, 676 8.665
0.5
0.5
0.5
0.5
0.0
0.5
0.5
-0.5
-0.5
-0.5
-0.5
-1.0
2w0.515 51551, 0.580
-1.0
Hidden Layers = 4.
MAE
MAE-Pearson
1.0
f-0.62
0.708
1.0
0.5
0.5
0.5
0.5
0.0
-0.5
-055
00.5
-0.5
-1.0
-1.0
-20
-0
20
-20
-20
20
Hidden Layers 6.
1.0
Poue
1.5
1.0
600 Dew
Bou.
0.3
0.54
0.5
0.5
0.5
0.5
-0.5
-0.
0.5
1.0
-1.5
-1.0
Pim 0444 Doue
-1.0
-20
20
-20
20
-20
20
-20
-20
20
Hidden Layers 8.
MAE
MSE
RNC
MAE-Paarson
GAR
1.0
P-0.533, De.. 0.598
1.0
1.0
A-OUSES, A.. 0.77
0.5
0.5
0.5
0.0
0.0
0.5
0.5
-1.0
p-0.500, Arna
0.545
p- 3532, Pear 0.397
1.0
0.377 Pea 0.548
-20
20
-20
20
-00
20
-20
20
-20
20
Hidden Layers 10.
Figure 6. Predictions from different numbers of hidden layers for FFNN Sine dataset. stands for the averaged Pearson correlation coefficient with the ground truth over the trials; Pmux is the maximal Pearson correlation coefficient with the ground truth over the trials.
17

---

Gradient Aligned Regression via Pairwisc Losses
Spearman's s rank correlation coefficient. The best testing performance for each baseline method each training fold is decided by its best validation performance the same training fold over (initial learning rate, weight decay, pepeeial-hyperprrraee
the epoch for evaluation) combinations. For IC50 with 15 targets, we report the averaged value across all targets. The selected for different datasets on different data fold are summarized in Tab. 4 For AgeDB dataset, we follow the same setting for RNC linear probe task and similar setting for training from scratch task. We include the complete details in Appendix [3.7 Because RNC a pre-training method, it doesn't have evaluation on training from scratch task. Instead, all baselines linear probe are based on RNC pre-trained latent features. Because RankSim and ConR are proposed to manipulate latent feature, they don't 't have evaluation on linear probe task where the trainable model linear. We also include the coefficient of determination (R3) evaluation metric for AgeDB in addition to the previous four evaluation metrics, to compare with prior research.
Table The selected a on the tabular data respecting to different evaluation metrics. Standard deviations are included in the parenthesis.
Dataset
Data Fold
#1
#2
#3
#4
#5
Average
MAE
10
10
10
10
10
10.0(0.0)
Concrete Compressive
RMSE
10
10
10
10
8.2(3.6)
Strength
Pearson
10
10
10
10
8.2(3.6)
Spearman
10
10
10
8.2(3.6) 10
MAE
0.1
0.1
0.1
0.1
0.1
0.1(0.0)
Wine
RMSE
10
10
10
10
10
10.0(0.0)
Quality
Pearson
10
10
10
10
10
10.0(0.0)
Spearman
10
10
10
10
10
10.0(0.0)
MAE
10
10
1
10
6.4(4.409)
RMSE
0.1
0.1
0.1
10
0.1
2.08(3.96)
Parkinson (Total)
Pearson
0.1
0.1
0.1
10
2.08(3.96) 0.1
Spearman
0.1
0.1
0.1
10
2.08(3.96) 0.1
MAE
0.1
0.1
0.1
0.1
0.1
0.1
RMSE
0.1
0.1
0.1
0.1
0.1
0.1(0.0)
Parkinson (Motor)
Pearson
0.1
0.1
0.1
0.1
0.1
0.1(0.0)
Spearman
0.1
0.1
0.1
0.1
0.1
0.1(0.0)
MAE
10
10
1
10
10
8.2(3.6)
RMSE
10
10
1
4.6(4.409) 1
SuperConductivity
Pearson
0.1
10
1
1
0.1
2.44(3.801)
Spearman
10
10
10
10
10
10.0(0.0)
МAE
1
1
1
1.0(0.0)
RMSE
0.1
0.1
0.64(0.441)
IC50
Pearson
10
10
10
6.4(4.409)
Spearman
10
10
10
6.4(4.409)
B.7. Experiment Settings for AgeDB
We follow the the same experiment settings from the previous work (Zha et al. 2023). The age range is between 0 and 101. It is split into 0220//140021400 images for araining/valddationneesttt annealing is utilized for training (Loshchilov & Hutter 2016). The batch size is set to 256. For both training from scratch and linear probe experiments, we select the best learning rates and weight decays for each dataset by grid search, with a grid of learning rates from (0.01, 0.05, 0. 0.2, 0.5, 1.0) and weight decays from (le-6, le-5, le-4, J.-3). For the predictor training two-stage methods, we adopt the same search setting as above except for adding additional 0 weight decay the search choices of weight decays. The RNC temperature parameter search from 10.1, 0.2, 0.5, 1.0, 2.0, 5.0} and select the best, which is 2.0. We train all one-stage methods and the encoder of two-stage methods for 400 epochs, and the linear regressor of two-stage methods for 100 epochs. The hyper- parameters for all the compared methods are searched in the same region as described in appendix B.6. For training from scratch experiments, because the ConR (Keramati et al. (023) is proposed to use two data augmentations, we utilize two data augmentations for each training data for all compared methods. The a for GAR is selected as 0.1 for training from scratch setting and as 1.0 for linear probe setting. For each method, we randomly repeat five times with different random seeds to report the mean and standard deviation for evaluations.
sets. The SGD optimizer and cosine learning rate
18

---

Gradient Aligned Regression via Pairwise Losses
B.8. Ablation Studies on GAR
Recall that GAR constitutes of 3 losses: Le CMSE and diffnorm' Therefore, we conduct the similar experiments as described in Appendix [B6 for the extra six variants of GAR the tabular datasets as the ablation studies. The results summarized in Tab. 5 From the results, we observe that the proposed GAR with all the three components performs the best over all the variants. Besides, the proposed pairwise losses, -ift' LAAA even applied separately.
CMS and diftnorm can also improve the conventional loss
Table 5. GAR constitutes of 3 different losses; thereby there are 6 variants for ablation studies on GAR by taking off the components. + means the smaller the better; ? means the larger the better. Mean and standard deviation (in the parenthesis) values are reported. The overall ranks for each variant are reported in the last row.
Dataset
Methed
CMAE
4
AMAE and Aun
CNAA and L
C and D
GAR
MAE L
22.518(6.911) 5,242(0.212)
27.182(4.86)
4.917(0.092)
4.621(0.137)
19.104(6.491)
4.6033(0.075)
Concrete
Compressive
RMSB
7.188(0.369)
23.632(6.829) 829)
29.125(4.202)
6.502(0.21)
6.244(0.138)
(0.266/6.088)
6.222(0.148)
Sirength
Pearson 1
0.903(0.01)
0.92(0.003)
0.928(0.005)
0.922(0.003)
0.928(0.003)
0.925(0.005)
0.929(0.004)
Spearnan
0.903(0.000)
0,925(0.003)
0.935(0.005)
0.928(0.002
0.928(0.005)
0.929(0.005)
(.931(0.003)
MAE L
0.494(0.004)
5.354(0.241)
4.386(1.165)
0,49800.000 006)
0.496(0.006)
(.278((885))
0.494(0.004)
Wine
RMSR
0.711(0.004)
5,381
4.791(1.053)
0.694(0.005)
0.691(0.003)
4.347(1
0.69(0.004)
Quality
Pearson
0.582(0.003)
0.622(0.008)
(0.00(
0.607(0.009)
0.608(D.013)
0.615(0.003)
0.613(0.011) I)
Spearman
0.612(0.008)
0.637(0.006)
0.636(0.005)
0.636(0.006)
0.631(0.01)
0.638(0.005)
0.635(0.007))
L
3.358(0.045)
24.393(2.48) 27,5630)902)
2.746(0.068)
2.68(0.13)
26.861(0.707)
2.64(0.071)
Parkinson (Total)
RMSB
5.384(0.05)
28 28.306(1 (0.100(1111))
26.489(1.7995
4.214(0.14)
4.402(0.253)
27.363(1.036)
(.258(0.206)
Pearson 1
0.866(0.002)
0.934(0.012)
0.951(0.004)
0.924(0.005)
0.912(0.01)
0.939(0.01))
0.922(0.009)
Spearman f
0.846(0.008)
0.922(0.012)
0.942(0.004)
0.91100006)
0.897(0.0133
(.0310
0,907(0.01)
MAE L
2.277(0:073)
811)
(.672000885)
L853(.1774
19. 7077
16652(00669)
Parkinson (Motor)
RMSF J
3.927(0.036)
19.687(0.583)
19.37 (5,
2.701
3,116(0.361
19. 799.0990))38)
2.68(0.134)
Pearson f
0.875(0.002))
0,942(0.00 009)
0.944(0.003)
0.942(0.005)
0.919(0.022)
0.943(0.009)
0.943 006)
Spearman
0.868(0.002)
0.939/0.009)
0.94(0.004)
0.93800000)
0.9130).224)
0.94(0.009)
0.939(0.006)
MAF
6,577(0.051)
20.445(0.88 88)
(427(0.631)
6,2930.072)
6.238(0.551)
18.616(1.762)
6.257(0.027)
Super
RMSB 4
1188150..11)
23 23.676(0.778)
17.094(0.607)
10.719(0.178)
10.935(0.042)
2..514(1.661)
10.745(0.183)
Conductivity
Pearson 1
0,94(0.001)
0,95(0.002)
0.947(0.001)
0.95(0.002)
0.948(0.0)
0.951(0.001)
0.949(0.002)
Spearmun
0.942/0.001)
0.942(0.001)
0,9390.002)
0.943(0.001)
0.945(0.001)
0,942(0.001)
0.944(0.000)
MAE
(.359(0.092)
3.815(0.009)
3.549(0.044)
1.36(000006)
(,62(0.012)
3.736(0.021)
1.59((0000))
RMSB| |
1.70%00012)
4.1340.011)
3.87(0.044)
1.706(0.003)
L705(0.018)
4.053(0.02)
1.7(0.05)
IC50
Pearson f
0.041(0.045)
0.146(0.022)
0.293(0.034)
0.07(0.062)
0.313(0.023)
0.313(0.021)
0,302(0 028)
Spearman
0.085(0.035)
0.152(0.034)
0,267(0.019)
0.115(0.031)
0.268(0.009)
0.263(0.029)
0.26(0.024)
Overall
Rank
5,208
5.312
4.104
3,583
3,312
4.104
2.375
B.9. Sensitivity Analysis
We include the sensitivity results for GAR on Concrete Compressive Strength, Wine Quality, Parkinson (Total), Parkinson (Motor), Super Conductivity as in Fig. 7 9 I0 where we present the box-plot with mean and standard deviation from .folddcross-validattie
The evaluation procedure is identical as the main benchmark experiments stated in Appendix (B.6) except that we fix different for GAR. As we mentioned in the main content, the ar is not very sensitive, but it is suggested to be tuned in [,01,10] region.
1. Different Batch Sizes
We conduct experiments different batch sizes (32, 64, 128, 256, 512) for GAR. The experimental setting is kept the same as the main experiments in Appendix B.6 The results are included in Tab. 6 Because the proposed GAR learns both the mean and variance of the label values by conventional loss and proposed pairwise losses (by Theorem
I we additionally provide the convergences curves for the estimated means and standard deviations in Fig. 12 From both the table and the figures, we observe a strong correlation between more accurate estimation of mean and standard deviation to better testing performance (Batch Size 32 on Concrete Compressive Strength and 64 on Parkinson Motor).
1. Difference with Standardization
The formulation for CMSE and C==2 in Eq. and Eq. might remind reader of data centering or standardization. In this diffform subsection, we highlight the difference:
CMSE and diffnorm 'centralize' or 'standardize' not only for ground truth label, but also for prediction. Besides, the predictions are dynamic during model training process.
2
19

---

Gradient Aligned Regression via Pairwisc Losses
test. MAE
Test RMSE
30
25
20
20-
15
0.
05
ON
1.23
23
15-
05
0.8
2.
10
10
0.25
0.5
08
0.75
2.5
test Poarson
test Spearman
0.93
0.91
0.88
0.80
0.87
0.2
6.25
0.5
08
1.25
0.25
0A
1.30
2.5
Figure 7. Different a (x-axis) for GAR on Concrete Compressive Strength.
test MAE
test RMSE
0.54
0.72
0.53
0.72
0.70
0.50
0.89
0.25
0.25
Pearson
test Spearman
0.64
0.63
0.62
0.58
0.00
025
5.25
2.5
Figure Different a (X-axis) for GAR on Wine Quality.
Table 6. Testing performance with different batch sizes.
Dotsst
Concrate Compressivv Strength
(eeal)
ParkinsonMMoto)
Buzch
Speamean
RMSE
Speamtan
RMSE
3866/0.133)
5.625(0.14))
(.942(0.005)
0.944(0.044
1.87500.000
3.95%(1.684
0.925/0.055)
0.44400014)
1318600044)
2.50100.201
0.964/0.008)
0.942/0.M77)
64
3.9660.189 183)
5,70/00 257)
0.99900005)
0.93720.000
1.431(0.109)
2.737(0.182)
0.967(0.004)
0.961(00077)
1.84(00033
2.321(0.098)
(.958/LL08)
0.954/0.114)
128
4.1670.113)
$38370.176)
(15770000)
0-938(0)002)
0.954(0.005)
1.3150006)
2455(0.057)
0.953(1004)
0.929/0.005)
256
4660(00075)
0.929/0.004)
0.931(0.003)
2.6400.071)
4.258/0.000
0.922(0.099)
0.907/0001
1.65200.069)
2.68/0.134)
0.943/0.006
0.939.0.006)
512
4.989(0.199)
6.568(0.226)
0.91700.066
0.921(0.088
4.032/0.056)
3.679X0.04)
0.855/0.007
0.83(0.00)
2.70900.662
3.819K0.126)
0.891(0.016)
0.883/00155
20

---

Gradient Aligned Regression via Pairwisc Losses
test MAE
test RMSE
5.00
2.8
4.50
4.25
3.50
0.2
0.5
08
0.75
0.0
Spearman
0.92
0.90
0.88
0.89
02
0.25
05:
0.8
0.75
0.8
Figure 9. Different (x-axis) for GAR on Parkinson (Total).
test MAE
test RMSE
38
22
3.6
3.4
2.0
1.8
2.8
2.
1.6
2.4
02
Spearman
0.95
0.34
0.93
0.92
0.92
0.91
0.90
0.80
0.2
0.5-
0.8
1.25
0.75
0.5
0.8
2.5
Figure 10, Different (X-axis) for GAR on Parkinson (Motor).
21

---

Gradient Aligned Regression via Pairwisc Losses
test MAE
test RMSE
--
0
20.0
22.5
17.5
20.0-
17.5
15.0-
10.0
12.5
73-
10.0
0.2
0.25
0.25
Spearman
0.940
0.952
0.944
0.942
0.948
02
0.5
0.75
Figure 11. Different a (x-axis) for GAR on Super Conductivity.
The proposed GAR, Alg. [ learns the mean and standard deviation dynamically for each mini-batch during training, rather than for the whole dataset.
We tested all the baselines on the five tabular datasets with standardization on both data features and labels, where the results are summarized in Tab. 7 From the results, the proposed GAR enjoys similar superiority on the standardized datasets the previous main results in Tab.
Table Testing performance on standardized tabular datasets/settings. Mean and standard deviation are reported. 'Gains' includes the percentages that GAR outperform the MAE and the best competitor (in parenthesis). The 'p-values' are calculated based on Student's
t-test for the 'Gains', p-value < 0.05 usually means significant difference. $ means the smaller the better; means the larger the better.
Sundardizal Datauct
Method
MAE
MSE
Huber
Focal (MAE)
Focal (MSE)
RackSim
RNC
Coolt
GAR (ours)
Guiss (8)
p-values
0:28400.003)
6,20000.065
0.278/000033
2:22(1.77)
0,000 24)
Coupressive
Coonrete
RMSE
0.390002)
0.987
0.396001)
0.3860.008)
0.5010:014)
8.004/0.005)
0.38V00071
6.37800080
0.3000011)
2.56(-0.6)
0.410.7)
Suength
Reason
(.92410008)
091900055)
091500000
0.92100004)
092100004
0.674/0.09)
0.925(1.033
0.925000922
0.925(10044)
0.11/0.00
0.68/0.93)
Speamman
0.92710.0033
092200003
0915/0004
0.92210.006)
0.9240.008
0.70610.00 (09)
0.92910.092)
0.92710.005)
0.929(0.022)
0.28/0.01
0,1000977)
0.57400.008)
0603:0200
061560000
0.5900001)
06300005
0.595/0.005)
061610.005)
0.579000025
0.565(0.077
1.561.56)
0.82400.005)
0.790.005
0.79400006)
0.81710.009)
0881100.006
Quality
Pearson
0.56600.044
061400011
061300005
0.66600005)
06110.0000
0.571000011
0,587(0.00)
0.5860.007)
0.61800003)
9:220..77)
0.000
Speaman
0.609(0007)
0.3/0.009)
06400006)
0.604/0.08)
066230.006
0.693/00006
0.0910.005)
0.616/00055
0.38/00007)
S 18)
MAE.
0.3300016)
D. 20000000
0.35400184)
0.341(1)999
X81122113
RMSE
0.507100133
0473300021
0.500000113
0457000014)
0.355/082)
24.14(18.75)
(Tota))
0.86400.00)
0.88300006)
0.866000008
0.87500.008
0,203000022)
0.551000005)
0.53300000)
0.92410808))
6.9144633
Spearman
0.84100.00)
02600/006)
08350.00
0.84500006)
0.82200006)
0,188/0.043)
(.00000009
0.66000000)
0,90900.008)
(0515.57)
0,000
MAE
0.3090.017)
(311
O. (.363
0.314(0.03
0.3390022
03130000
0.3910.011)
0. 0.22010.14)
27.1327.13
Parkisson
RMSE
0.43700.028
0.140.004
0.49500.01)
0.466000266
0.99310.003)
(47770.024)
0.512100011
24.0519.555
0.000.0
(Motoe)
0.87100.015)
088660006)
0.85600002
0.85800006)
0.622000114
07990.0077
0.8770.000)
0.92800008)
6.51(4.72)
Spearman
0.86400.016)
08720018
084700001 001)
O (88000000
(851000115)
0. 046)
88660.000
0.84700.000 0000
(.2200000)
6.79(5881)
0.000
MAR
0:34400.013
(355000
(0
D. 1000000
0024100001
RA507.00
Super
RMSE
0.306/0.002)
(3800000
O. 035900000
O 0.904000088
(.762/00633)
O. 0.385000011
0.304/00011
0.259/0.003
9.687.00
Confuctiity
0.92910.00
0:92000111
0917000
0.811000000
0.920.0
(.934000011
16661.14)
Spearman
0.916/0.001
0907000011
000100001)
0.920000011
0,9000021
0.52/100004)
0916100001
091710.001)
0,9280.0011
1.251_177)
MAB
0,0200000)
03240003
08200000
0.82100.002
0.30100002
0.REN0.00)
0.81410.001)
0.79H0.177)
3.72.911
0,0100021
RMSE L
1.02600.002)
1016000011
000
L025/0.002)
(0210.002)
100100000
I 100600.002
(002410.000)
0.992/0.824)
3.3312335)
0.02/0.08)
IC50
Peasson
0.07990.055)
1166600000 027)
(2230055)
0.88000.019)
O. (5600.021)
0.015000055)
-013300017
0,234000113
0.340833)
278. 81138113..6 82)
0.000
Speammnn
0.1200.25))
0,1100.2)
0,026/0.441
0.277(11027)
22

---

Gradient Aligned Regression via Pairwisc Losses
Compressive Strength
Compressive Strength
40
15
30
20
bs#64
bss
b5=256
bs= 256
b5=
512
Truth Mean
Tuuth Standard Deviation
60
Parkinson (Total)
Parkinson (Total)
10
20
bs:64
bs:
bsm 128
bsm 128
bs:256
bs=
bsmS12
bam 512
Mean
Standard Deviation
100
20
60
100
Parkinson (Motor)
Parkinson (Motor)
25
20
15
10
bs=32
bse
bS-64
bs#128
bss 128
256
bs=512
bs 512
0
20
40
60
80
100
20
40
60
B0
100
Figure 12. Batch Sizes on mean and standard deviation estimation for GAR
23


Reviews:
[{"review_id": "6PmmPs6QNK", "reviewer": "Reviewer_qJh5", "summary": "In this work, the authors focus on the problem of regression with pairwise losses. Firstly, it is revealed that the conventional regression losses (e.g., MSE and MAE) ignore the pairwise information of training samples and can have larger error variance. Inspired by the success of existing pairwise losses, the authors provide a loss that penalizes the mismatch between prediction and target's pairwise difference instead of pointwise deviation, which can be computed in linear time w.r.t. sample size. A relaxed version is also provided, which allows the mismatch of pairwise difference's magnitude. An aggregation scheme is also provided to combine the conventional loss, pairwise loss, and the relaxed version with only one hyperparameter. Finally, the intuition behind learning pairwise difference is shown by its equivalence to learning the ground truth function's gradient. Experimental results demonstrate the effectiveness of the proposed methods on various regression benchmark datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your invaluable comments for our manuscript! Here are our responses:\n\n1. \"*The cited works primarily focus on regression. However, similar types of loss functions have also been explored in the context of AUC optimization...*\"\n\n\t**Response**: we sincerely thank you for raising this point and we agree that the formulations share a similarity between the proposed pairwise regression losses and the pairwise losses utilized for AUROC optimization. Nevertheless, we will also highlight the difference: the propose pairwise regression losses do not define positive/negative pairs and the target is continuous. We will add the discussion in the related work section with the recommended reference.\n\n2. \"*In this paper, the target is assumed to be a deterministic function $\\mathcal Y$. This assumption is more restrictive than the commonly considered stochastic function setting. Generalizing the results to the stochastic case appears to be non-trivial, as key conclusions (e.g., Theorem 4) rely on this deterministic assumption.*\"\n\n\t**Response**: Thank you for pointing out the implicit assumption and the potential extension on our current Theorem 4. Yes, its current proof relies on the assumption that $\\mathcal Y(\\mathbf x)$ is deterministic. We will explicitly add the assumption in our updated manuscript. It is non-trivial to extend to arbitrarily general stochastic $\\mathcal Y(\\mathbf x)$ setting but we're happy to provide the following extensive discussion and an extended theorem that works on the stochastic function under certain forms (should cover the \"*commonly considered stochastic function setting*\" as you mentioned) by expectation trick.\n\n\tIf $\\mathcal Y(\\mathbf x)$ is stochastic, the $\\nabla\\mathcal Y(\\mathbf x)$ would also be stochastic. Therefore, the deterministic model $\\nabla f(\\mathbf x)$ inherently could not capture stochastic $\\nabla\\mathcal Y(\\mathbf x)$. A simple example could be $\\mathcal Y(\\mathbf x, z) = m(\\mathbf x) +z\\cdot s(\\mathbf x)$, where $z\\sim N(0, 1)$ is stochastically sampled from the standard normal distribution; $m(\\mathbf x)$ and $s(\\mathbf x)$ represent the ground truth mean and standard deviation for general heteroskedastic case. In this case, the fitting model have to learn with the randomness (like the reparameterization trick in VAE) in order to capture the stochastic $\\nabla\\mathcal Y(\\mathbf x, z)$. Similarly as Theorem 4, we can try to define the extended theorem: \n\t$f(\\mathbf x_1, z_1) - f(\\mathbf x_2, z_2)=\\mathcal Y(\\mathbf x_1, z_1) - \\mathcal Y(\\mathbf x_2, z_2), \\forall (\\mathbf x_1, \\mathcal Y(\\mathbf x_1, z_1)), (\\mathbf x_2, \\mathcal Y(\\mathbf x_2, z_2))\\in\\mathcal D; \\forall z_1,z_2\\sim N(0,1),$\n\t**iif**\n\t$\\nabla^kf(\\mathbf x, z)=\\nabla^k\\mathcal Y(\\mathbf x, z),\\forall (\\mathbf x, \\mathcal Y(\\mathbf x, z))\\in\\mathcal D; \\forall z\\sim N(0,1)$.\n\n\tWe still could use the same logic for the proof by treating $z$ as an extra variable. However, the difficulty is learning the stochastic $z\\cdot s(\\mathbf x)$ term for the model because $z$ is unknown during training (or even assume the distribution for $z$ is known, the sampled value is still unknown). $f(\\mathbf x, z)$ may not converge to $\\mathcal Y(\\mathbf x, z)$.\n\n\t**However, we could be more conservative and only argue the model can capture the expected gradient on $\\mathbf x$**, i.e. $\\mathbb E_z\\big[\\nabla_{\\mathbf x}\\mathcal Y(\\mathbf x, z)\\big]$, where deterministic model $f(\\mathbf x)$ is competent. We can try to define the extended theorem:\n\t$f(\\mathbf x_1) - f(\\mathbf x_2)=\\mathbb E_{z_1}\\big[\\mathcal Y(\\mathbf x_1, z_1)\\big]- \\mathbb E_{z_2}\\big[\\mathcal Y(\\mathbf x_2, z_2)\\big], \\forall (\\mathbf x_1, \\mathcal Y(\\mathbf x_1, z_1)), (\\mathbf x_2, \\mathcal Y(\\mathbf x_2, z_2))\\in\\mathcal D; \\forall z_1,z_2\\sim N(0,1),$\n\t**iif**\n\t$\\nabla^kf(\\mathbf x)=\\mathbb E_z\\big[\\nabla^k_{\\mathbf x}\\mathcal Y(\\mathbf x, z)\\big],\\forall (\\mathbf x, \\mathcal Y(\\mathbf x, z))\\in\\mathcal D; \\forall z\\sim N(0,1)$.\n\n\tNotice that $\\mathbb E_{z}\\big[\\mathcal Y(\\mathbf x, z)\\big]=m(\\mathbf x)$, $\\mathbb E_z\\big[\\nabla^k_{\\mathbf x}\\mathcal Y(\\mathbf x, z)\\big]=\\nabla^km(\\mathbf x)$. Our previous proof still goes through by replace $\\mathcal Y(\\mathbf x)$ as $m(\\mathbf x)$. Actually, this extended theorem works for any stochastic function as far as the stochastic term can be decoupled as 0 mean and can be canceled under expectation. \n\t\n\tWe will add this discussion and extended theorem to the updated manuscript!\n\n\nThanks again for your constructive comments! Please let us know if our responses address your questions.", "claims_and_evidence": "The claims in this submission are formulated as theorems and are supported by rigorous proof.", "methods_and_evaluation_criteria": "The proposed method is compared with state-of-the-art on various real-world datasets, which make sense for the problem of regression.", "theoretical_claims": "I have examined the proofs of Theorem 1, Corollary 2, and Corollary 3 and found no apparent flaws.", "experimental_designs_or_analyses": "A detailed ablation study is conducted, including sensitivity analyses on various hyperparameters. Additionally, the running time analysis experimentally confirms the linear time complexity of the proposed method.", "relation_to_broader_scientific_literature": "The relationship between this work and previous studies on pairwise losses for regression is discussed, with a particular focus on contrastive learning-based methods.", "essential_references_not_discussed": "The cited works primarily focus on regression. However, similar types of loss functions have also been explored in the context of AUC optimization. For example, [1] investigates the consistency of AUC pairwise optimization, which is relevant to the discussion and it is encouraged to consider them for a more comprehensive review of related methodologies.\n\n[1]. Gao W, Zhou Z H. On the Consistency of AUC Pairwise Optimization. IJCAI, 2015.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The result of Theorem 4 is non-trivial, which reveals the rationale of learning pairwise difference.\n\n2. The writing is clear and easy to follow, and the motivation of each proposed method are elaborated.\n\nWeakness:\n\nPlease see the questions.", "other_comments_or_suggestions": "Please see the questions.", "questions_for_authors": "In this paper, the target is assumed to be a deterministic function $\\mathcal{Y}$. This assumption is more restrictive than the commonly considered stochastic function setting. Generalizing the results to the stochastic case appears to be non-trivial, as key conclusions (e.g., Theorem 4) rely on this deterministic assumption.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vpbosJffCs", "reviewer": "Reviewer_pc7o", "summary": "This paper proposes to use pairwise comparison to improve regression accuracy. Authors propose to focus on the difference of pairs, and propose two loss terms comparing the differences of pairs. Authors then show the equivalence form that enjoys linear computational efficiency using specific loss functions. Furthermore, authors propose a reformulation to reduce the number of hyperparameters to only one, and a computational stability trick. Extensive empirical evaluations are conducted to show the efficiency and improvement of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your invaluable feedback on our manuscript! Here are our responses:\n\n1. \"*However, the significance of how the proposed method is located in the literature is not very clear for readers who's not familiar with the specific literature of using pairwise data for regression. For example, being said \"However, these methods heuristically approximate label similarity information as a relative rank given an anchor data point, which may lead to the loss of some original pairwise quantitative information.\", it would be desirable to have more intuitive demonstration on the weakness of existing methods. What does the anchor data point mean in this case?*\"\n\n\t**Response**: Thank you for pointing out this for us to improve the readability for the broader audience. We plan to modify this part on introduction as follows:\n\n\tFor those latent space based pairwise regression methods, they define an 'anchor data point', which either act as a reference point for calculating the relative ranks to the anchor for the other data (Gong et al., 2022), or the ranks are consequently utilized to construct positive and negative pairs (Zha et al.,2023; Keramati et al., 2023). We refer the readers to the related work section (line 62, right column) for more information about those methods.\n\n\tOne of the imperfectnesses for those approaches is conceptually to see: the original pairwise label similarities (like the pairwise label differences in our work) are converted to ranks or positive/negative pairs. There is an approximation loss because the conversion is single directional (and the original label similarities cannot be recovered from the coarser converted information).\n\nPlease let us know if our explanation addresses your question. Thank you again for your useful comments!", "claims_and_evidence": "The three claims in the paper, as well as two advantage points listed in introduction are well supported by the demonstration.", "methods_and_evaluation_criteria": "The proposed methods improve regression accuracy, and common regression measures are used for evaluation.", "theoretical_claims": "I have checked the presentation of theoretical claims and their brief prood sketch in the main paper.", "experimental_designs_or_analyses": "The classical regression experiment setting is conducted and the implementation codes in python is also provided. Ablations studies including the only hyperparameter are properly conducted.", "relation_to_broader_scientific_literature": "As mentioned in the paper, this paper related to the broad regression literature, and shows the efficient utilization of pairwise data on improving regression accuracy.", "essential_references_not_discussed": "No to the best of my knowledge.", "other_strengths_and_weaknesses": "For strengths, the paper is well self-contained by proposing a pragmatic method with theoretical support and extensive empirical evaluations. It also clearly demonstrates its limit of considering single scalar regression with no noise.\nHowever, the significance of how the proposed method is located in the literature is not very clear for readers who's not familiar with the specific literature of using pairwise data for regression. For example, being said \"However, these methods heuristically approximate label similarity information as a relative rank given an anchor data point, which may lead to the loss of some original pairwise quantitative information.\", it would be desirable to have more intuitive demonstration on the weakness of existing methods. What does the anchor data point mean in this case?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I have listed my questions in above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HEMeiOSi0k", "reviewer": "Reviewer_DxkR", "summary": "This paper proposes GAR for solving regression problems while maintaining label similarity. The authors introduce two pairwise label difference losses and explain their theoretical insights. Various experiments and ablation studies demonstrate that GAR outperforms other baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your invaluable comments! Here are our responses:\n\n1. \"*The proposed method mainly focuses on the clean data setting, while other baselines in the experiments, for example, RNC, have applications in improving generalization and robustness.*\"\n\n\t**Response**: RNC is a nice regression method that incorporates pairwise label similarities with contrastive learning style method in latent feature space. It enjoys better generalization and robustness with $\\delta$-ordered feature embedding, which regularizes hypothesis space to be smaller. Although we limit our scope on clean data setting, we want to highlight the following points:\n\t\n\ti) The proposed GAR is a novel loss function for regression and RNC is a kind of regularization on latent feature space. Those methods regularizing the latent feature space, such as RNC, are complementary to GAR, as our results on AgeDB (Table 1) demonstrate that GAR with RNC can enjoy better performance.  \n\t\n\tii) Different with the latent feature space based methods, GAR is proposed with more focus on the computational efficiency and theoretical insights (Theorem 1, Corollary 3 and Theorem 4), which demonstrates the unique advantages for GAR among the other pairwise regression methods.\n\n2.  \"*$\\mathcal L_{\\text{diffnorm}}$ is claimed to be a relaxed version of $\\mathcal L_{\\text{diff}}$. It is unclear why the objective function includes both $\\mathcal L_{\\text{diffnorm}}$ and $\\mathcal L_{\\text{diff}}$, rather than just $\\mathcal L_{\\text{diffnorm}}$*\"\n\n\t**Response**: As the Theorem 4 shows that fitting the label difference equals to fitting the gradients of the underlying ground truth function, therefore $\\mathcal L_{\\text{diff}}$ is optimized to fit $\\nabla\\mathcal Y$ and the relaxed version $\\mathcal L_{\\text{diffnorm}}$ is optimized to fit the normalized $\\nabla\\mathcal Y$ (captures the function shape). \n\t\n\tIf we only include $\\mathcal L_{\\text{diffnorm}}$ to fit the normalized $\\nabla\\mathcal Y$, the objective function only could capture the function shape of $\\mathcal Y$ and could lose the mechanism on capturing the original $\\nabla\\mathcal Y$ with magnitude information on gradients. \n\t\n\tOn the other hand, adding $\\mathcal L_{\\text{diffnorm}}$ doesn't involve any extra hyper-parameter tuning burden, as we balance all the terms by a DRO framework (Theorem 5). Besides, our ablation study (Table 5 in the appendix) shows that including both $\\mathcal L_{\\text{diff}}$ and $\\mathcal L_{\\text{diffnorm}}$ achieves the best overall performance. Only including $\\mathcal L_{\\text{diffnorm}}$ is the suboptimal (although it is the second best).\n\n3. \"*Other Comments Or Suggestions: 1. $\\mathcal L_{\\text{diff}}^{\\text{MSE}}$ in Theorem 1 is not defined. 2. $L_{\\text{diff}}$ should be $\\mathcal L_{\\text{diff}}$ at Line 111.*\"\n\n\t**Response**: We appreciate your careful checks on our writing to improve the readability. \n\t\n\tWe will update $\\mathcal L_{\\text{diff}}^{\\text{MSE}}\\coloneqq\\frac{1}{N^2}\\sum_{i=1}^{N}\\sum_{j=1}^N \\ell_{\\frac{1}{2}\\text{MSE}}(f(\\mathbf x_i)-f(\\mathbf x_j), \\mathcal Y(\\mathbf x_i)-\\mathcal Y(\\mathbf x_j))$ right above Eq 4.    We will correct the typo at Line 111 as $\\mathcal L_{\\text{diff}}$.\n\n\nThank you again for providing thoughtful comments! Please let us know if our responses address your concerns.", "claims_and_evidence": "The claims are supported by authors' theoretical analysis and experiments.", "methods_and_evaluation_criteria": "The authors introduce two pairwise label difference losses and explain their theoretical insights. They also propose a DRO-based method to combine the two losses into a single objective function. The experiments mainly focus on test set loss and statistical correlations, and the results show that GAR can improve the model's fitting ability.", "theoretical_claims": "The authors mainly prove that the two proposed losses are equivalent to the variance of prediction errors and the negative Pearson correlation coefficient between predictions and ground truth, as well as the theoretical insight between learning pairwise label difference and learning the gradients of ground truth function. Proofs seem correct to me.", "experimental_designs_or_analyses": "The experiments mainly focus on test set loss and statistical correlations. Extensive ablation experiments validate the effectiveness of GAR. The only drawback is that the combination of the three loss functions relies on a heuristic method and is sensitive to hyperparameters.", "relation_to_broader_scientific_literature": "As the authors point out in the Related Work section, their proposed method has applications in contrastive learning.", "essential_references_not_discussed": "Related works are well discussed as far as I am concerned.", "other_strengths_and_weaknesses": "Strengths \n1. This paper is well-written and easy to follow.\n2. Numerical experiments and ablation studies are extensive and demonstrate the advantages of the method in fitting the data distribution.\n\nWeaknesses\n1. The proposed method mainly focuses on the clean data setting, while other baselines in the experiments, for example, RNC, have applications in improving generalization and robustness.\n2. $\\mathcal{L}\\_{\\text{diffnorm}}$ is claimed to be a relaxed version of $\\mathcal{L}\\_{\\text{diff}}$. It is unclear why the objective function includes both $\\mathcal{L}\\_{\\text{diffnorm}}$ and $\\mathcal{L}\\_{\\text{diff}}$, rather than just  $\\mathcal{L}\\_{\\text{diffnorm}}$.", "other_comments_or_suggestions": "1. $\\mathcal{L}\\_{\\text {diff }}^{M S E}$ in Theorem 1 is not defined.\n2. $L_{\\mathrm{diff}}$ should be $\\mathcal{L}\\_{\\text {diff }}$  at Line 111.", "questions_for_authors": "See strengths and weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies regression by incorporating label similarity information to mitigate the estimation variance. All reviewers agree on that the paper is well presented and the authors' claims are verified thoroughly from theory and practice. Thus I accept this submission for acceptance.
Decision: Accept (poster)