Variational Imbalanced Regression: Fair Uncertainty Quantification via Probabilistic Smoothing
Authors: Ziyan Wang
Hao Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: probabilistic methods
imbalanced regression
variational inference
TL;DR: We propose a probabilistic deep learning model, dubbed variational imbalanced regression (VIR), which not only performs well in imbalanced regression but naturally produces reasonable uncertainty estimation as a byproduct.

Abstract: Existing regression models tend to fall short in both accuracy and uncertainty estimation when the label distribution is imbalanced. In this paper, we propose a probabilistic deep learning model, dubbed variational imbalanced regression (VIR), which not only performs well in imbalanced regression but naturally produces reasonable uncertainty estimation as a byproduct. 
Different from typical variational autoencoders assuming I.I.D. representations (a data point's representation is not directly affected by other data points), our VIR borrows data with similar regression labels to compute the latent representation's variational distribution; furthermore, different from deterministic regression models producing point estimates, VIR predicts the entire normal-inverse-gamma distributions and modulates the associated conjugate distributions to impose probabilistic reweighting on the imbalanced data, thereby providing better uncertainty estimation. Experiments in several real-world datasets show that our VIR can outperform state-of-the-art imbalanced regression models in terms of both accuracy and uncertainty estimation. Code will soon be available at https://github.com/Wang-ML-Lab/variational-imbalanced-regression.

Variational Imbalanced Regression: Fair Uncertainty Quantification via Probabilistic Smoothing
Ziyan Wang
Hao Wang
Georgia Institute of Technology wzy@gatech. edu
Rutgers University hw4880cs rutgeru..ee... edu
Abstract
Existing regression models tend fall short in both accuracy and uncertainty estimation when the label distribution is imbalanced. this paper, we pro- pose probabilistic deep learning model, dubbed variational imbalanced regres- sion (VIR), which not only performs well in imbalanced regression but natu- rally produces reasonable uncertainty estimation as byproduct. Different from typical variational autoencoders assuming I.I.D. representations (a data point's s representation is not directly affected by other data points), our VIR borrows data with similar regression labels to compute the latent representation' 's vari- ational distribution; furthermore, different from deterministic regression mod- els producing point estimates, VIR predicts the entire pormal-inverse-gammee distributions and modulates the associated conjugate distributions to impose probabilistic reweighting on the imbalanced data, thereby providing better un- certainty estimation. Experiments in several real-world datasets show that our VIR can outperform state-of-the-art imbalanced regression models in terms of both accuracy and uncertainty estimation. Code will soon available at htttps //github. htcchuu..oom//eeee cwm Wang- ML- Lab-Laaaaaaiiiiinnna htw.riioonnnmmmmaamaeemeeeee www.iiaaaaaddddrreeeeeeeee
Introduction
Deep regression models are currently the state of the art in making predictions in continuous label space and have a wide range successful applications in computer vision [50]. natural language processing 22 healthcare [43] 45). recommender systems [16] 42), etc. However, these models fail however when the label distribution training data is imbalanced. For example, in visual age estimation 30 where model infers the age of person given her visual appearance, models typically trained on imbalanced datasets with overwhelmingly more images of younger adults, leading to poor regression accuracy for images of children elderly people [48] [4]. Such unreliability in imbalanced regression settings motivates the need for both improving performance for the minority in the presence imbalanced data and, more importantly, providing reasonable uncertainty estimation to inform practitioners on how reliable the predictions are (especially for the minority where accuracy lower).
Existing methods for deep imbalanced regression (DIR) only focus on improving the accuracy of deep regression models by smoothing the label distribution and reweighting data with different labels [48] 49). On the other hand, methods that provide uncertainty estimation for deep regression models operates under the balance-data assumption and therefore do not work well in the imbalanced setting 1 g 29].
To simultaneously cover these two desiderata, propose probabilistic deep imbalanced regres- sion model, dubbed variational imbalanced regression (VIR). Different from typical variational autoencoders assuming I.I.D representations (a data point's 's representation is not directly affected by other data points), our VIR assumes Neighboring and Identically Distributed (N.LD.) and borrows
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

data with similar regression labels to compute the latent representation's 's variational distribution. Specifically, VIR first encodes a data point into a probabilistic representation and then mix it with neighboring representations (i.e., representations from data with similar regression labels) to produce its final probabilistic representation; VIR is therefore particularly useful for minority data as it can borrow probabilistic representations from data with similar labels (and naturally weigh them using our probabilistic model) to counteract data sparsity. Furthermore, different from deterministic regression models producing point estimates, VIR predicts the entire Normal Inverse Gamma (NIG) distributions and modulates the associated conjugate distributions by the importance weight computed from the smoothed label distribution to impose probabilistic reweighting on the imbalanced data. This allows the negative log likelihood to naturally put more focus on the minority data, thereby balancing the accuracy for data with different regression labels. Our VIR framework is compatible with any deep regression models and can be trained end to end.
We summarize our contributions as below:
We identify the problem of probabilistic deep imbalanced regression as well as two desider- ata, balanced accuracy and uncertainty estimation, for the problem.
We propose VIR simultaneously cover these two desiderata and achieve state-of-the performance compared to existing methods.
As a byproduct, we also provide strong baselines for benchmarking high-quality uncertainty estimation and promising prediction performance on imbalanced datasets.
2 Related Work
Variational Autoencoder. Varia- tional autoencoder (VAE) (25) is
y
VAE
VIR (Ours)
100
an unsupervised learning model that aims to infer probabilistic representa- tions from data. However, as shown in Figure I VAE typically assumes I.I.D. representations, where data point's representation is not directly affected by other data points. In contrast, our VIR borrows data with similar regres- sion labels compute the latent rep- resentation's 's variational distribution.
71
X-
Z
Z
X
33
x
Z
25
Imbalanced Regression. Imbalanced regression is under-explored in the machine learning community. Most existing methods for imbalanced re- gression are direct extensions of the SMOTE algorithm (9]. a commonly used algorithm for imbalanced classification, where data from the minority classes is over-sampled. These algorithms usually synthesize augmented data for the minority regression labels by either
Figure I: Comparing inference networks of typical VAE 25 and our VIR. In VAE (left), a data point's latent representa- tion (i.e., 2) is affected only by itself, while in VIR (right), neighbors participate to modulate the final representation.
interpolating both inputs and labels [4] or adding Gaussian noise [ 6 (more discussion on augmentation-based methods in the Appendix).
Such algorithms fail measure the distance in continuous label space and fall short in handling high-dimensional data (e.g., images and text). Recently, DIR [49 addresses these issues by applying kernel density estimation smooth and reweight data on the continuous label distribution, achieving state-of-the-an performance. However, DIR only focuses on improving the accuracy, especially for the data with minority labels, and therefore does not provide uncertainty estimation, which is crucial to assess the predictions' reliability. [32 focuses on re-balancing the mean squared error (MSE) loss for imbalanced regression, and introduces ranking similarity for improving deep imbalanced regression. In contrast, our VIR provides principled probabilistic approach to simultaneously achieve these two desiderata, not only improving upon DIR in terms of performance but also producing reasonable uncertainty estimation as much-needed byproduct to assess model reliability. There is also related work on imbalanced classification 11. which is related to our work but focusing on classification rather than regression.

---

Uncertainty Estimation in Regression. There has been renewed interest in uncertainty estimation
in the context of deep regression models [2 18
36 38 [T] Most existing methods rely on 51 or using Bayesian neural networks
23 26
directly predict the variance of the output distribution as the estimated uncertainty 1
23 52
post-hoe confidence interval calibration [26 37
47]:
there are also training-fiee free approaches, such as Infer Noise and Infer Dropout (29), which produce multiple predictions from different perturbed neurons and compute their variance as uncertainty estimation. Closest to our work is Deep Evidential Regression (DER) M. which attempts to estimate both aleatoric and epistemic uncertainty [20 [ on regression tasks by training the neural networks to directly infer the parameters of the evidential distribution, thereby producing uncertainty measures. DER [ is designed for the data-rich regime and therefore fails to reasonably estimate the uncertainty if the data is imbalanced; for data with minority labels, DER [I] tends produce unstable distribution parameters, leading to poor uncertainty estimation (as shown Sec. 5) In contrast, our proposed VIR explicitly handles data imbalance in the continuous label space avoid such instability; VIR does so by modulating both the representations and the output conjugate distribution parameters according to the imbalanced label distribution, allowing training/inference proceed if the data is balance and leading to better performance as well as uncertainty estimation (as shown in Sec. 5
3 Method
In this section introduce the notation and problem setting, provide overview of VIR, and then describe details on each of VIR's & key components.
3.1 Notation and Problem Setting
Assuming an imbalanced dataset in continuous space (4 where N is the total num- ber of data points,
E Rd is the input, and yi E y C R is the corresponding la- bel from a continuous label space y. In practice, y is partitioned into equal-interval bins (") ( (1) ,,(")) y()),, with slight notation overload. To directly com- pare with baselines, we use the same grouping index for target value b E B as in [49]. We denote representations as 2,, and use (z"" -,-) = (z)x),,) to denote the probabilis- tic representations for input x; generated by probabilistic encoder parameterized by Φi fur- thermore, we denote as the mean of represen-
z
tation z; in each bin, i.e., = Na E z, for a bin with N₂ data points. Similarly we use (Ti, 8.) denote the mean and variance of the predictive distribution generated by a probabilis- tie predictor po(y)(z).
Z
θ
X
N
Figure 2: Overview of our VIR method. Left: The inference model infers the latent representations given input x's in the neighborhood. Right: The generative model reconstructs the input and pre- dicts the label distribution (including the associated uncertainty) given the latent representation.
3.2 Method Overview
In order to achieve both desiderata in proba- bilistic deep imbalanced regression (i.e., perfor- mance improvement and uncertainty estimation), proposed variational imbalanced regression (VIR) operates on both the encoder (z (x; ) and the predictor ps (ye [z.).
Typical VAE 25
lower-bounds input x, 's marginal likelihood; in contrast, VIR lower-bounds the marginal likelihood of input x, and labels y::
log pe (x y:) Dico as (z, (xx ) (p. (2, v:)) + L(O, di x;, v.).
Note that our variational distribution go (z. J (1) does not condition labels ys. since the task is to predict yi and (2) conditions all (neighboring) inputs (x,).N rather than just x,. The second term L(,, oi xx, yt) is VIR's evidence lower bound (ELBO), which is defined as:
C(,, Φ; x., v.) = By log pe (xx.eeeee l2.] B. log pa (v.l2)) Dice (gs (z, Hx, N (llpo(z))))
(I)
200
3

---

where the (z(z;) is the standard Gaussian prior (O,,)) I), following typical VAE [25], and the expectation is taken over q0 [2, which infers z, by borrowing data with similar regression labels to produce the balanced probabilistic representations, which is beneficial especially for the minority (see Sec. 3.3 for details).
Different from typical regression models which produce only point estimates for yi, our VIR's predictor, pe (y: 2,), directly produces the parameters of the entire NIG distribution for yi and further imposes probabilistic reweighting on the imbalanced data, thereby producing balanced predictive distributions (more details in Sec. 3.4p
3.3 Constructing g(z, (x))
To cover both desiderata, one needs to (1) produce balanced representations to improve performance for the data with minority labels and (2) produce probabilistic representations to naturally obtain reasonable uncertainty estimation for each model prediction. To learn such balanced probabilistic representations, we construct the encoder of our VIR (i.e., go [o(2)((x,] 1) by (1) first encoding a data point into probabilistic representation, (2) computing probabilistic statisties from neighboring representations (i.e., representations from data with similar regression labels), and (3) producing the final representations via probabilistic whitening and recoloring using the obtained statistics.
Intuition on Using Probabilistic Representation. DIR uses deterministic representations, with one vector as the final representation for each data point. In contrast, our VIR uses probabilistic representations, with one vector as the mean of the representation and another vector as the variance of the representation. Such dual representation is more robust to noise and therefore leads to better prediction performance. Therefore, We first encode each data point into a probabilistic representation. Note that this is in contrast to existing work [49] that uses deterministic representations. We assume that each encoding z; is a Gaussian distribution with parameters {z." z J. which are generated from the last layer in the deep neural network.
From I.I.. to Neighboring and Identically Distributed (N.LD.)). Typical VAE 25 is an unsuper- vised learning model that aims to learn a variational representation from latent space to reconstruct the original inputs under the I.I.D. assumption; that is, in VAF, the latent value (i.e., z;) generated from its own input x,. This L.D. assumption works well for data with majority labels, but significantly harms performance for data with minority labels. To address this problem, we replace the I.I.D. as- sumption with the N.L.D. assumption; specifically, VIR's variational latent representations still follow Gaussian distributions (i.e.. ((z, 2 but these distributions will be first calibrated using data with neighboring labels. For a data point (x,, ys) where y is in the b' t'th bin, i.c., yt E (b ())),, we compute ( [ _ (Z, z z with the following four steps.
(1) Mean and Covariance of Initial z,:
2 = I(xi),
(2) Statistics of Bin b's Statistics:
Mh Mb Σ4
A( N
(3) Smoothed Statistics of Bin 6's Statistics:
1h 14 E S
E E/ Σ ( 1),
(4) Mean and Covariance of Final z;:
= Flz! 24 p
- pb
D
14
Σ >
where the details of functions
S(),, and F()) are described below.
(1) Function I(-): From Deterministic to Probabilistic Statistics. Different from deterministic statistics in [9]. our VIR's encoder uses probabilistic statistics, i.c., statistics of statistics. Specifi- cally, VIR treats z, as a distribution with the mean and covariance (z,", 2 = I(x;) rather than a deterministic vector.
As result, all the deterministic statistics for bin b, Mbx Ee, Pb. and Eg are replaced by distributions with the means and covariances, (44) (.)), ( ) (7t ( and (E,,, Σ respectively (more details in the following three paragraphs on A(-). S()). and F(.)).
(2) Function A()): Statistics of the Current Bin I's Statistics. In VIR, the deterministic overall mean for bin & (with Ni data points), 14g = 2 = N, No Zi, becomes the probabilistic overall mean, i... a distribution of ne with the mean μ and covariance Hu (assuming diagonal covariance) as

---

follows:
No No F V N
No
f - 3
Blzi]
t=1 2
No.
Σ V/z,]
1
No
Similarly, the deterministic overall covariance for bin b, Σか = probabilistic overall covariance, i.e., matrix-variate distribution
Ei= (z, z)². becomes the
No.
[15] with the mean:
Σg - No Σ No 1 Bi 2) N, Σ 2 (z
[1 + ([=)))))}
since E|z] 1 and Viz p. Note that the covariance of Σg, i.e., Eg & V/E,), involves computing the fourth-order moments, which is computationally prohibitive. Therefore in practice, we directly set Σ to zero for simplicity; empirically we observe that such simplified treatment already achieves promising performance improvement upon the state of the art. More discussions on the idea of the hierarchical structure of the statistics of statistics for smoothing are in the Appendix.
(3) Function S(-): Neighboring Data and Smoothed Statisties. Next, we can borrow data from neighboring label bins b' to compute the smoothed statistics of the current bin b by applying a sym- metric kernel -(,,,) (e.g., Gaussian, Laplacian, and Triangular kernels). Specifically, the probabilistic smoothed mean and covariance are (assuming diagonal covariance):
16 = Ever k(yb, yo )
-E -Eves k2 yu ) Σ" = k(yb yo )Ey.
14 =
(4) Function F(:): Probabilistic Whitening and Recoloring, We develop probabilistic version of the whitening and re-coloring procedure in [39 49]. Specifically, we produce the final probabilistic representation (z) " for each data point as:
2 = (zl m
+ (2 +
DS
E ub.
(2)
During training, we keep updating the probabilistic overall statistics, {Mi u5. }. and the proba- bilistic smoothed statistics, (71 T.l Σn across different epochs. The probabilistic representation (Z,,,,") are then re-parameterized (25] into the final representation 2,, and passed into the final layer (discussed in Sec. (34) to generate the prediction and uncertainty estimation. Note that the computation of statistics from multiple x's is only needed during training. During testing, VIR directly uses these statistics and therefore does not need re-compute them.
3.4 Constructing p(3 z;)
Our VIR's predictor p(y ( & NN Dc. 3:) predicts both the mean and variance for by first predicting the NIG distribution and then marginalizing out the latent variables. motivated by the following observations on label distribution smoothing (LDS) in 49] and deep evidental regression (DER) in [1 as well intuitions on effective counts in conjugate distributions.
LDS's Limitations in Our Probabilistic Imbalanced Regression Setting, The motivation of LDS [49] that the empirical label distribution can not reflect the real label distribution in an imbalanced dataset with continuous label space; consequently, reweighting methods for imbalanced regression fail due these inaccurate label densities. By applying smoothing kernel on the empirical label distribution, LDS tries to recover the effective label distribution, with which reweighting methods can obtain 'better' weights to improve imbalanced regression. However, our probabilistic imbalanced regression, one needs to consider both (1) prediction accuracy for the data with minority labels and (2) uncertainty estimation for each model. Unfortunately, LDS only focuses on improving the accuracy, especially for the data with minority labels, and therefore does not provide uncertainty estimation, which is crucial to assess the predictions' reliability.
DER's Limitations in Our Probabilistic Imbalanced Regression Setting. In DER 11, the predicted labels with their corresponding uncertainties are represented by the approximate posterior parameters (%, v. a, B3 of the NIG distribution N NIG(y, a. 3). A DER model is trained via minimizing the negative log- likeeihodood (NLL) of Student-t distribution:
C DER
log {) + (a + 1/) log((() >)2v + 22) - a log(!)) + log(
[(a)
(3)
I(at
5

---

where S = 23 It is therefore nontrivial to properly incorporate reweighting mechanism into the NLL. One straightforward approach is directly reweight C for different data points (X,, Us). However, this contradicts the formulation of NIG and often leads to poor performance, as we verify Sec. 5
Intuition of Pseudo-Counts for VIR. To properly incorporate different reweighting methods, our VIR relies on the intuition of pseudo-counts (pseudo-observatioon) in conjugate distributions [4]. Assuming Gaussian likelihood, the conjugate distributions would be an NIG distribution [ i.c., (p,E) Σ) NIG v, a, B), which means:
p~ ~~,,,,
1~1~1(a, 8),
where (a, B)
inverse gamma distribution. With an NIG prior distribution NIG( Mo, Cwu, Bo), the posterior distribution of the NIG after observing n real data points {u,}
are:
O2 = 00 n Bn= 3o + (0) +
7n
Vn 21 n,
Pn
where V = and 1 1C - v.). Here My and an can be interpreted as virtual observations, that contribute to the posterior distribution. Overall, the
ie., pseudo-counts or pseudo-observatiinn mean posterior distribution above can interpreted as an estimation from (2ar + observations, with 20o virtual observations and real observations. Similarly, the variance be interpreted an estimation from (14- + n) observations. This intuition is crucial in developing our VIR's s predictor.
From Sheudo-Counts to Balanced Predictive Distributions. Based on the intuition above, we construct our predictor (i.e., p(ya 2z)) by (1) generating the parameters in the posterior distribution of NIG, (2) computing re-weighted parameters by imposing the importance weights obtained from LDS, and (3) producing the final prediction with corresponding uncertainty estimation.
we feed the final representation {z, ] generated from the Sec. linear layer to output the intermediate parameters ni, V., &; for data point (x,3)))
Based on Eqn. 4
3.3 (Bqn, into
nii Ur 4. = G(zi), 2, ~ alz Kx
NZ 22,
We then apply the importance weights (EvEr k(yb, yb (p(())(()))) - 1/2 calculated from the smoothed label distribution to the pseudo-count to produce the re-weighted parameters of posterior dis- tribution of NIG, where p(v) denotes the marginal distribution of y. Along with the pre-defined prior parameters (7o. Vo, Qo: Po), we are able to compute the parameters of posterior distribution NIG( V4, Oc, 3r) for (x.: M:):
:.- 70 Eveep Me (p 1/2 m. V.
v = vp + Σ
k(yb, yoe yo (((w)) 1/2 -1/2 mi
Vi
VEB
af = ao + (Σ
k(y Br (p(w)))))
n 2
B = 3o + (()2)) +
VEB
Based the NIG posterior distribution, then compute final prediction and uncertainty estimation as
T1 7
We use an objective function similar to Eqn. but with different definitions of (7, 3), to optimize our VIR model:
go (xx [ log( 1og(;))) (a + log((1k 202 + +2) oj log(w]) log( [(a)) (5)
}]
I(a +
where - 2 (1 v²))
Note that L.P part of the ELBBO in Eqn. Similar 1 we use an additional regularization term to achieve better accuracy:
C = + 2x.) 1y 9.1
CP and C. together constitute the objective function for learning the predictor ((y))

---

Table 1: Accuracy on Age AgeDB-DIE DB-DIR.
Table 2: Accuracy on IW-DIR.
Merses
MAE
GM
Menies
MAE 1
GM
Stot
all
mediuu
all
many
mediam
lew
Stus
All
Many
Medium
Few
All
Many
Medium
Few
VANELA
9.55
13.67
10:75
26.33
10.59
VAF
6.38
9.21
13.45
26.30
DEBP Ehs.
22
7.53
6.62
9.37
13.90
4.87
4.37
6.50
11.35
DEEF EXS.
2+
8.05
7.31
15.09
26.47
4.59
4.26
10.61
21.13
INTTR Noise
8.53
7.6
9.33
13.82
5.57
4.95
6.58
10.66
INFER NoIsE
P9
5.11
7.36
15.23
26.29
4.68
4.33
10,65
20.31
00
25.25
SMOGN
8:26
7.64
9:01
12.09
5.36
6.19
8:46
SMOGN
7.30
25.93
SQUav
7.81
7.16
8,50
11.2
4.99
4.57
5.75
1.37
SQlav
7.87
7.34
17.44
22.76
4.47
4.22
7.25
15.10
7.31
8.99
12.06
DER U
7:85
7.18
13.55
24.12
4.47
4.88
K.15
15.18
LDS
7.45
FDS [X
7.69
7.10
8.86
9.98
4.83
4.41
5.77
6.29
RDS
7.33
LDS e FDS AP
7.55
7.01
8.34
10.79
4.72
4.35
5.45
6.79
LDS + FDS
7.78
7.30
12.61
22.
4.37
4.12
7.30
12.11
RANKSIM
RANKSIM
6.93
12.00
21
4.19
13.26
LDDS FDS
8:38
7.44
9.52
7.68
VIR (OURS)
6.99
6.39
7,47
9.51
4.41
4.07
508
6.23
VIR (OURS)
7.19
6,56
11.81
20.96
3.8
343
6.51
12:23
OuaS vs. VANILLA
+0.78
+0.25
+2.08
+4.16
10.64
+0.16
+1.96
+4.52
OuBs INEEK Noise
-1.54 +1.25
+2.26
+4.31
-1.16
+0.85
+1.53
+463
Ouns vs. IsPER Noist
+0.93 +0.50
+3.43
+5.33
+8.83
+0.70
+4.14
+8.08
63.15
Ouns vs. DER
+8.66
+0.62
+1.54
+3.16
+8.61
+0.55
+1.07
+2.95
OuRS L.S
+0.37
40.31
+0.56
FDS
Ours vs. RANKSIM
-4.83 40.10
+0.37
+0.37
40.12
40.06
+0.32
41.66
Ouns RANKSIM
+0.72
+6.34
+0.34
3.5 Final Objective Function
Putting together Sec.
3.3 and Sec.
3.4 our final objective function (to minimize) for VIR is:
VIR Σ
VIR
VIR
AC
C(0, Φ; x., = AC
KC
where L(O, 6; x, 14) = C? + CP - c is the ELBO in Eqn. adjusts the importance of the additional regularizer and the ELBO, and thus lead to better result both accuracy and uncertainty estimation.
Theory
Notation. As mentioned in Sec. 3.1 we partitioned {y,}/=1 into [B] equal-interval bins (denote the set of bins as B), and {X,} are sampled from the label space V. In addition, We use the binary
[8] to represent the label distribution (frequency) for each bin i, i.c., P. & P(Y, E B.). We also use the binary set [ to represent whether the data point (x,, is observed (i.e., O, ~ PP = 1) x PB(Y)), and [O,] ~ PB(Y))) where B(Y;) represents the bin which (æ,, Mj) belongs to. For each bin i B, we denote the associated set of data points as
set {P:)
u L Y, i}.
When the imbalanced dataset is partially observed, we denote the observation set as:
S - (j O, 1 & B(Y;)= =)-
Definition 4.1 (Expectation over Observability Eo). We define the expectation over the observabil- ity variable O as Eol:] = E0,--(,,=1)11 Definition 4.2 (True Risk). Based on the previous definitions, the true risk defined as:
B
R(Y)
Σ
,,
B
jEU:
where ,y refers to some loss function (e.g. MAE, MSE). In this paper, assume the loss is
upper bounded by A, ie.. 0 S 5Y A.
Below we define the Naive Estimator. Definition 4.3 (Naive Estimator). Given the observation set, the Naive Estimator is defined as:
[5]
RYAIVE(E)
8),
S
1=
i=1 JES,

---

It is easy to verify that the expectation of this naive estimator is not equal to the true risk, as Bol (RNAIIE (Y) + R(Y).
Considering an imbalanced dataset subset of observations from balanced one, we contrast with the Inverse Propensity Score (IPS) estimator [34].
Definition 4.4 (Inverse Propensity Score Estimator). The inverse propensity score (IPS) estimator (an unbiased estimator) is defined
5,
RIPS ( P)
B
U
P.
JES,
The IPS estimator is an unbiased estimator, as we can verify by taking the expectation value over the observation set:
1
B
YY
Io Rips (
P))
[O[
B
U
P.
JEUL
B
Σ
,,
R(Y).
B
U
,EUf
Finally, we define our VIR/DIR estimator below.
Definition 4.5 (VIR Estimator). The VIR estimator, denoted by RIVIR (P)P), is defined as:
,Y
RVIR (PIP)
(6)
B
U4
P.
JES.
where {P:),E) represents the smoothed label distribution used in our VIR's objective function., It is important to note that our VIR estimator biased.
For multiple predictions, we select the "best" estimator according to the following definition.
Definition 4.6 (Empirical Risk Minimizer). For given hypothesis space of predictions Y, the Empirical Risk Minimization (ERM) identifies the prediction Y H as
FERM argming EH [BVIIR (8)P))
With all the aforementioned definitions, we can derive the generalization bound for the VIR estimator. Theorem 4.1 (Generalization Bound VIR). In imbalanced regression with bins B, for any finite hypothesis space of predictions H = (VI....,Yx) the transductive prediction ervor of the empirical risk minimizer VERM using the VIR estimator with estimated propensities P (P: >> 0) and given training observations O from y with independent Bernoulli propensities P, bounded by:
B
P P.
log (((ll)))))
18 Σ i=1
R(ÝERM) - RVVIR ( ERM P)
B
2
P2
Bias Term
Variance Term
has large bias and large variance. If one directly uses the original label distribution in the training objective (i.e., Definition 4.4p. i.e., P P., the "bias" term will O. However, the "variance" term will be extremely large for minority data because P; is very close to O. contrast, under VIR's N.L..,, P; used in the training objective function will be smoothed. Therefore, the minority data's label density P; will be smoothed out by its neighbors and becomes larger (compared the original P,). leading smaller "variance" in the generalization error bound. Note that P; # P.. VIR (with N.I.D.) essentially increases bias, but significantly reduces its variance in the imbalanced setting, thereby leading to lower generalization emror.
Remark.
The naive estimator (i.c., Definition 4.3

---

5 Experiments
Datasets. We evaluate our methods in terms of prediction accuracy and uncertainty estimation on four imbalanced dataset AgeDB-DIR [30]. IMDB-WIKI-DIR [33], STS-B-DIR 77.. and NYUD2- DIR 335 Due to page limit, results for NYUD2-DIR (35 are moved to the Appendix. We follow the preprocessing procedures in DIR 49).. Details for each datasets are in the Appendix, and please refer
to [49 for details on label density distributions and levels of imbalance.
Baselines. We use ResNet-50 [7 (for AgeDB-Dil DIR and IMDB- WIII-DDRR and BiLSTM [19] (for STS-B-DIR) as our backbone networks, and more details for baseline are in the Appendix. we describe the baselines below.
Vanilla: We use the term VANILLA to denote a plain model without adding any approaches. Synthetic-Smmple Baaed Methods:
Various existing imbalanced regression methods are also included as baselines; these include Deep Ensemble (27). Infer Noise 229 ..STTTRR (40) and SMOGN [5].
Cost-Sensitive Reweighting: As shown in DIR 49 the square-root weighting variant
-1/2 always outperforms Vanilla. There- fore, for fair comparison, all our experiments (for both baselines and VIR) use SQINV
(SQINV) baseline (i.e., (Eres k(bb, Yo )p(w))
weighting.
Evaluation Metrics. We follow the evaluation metrics in 49 to evaluate the accuracy of our proposed methods; these include Mean Absolute Error (MAE), Mean Squared Error (MSE). Furthermore, for AgeDB-DIR and IMDB-WIKI-DIE WIKI-DIR, we use Geometric Mean (GM) to evaluate the accuracy; for STS-B-DIR, we use Pearson correlation and Spearman correlation. We use typical evaluation metrics for uncertainty estimation in regression problems to evaluate our produced uncertainty estimation; these include Negative Log Likelihood (NLL), Area Under Sparsification Error (AUSE). Eqn. [] shows the formula for NLL, and more details regarding AUSE can be found in 21 include calibrated uncertainty results for VIR in the Appendix.
We also
Evaluation Process. Following 28
Table 3: Accuracy STS-B-DIR.
49), for a data sample I; with its label y which falls into the target bins b, we divide the label space into three disjoint subsets: many-shot region {br E B ys E by & |yal > 100}. medium-shot region {b; B yi E b, & 20 < |ud < 100), and few- shot region (b: E B y/ E b, & [yil < 20). where
Metrics
Pearson
Shot
Many
Medium
All
Many Madiom
VANILLA
0.974
0.851
VAF X5
0.968
0.833
1.511
1.102
0.751
0.724
0.621
0.749
0.846
0.746
INFER
NoiST
20
0.954
0.980
1.408
0.967
0.747
0.711
0.631
0.756
SMOTDR
1.466
0.924
0.726
SMOGN
0.900
0.396
1.327
1.175
0,752
0.704
0.655
0.092
INV
1.005
0.891
1.482
1,066
0.728
0,703
0.625
0.732
denotes the cardinality of the set. We report re- sults on the overall test set and these subsets with the accuracy metrics and uncertainty metrics discussed above.
DER
11
1.001
0.912
1.368
1.055
0.752
0.711
0.646
0.742
LDS
0.14
0.819
1,319
0.955
0.756
0.734
0.638
0.762
FDS
4M]
0.977
0.851
1.225
1,012
0.750
0.724
0.667
0.742
1.DS FDS AX
0.907
0.802
1,363
0.942
0.760
0,740
0.622
0.766
RANKSIM
L3
0.903
0.908
0.911
0.804
0.758
LDS FDS + DER 0
1.007
0.880
1.535
1.086
0.729
0.714
0.635
0.731
VIR (OURS)
0.892
0.781
0,776
Implementation Details. We con- ducted five separate trials for our method using different random seeds. The error bars and other implementa- tion details are included in the Appendix.
OURS vs. VANILLA
+0,882
40,056
40.621
+0.203
+0.034
+0.032
.0.069
+0,093
OuRS vs. INrE Noish
+0.862
40.185
+11.509
40.186
+0,029
+0.041
+0.065
+0.089
OURS vs. DER
+0.117
+0.274
+0.044
+0.041
+0.050
OURS Vs. LDS + FDS
+0.015
+0.007
+0.464
+0.16]
+0,016
+0.012
+0.04
+0.079
OURS VS. RANKSIM
+0,011
+0.113
+0,012
+0.023
+0.018
+0.006
+0.018
5.1 Imbalanced Regression Accuracy
We report the accuracy of different methods in Table Table and Table 3 for AgeDB-DIR, IMDB- WIKI-DIR and -TS-B-DIR respectively. In all the tables, we can observe that our VIR consistently
outperforms all baselines in all metrics.
As shown in the last four rows of all three tables, our proposed VIR compares favorably against strong baselines including DIR variants [49] and DER D. Infer Noise (29), and Rankšim [ especially on the imbalanced data samples (i.c., in the few-shot columns). Notably, VIR improves
Amoog the five datasets proposed in 49 only four of them publicly available.

---

Table 4: Uncertainty estimation on AgeDB-DIR.
Table 5: Uncertainty estimation on IW-DIR.
Masic
XLL.
MUSE 1
Marica
NLL
Shat
Meag
Medl
Fee
A1
Mary
Midl
Shor
All
Mary
Medior
FrM
AlI
Mars
Midterr
Fixe
Deer Eos. 15
5311
4.31
6.756
8553
0.541
06555
0.005
8.885
DEED EXR
27
5.199
4,102
1.125
8852
6:94
01952
01343
0771
NNrk Noms
4.15
5862
11455
0A49
0.457
Fooe Noteo
1.511
1.778
5:336
1.500
0777
0.714
[1561]
D/77
DER (
8918
3.341
0.533
0464
0.49)
BIK 1
1890
1,09
1967
6.6%
0311
01827
(669)
1.51
LDS
0451
0469
LDS FDS DER
1383
1,002
4.591
5900
0.644
0550
0449)
B483
VIR (O0a5)
3783
4.196
0.44
0456
VIR (00R5))
3661
3,59
4.266
5.88
8634
0,64
0.34
0,79
OURS V5. DER
18.255
(8.143
10.14
10.236
10,859
18,008
10.135
100073
Ouess vs. BUB
+0.10
+8.120
+0.701
+1.330
+0.179
+6115
+0.216
+9.16
OuRs v5. LD6 FDS DER
-0,854
+8,09
10.107
10.36
+80044
10,005
+0.151
Ouas vs. LBS + 113 + DER
(4883
+8005
+0.065
+0.379
-4150
+BUC1
+0105
+8.104
upon the state-of-the-ar art method RankSim by 9.6% and 7.9% on AgeDB DIR and IMDB-WII DIR, respectively, in terms of few-shot GM. This verifies the effectiveness of our methods in terms of overall performance. More accuracy results on different metrics are included in the Appendix. Besides the main results, also include ablation studies for VIR in the Appendix, showing the effectiveness of VIR's 's encoder and predictor.
5.2 Imbalanced Regression Uncertainty Estimation
Different from DIR 49 which only focuses on accuracy, we create a new benchmark for uncertainty estimation in imbalanced regression. Table 4 Table and Table ( show the results uncertainty estimation for three datasets Age -IR, IMDB- WIKI-DIR, and And-SS-B -TS-B- respectively. Note that most baselines from Table [ Table 2 and Table are deterministic methods (as opposed to probabilistic methods like ours) and cannot provide uncertainty estimation; therefore they are not applicable here. To show the superiority of our VIR model, create strongest baseline by concatenating the DIR variants (LDS + FDS) with the DER [I.
Results show that our VIR consistently outperforms all baselines across different metrics, especially in the few-shot metrics. Note that our proposed methods mainly focus on the imbalanced setting, and therefore naturally places more emphasis the few-shot metrics. Notably, on AgeDB-DIR, IMDB- WIKI-DIR, and "TS-B-DIR our VIR improves upon the strongest baselines, by 14.2% ~ 17.1% in terms of few-shot AUSE.
5.3 Limitations
Although methods successfully improve both accuracy and uncer- tainty estimation on imbalanced re- gression, there are still several limi- tations. Exactly computing variance of the variances in Sec. 3.3 is chal- lenging; we therefore resort fixed variance as an approximation. Devel- oping more accurate and efficient ap- proximations would also be interesting future work.
Table 6: Uncertainty estimation on STS-B-DIR.
Metric
NLL
AUSE
Shoc
All
Many
Medium
Few
All
Mazy
Mediun
Few
Deme ENS.
3,913
3,911
4.223
4.106
0.676
0.663
INTER Naise
20
3.748
3.753
3.755
3,688
0.673
0.631
0.644
0.639
DER 0
2.657
2.601
3/013
2.401
0.681
0.585
0.613
0.694
LDS
2,561
2.514
2,580
0.609
0.615
VIR (OURS)
1.9%6
1.510
2.754
2.122
0.591
0.575
0.602
0.510
OURS DER OuRs vs. LDS LDS+ DER
[+8,71
+0.791
+0.359
+0.249
+0.091
-0.008
+0,011
+0.114
+0.565
+0,704
+0,126
+0.206
+0.081
+0,006
+0.007
+0,105
Conclusion
We identify the problem of probabilistic deep imbalanced regression, which aims to both improve accuracy and obtain reasonable uncertainty estimation in imbalanced regression. We propose VIR, which can use any deep regression models backbone networks. VIR borrows data with similar regression labels to produce the probabilistic representations and modulates the conjugate distributions to impose probabilistic reweighting on imbalanced data. Furthermore, we create new benchmarks with strong baselines for uncertainty estimation on imbalanced regression. Experiments show that our methods outperform tfate-of-the- art imbalanced regression models in terms of both accuracy and uncertainty estimation. Future work may include (1) improving VIR by better approximating variance of the variances in probability distributions, and (2) developing novel approaches that can achieve stable performance even on imbalanced data with limited sample size, and (3) exploring techniques such as mixture density networks to enable multi-modality in the latent distribution, thereby further improving the performance.
10

---

Acknowledgement
The authors thank Pei Wu for help with figures, the reviewers/AC for the constructive comments to improve the paper, and Amazon Web Service for providing cloud computing credit. Most work is done when ZW is master student UMich. HW partially supported by NSF Grant IIS-2127918 and an Amazon Faculty Research Award. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.
References
[1] A. Amini, W. Schwarting, A. Soleimany, and D. Rus. Deep evidential regression. Advances in Neural Information Processing Systems, 33:14927-14937, 2020.
[2] Bergstra and Y. Bengio. Random search for hyper-parameee optimization. Journal of machine learning research, 13(2), 2012.
[3] C. M. Bishop. Mixture density networks. 1994.
[4] C. M. Bishop.
Pattern recognition and machine learning. springer, 2006.
[5] Branco, L. Torgo, and R. P. Ribeiro. Smogn: pre-processing approach for imbalanced regression. In First international workshop on learning with imbalanced domains: Theory and applications, pages 36-50. PMLR, 2017.
[6] Branco, L. Torgo, and R. Ribeiro. Rebagg: Resampled bagging for imbalanced regres- sion. In Second International Workshop on Learning with Imbalanced Domains: Theory and Applications, pages 67-81 PMLR, 2018.
[7] D. Cer, Diab, Agirre, I. Lopez- Gazpio, and L.. Specia. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation, pages 1-14, 2017.
[8] B. Charpentier, O. Borchert, D. Zügner, Geisler, and Günnemann. Natural posterior network: Deep bayesian predictive uncertainty for exponential family distributions. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[9] N. V. Chawla, K. W. Bowyer, L.. O. Hall, and W. P. Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
[10] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1607- PMLR, 2020.
[11] Z. Deng, H. Liu, Y. Wang, C. Wang. Z. Yu, and X. Sun. PML: progressive margin loss for long-tailed age classification. IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2021, virtual, June 19-25, 2021, 2021.
[12] Y. Gal and Z. Ghahramani. Dropout bayesian approximation: Representing model un- certainty in deep learning. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1050-1059. JMLR.rrg, 2016.
[13] Y. Gong, G. Mori, and F. Tung, Ranksim: Ranking similarity regularization for deep imbalanced regression. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 7634-7649. PMLR, 2022.
[14] C. Guo, Pleiss, International conference machine learning, pages 1321-1330. PMLR, 2017.
Sun, and K. Q. Weinberger. calibration of modern neural networks, In
[15] A. K. Gupta and D. K. Nagar. Matrix variate distributions, volume 104. CRC Press, 2018.

---

[16] S. Gupta. H. Wang, Z. Lipton, and Y. Wang. Correcting exposure bias for link recommendation. In ICML, 2021.
[17] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition, In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2016.
[18] Heiss, J. Weissteiner, H. Wutte, S. Seuken, and J. Teichmann. Nomu: Neural optimization- based model uncertainty. arXiv preprint arXiv:2102. 2021.
[19] Z. Huang, W. Xu, and K. Yu. Bidirectional LSTM-CRE models for sequence tagging. CoRR, 2015.
[20] E. Hüllermeier and W. Waegeman. Aleatoric and epistemic uncertainty in machine learning: tutorial introduction. CoRR, abs/1910.09457, 2019.
[21] E. Ilg. Ö. eCiecek, S. Galesso, A. Klein, O. Makansi, F. Hutter, and T. Brox. Uncertainty estimates and multi-hypothesee networks for optical flow. In Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VII. volume 11211 of Lecture Notes in Computer Science, pages 677-693. Springer, 2018.
[22] H. Jiang, He, W. Chen, X. Liu, J. Gao, and Zhao. SMART: robust and efficient fine- tuning for pre-trained natural language models through principled regularized optimization. In D. Jurafsky, J. Chai, N. Schluter, and R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pages 2177-2190. Association for Computational Linguistics, 2020.
[23] A. Kendall and Y. Gal. What uncertainties do we need in bayesian deep learning for computer vision? arXiv preprint arXiv: 04977, 2017.
[24] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[25] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[26] V. Kuleshov, N. Fenner, and S. Ermon. Accurate uncertainties for deep learning using calibrated regression. In International Conference on Machine Learning, pages 2796-2804. PMLR, 2018.
[27] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6402-6413, 2017.
[28] Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu. Large-scale long-tailed recognition in an open world. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 2537-2546. Computer Vision Foundation IEEE, 2019.
[29] L. Mi, H. Wang, Y. Tian, and N. Shavit. Training-free uncertainty estimation for neural networks. In AAAI, 2022.
[30] S. Moschoglou, A. Papaioannou, C. Sagonas, J. Deng, I. Kotsia, and S. Zafeiriou. Agedb: The first manually collected, in-the-wild age database. In 2017 IEEE Conference on Computer Vision and Puttern Recognition Workshops, CVPR Workshops 2017, Honolulu, HI, USA, July 21-26, 2017, pages 1997-2005, 2017.
[31] J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.
12

---

[32] J. Ren, M. Zhang, C. Yu, and Z. Liu. Balanced mse for imbalanced visual regression. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA,, USA, June 18-24, 2022, pages 7916-7925. IEFF, 2022.
[33] R. Rothe, R. Timofte, and L. V. Gool. Deep expectation of real and apparent age from single image without facial landmarks. Int. J. Comput. Vis., 126(2-4)) 144 157, 2018.
[34] Schnabel, A. Swaminathan, A. Singh, N. Chandak, and Joachims. Recommendations as treatments: Debiasing learning and evaluation. In international conference on machine learning, pages 1670-1679. PMLR, 2016.
[35] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor segmentation and support inference from rgbd images. ECCV (5), 7576:746-760, 2012.
[36] J. Snoek, Y. Ovadia, E. Fertig, B. Lakshminarayanan, S. Nowozin, D. Sculley, J. V. Dillon, J. Ren, and Z. Nado. Can you trust your model'ss 's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 1 13969-13880, 2019.
[37] H. Song, T. Diethe, M. Kull, and Flach. Distribution calibration for regression. In International Conference Machine Learning, pages 5897-5906. PMLR, 2019.
[38] M. Stadler, B. Charpentier, S. Geisler, D. Zügner, and S. Günnemann. Graph posterior network: Bayesian predictive uncertainty for node classification. In Advances Neural Information Processing Systems 34: Anmual Conference Neural Information Processing Systems 2021, NeurlPS 2021, December 6- 14, 2021, virtual, pages 18033-18808 18048, 2021.
[39] B. Sun, Feng, and K. Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA, pages 2058-2065. AAAI Press, 2016.
[40] L. Torgo, R. Ribeiro, B. Pfahringer, and P. Branco. SMOTE for regression. In L. Correia, L, P. Reis, and J. Cascalho, editors, Progress in Artificial Intelligence 16th Portuguese Conference on Artificial Intelligence, EPIA 2013, Angra do Heroismo, Azores, Portugal, September 9- 12. 2013. Proceedings, volume 8154 of Lecture Notes in Computer Science, pages 378-389. Springer, 2013.
[41] A. Wang, A. Singh, J. Michael, F, Hill, O. Levy, and R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. EMNLP 2018, page 353, 2018.
[42] H. Wang, Y. Ma, H. Ding, and Y. Wang. Context uncertainty contextual bandits with applications to recommender systems. In AAAI, 2022.
[43] H. Wang, C. Mao, H. He, M. Zhao, T. S. Jaakkola, and D. Katabi. Bidirectional inference networks: class of deep bayesian networks for health profiling. In AAAI, volume 33, pages 766-773, 2019.
[44] H. Wang, S. Xingjian, and D.-Y. Yeung. Natural- Natural-parameter networks: A class of probabilistic neural networks. In NIPS, pages 118-126, 2016.
[45] H. Wang and J. Yan. Self-interpretable time series prediction with counterfactual explanations. In ICML, 2023.
[46] H. Wang and D.-Y. Yeung. Towards bayesian deep learning: A framework and some existing methods. TDKE, 28( 28(12):33553400 2016.
[47] H. Wang and D.-Y. Yeung. A survey on bayesian deep learning. CSUR, 53(5):1-37, 2020.
[48] Y. Yang. H. Wang, and D. Katabi. On multi-domain long-tailed recognition, generalization and beyond. In ECCV, 2022.
[49] Y. Yang, K. Zha, Chen, Wang, and D. Katabi. Delving into deep imbalanced regression.
In International Conference on Machine Learning, pages 11842-11851. PMLR, 2021.
13

---

[50] W. Yin, J. Zhang, O. Wang. S. Niklaus, L. Mai, S. Chen, and C. Shen. Learning to recover 3d scene shape from a single image. In IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2021, virtual, June 19-25, 2021, pages 204-21222 213. Computer Vision Foundation / IEEE,
[51] E. Zelikman, C. Healy, S. Zhou, and A. Avati. Crude: calibrating regression uncertainty distributions empirically. arXiv preprint a7X44:2005.12496, 2020.
[52] Z. Zhang, A. Romero, M. J. Muckley, P. Vincent, L. Yang, and M. Drozdzal. Reducing uncertainty in undersampled mri reconstruction with active acquisition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2049-2058, 2019.
14

---

A Theory
Notation. As defined in Sec.3 in main paper, we partitioned (yg) into 1B equal-interval bins (denote the set of bins as B), and {Y;)ý= are sampled from the label space y. In addition, We denote the binary set {P.) as the label distribution (frequency) for each bin, i.., - P(Y, E B.). We also denote the binary set { represent whether the data (x,, My are observed (i.e.. O2 PO(; = 1) x PB()) and [o] ~ PB(Y))), where B(Y;) represents the bin which (zj, Uj) belongs to. For each bin i E B, we denote the global set of samples as
U; {j Y,, =i].
When the imbalanced dataset is partially observed, we denote the observation set as:
S A 0 = & B(Y;)= i).
Definition A.1 (Expectation over Observability Eo)) We define the expectation over the observ- ability variable O as [o]l-] = =0,,)(
Definition A.2 (True Risk). Based on the previous definitions, the true for our model defined as:
ROY)
8)
B
jEU:
where (Y,Y refers to some loss function (e.g. MAE, MSE). In this paper we assume these loss is
upper bounded by A. i.e. 0 S (Y, (Y, S S.A.
Then in next step we define the Naive Estimator. Definition A.3 (Naive Estimator). Given the observation set, the Naive Estimator is defined as:
RNAIVE
Σ 8,0,8
ΣE 8
jES.
is easy to verify that the expectation of this naive estimator is not equal to the true risk, as Eol (NAIVE(())) + R(Y).
Considering an imbalanced dataset as a subset of observations from balanced one, we contrast it with the Inverse Propensity Score (IPS) estimator (34), underscoring the superiorities of our approach. Definition A.4 (Inverse Propensity Score Estimator). The inverse propensity score (IPS) estimator (an unbiased estimator) is defined
18
,Y,
RIPS P
B
44
P.
JES,
IPS estimator is an unbiased estimator, as we can verify by taking the expectation value over the observation set:
B
,Y
Eo Rirs P)]
B
UA
P.
JEUL
B
8,(Y,
R(Y).
B
U
JEu,
Finally, we define our VIR/DIR estimator below.
Definition A.5 (VIR Estimator). The VIR estimator, denoted by REVIR ( is defined as:
1
18
,,
8
RVTRR (8I))
L
P.
JES:
where (P represents the smoothed label distribution utilized in our VIR's objective fancettioe (see Eqn.5 in the main paper). It's important to note that VIR estimator is biased.
B1
15

---

For multiple predictions, we select the "best" estimator according to the following definition.
Definition A.6 (Empirical Risk Mininizer). For given hypothesis space H of predictions V, the Empirical Risk Minimization (ERM) identifies the prediction Y H
PERM argming ER [VVIR (81P)}
Lemma A.1 (Tail Bound for VIR Estimator). For any given Y and Y, with probability 7) the VIR estimator ( (P) does not deviate from its expectation 18 [RVIR (Y EYER [)] by more than
[8|
RVIR (Y ERRRRM - 1o [RVIR ( ERM [P]
(ERMM
A B
log 22
Σ
1
2
P2
Proof. For independent bounded random variables XI, Xn that takes values in intervals of sizes Pi, Pn. with probability 1, and for any >> O,
[ - Σ
n
-262
X;
26
exx
Σx P
Consider the error term for each bin i in RVIR (oee(---t)) FERM P)) as X₃. Using Hoeffding's inequality, define
P(X; 5,(Y,V) P. and P(X; 0) 1 P.. Then, by setting eD [B] e, are then able
P.
show that:
P
[B] Revix ( ( ERM (P) 13-1 |B] TI.-Eoll (RVIR EREM [}] 2 c0
S 2exp
2c
A 2i=1
18
2jet,,
[
p2
= P RvIR =P(TTvR
-2c |
(EER (P) Eo[RViR ( (RER [P]] 26
|P]]
s 2exp
A \3>13 1=
I
P2
We can then solve for e, completing the proof.
With all the aforementioned definitions, we can derive the generalization bound for the VIR estimator. Theorem A.1 (Generalization Bound of VIR). In imbalanced regression with bins B. for any finite hypothesis space of predictions H. = Yi.,,..,,,, the transductive prediction error of the empirical risk minimizer y EERM using the VIR estimator with estimated propensities P (P, >> 0) and given training observations O from y with independent Bernoulli propensities P, is bounded by:
18
[88
R(SERM) S ( ERM P)
P
A
log (2 /m)
B
B
2
P2
Proof. We first re-state the generalization bound for our VIR estimator: with probability have
77. we
[8]
R(Y-RRM) S (
(P)
P
A
log (2
B
B
2
P
We start to prove it from the LHS: R(P-RRM
RRPERM Eo EoIRRvR FRM [P]] BoliBvR (ARiIIRRRRRRRR FERM [P)
bias( RVIR CAAURVERRA EER P)) + Eoo RAVIR EREM [P)] RAVIR
( BRM (P) + RRIR (VVI EERR (P)
Bias Term
Variance Term
bias RVIR FRR P))
RVIR (y ERM (P) EolRvIR FRM
ERM [P]
+ RVIR ( &ERMIP)
Bios Term
Variance Term
16

---

Below we derive each term:
Variance Term. With probability 1/ the variance term is derived as
RVIR (FERM P) Eol REoRRII( ( BRM P1] Ke
2 21-7
P
max 8,
RVIR (P)P) Eo (
Se
2 1 -
RVIR (Pln)P)))))))))))))) (
Bo [RVIR ( ||]]]
2e <1
P
U RVIR ( LEV 26 <1
Hl
PP (1 [RViR (91P) - (oolvvi (P|p)) 26 <1
Σ
(10)
12)2
[H| 2 2exp(
<n
(11)
A
>
-88 -iml
EjEt,
[24
P?
22 |3]
|7| expp(
13 1
< 17
A i=1
P
where inequality (10) is by Boole's inequality (Union bound), and inequality 11 holds by
Lemma A.I Then, by solving for e, we can derive Variance Term that with probability 77
Eo RRVRR (FERM P}] RVIR (y FERM PP) <
RVIR ( P)
EolRVIR ( FRM P}]
log 22/ /m)
B
2
p2
Bias Term. By definition, we can derive:
bias( RVIRR P (yERMM P)) R(PERM, Eo RRIR PERM |ERMIP))
[P]]
B
18
1
Σ
,, PERM)
Σ
P.
(, pERM)
5,
B
[4,1
B
U4
P
jEMi
jeUi
B
1
P
B
U
P
jEM.
A
P
B
P
concluding the proof for the Bias Term, hence completing the proof for the whole generalization bound.
B Details for Experiments
Datasets. In this work, we evaluate our methods in terms of prediction accuracy and uncertainty estimation on four imbalanced dataset: AgeDB (30], MDB-WIKI (33], STS-B [7]. and NYUD2- DIR (35] Due to page limit, the results for NYUD2-DIR 35 are the supplementary. We follow
Among the five datasets proposed in 49 only four of them publicly available.
17

---

Table 7: Accuracy on AgeDB-DIR. For baselines, we directly the reported performance in their paper and therefore do not have error bars.
J
Many
Medium
Many
Fex
Many
Medium
VANILLA
101.00
75,40
138.52
253.74
1.77
6.62
9.55
13.67
5.05
4.23
7,01
10.75
223.09
7.63
6.58
13.45
4.31
10.24
DERP ENSEMBBE 101
100.94
79.3
129.55
249.11 18
7.23
6.62
9.37
13.90
4.87
4.37
6.50
11.35
INPER Nois:
A9
119.45
95,02
149.34
266.29
8,53
7.62
9.75
13:82
5.57
4,95
6.58
10.96
SMOTER
114.34
93.35
7.39
12:28
4.65
8.49
SMOGN
117.29
101, 36
90
8.26
7.64
9.1
12.09
5.16
4.9
6.19
8.44
SQQlvv 59
105.14
87.21
127.66
212.30
7.81
7.15
8:30
11.2
4.99
4,57
5,73
7,77
DER 0
106.77
209.69
8.09
12.66
4.59
10.49
101.22
R3.62
178.
64
7.67
6.98
8.6
10,89
4.99
7,45
FOS 20
101,67
86.09
129.61
167.75
7.69
7.10
8.86
9.98
4.83
4.41
3.97
6.29
LDS FDS !
99.46
$4.10
10.79
4.36
6.79
RANKSIM
13]
83.51
71.99
99.14
140.05
7.02
6.49
9.68
4.33
6.89
LDS TDS + DEK 1
112.62
14,21
140.08
210.72
8.18
7.44
9.52
11:45
5:30
4,75
6,74
7.65
(OURS)
$1.76:5.10
70.61:918
91.47s130
142.36:330
6.99
6.39.2222
7.47:804
9,51::000
4.41:00
4.07.x022
50059925
6:23--03
OURS vs. VANILLA
+19.84
47.79
+47.05
+111.38
+0.78
+0.23
+2.08
+4.16
+1.64
40.16
+1.96
+4,52
OURS
+1.23
OURS vs. DER
+25.01
120.68
+30.96
+67.33
+1.10
+0.92
+1.52
+3.15
+0.78
+0.52
+1.38
+4.26
OURS vs. LDS + FDS
+17.70
+13.49
+20.73
+66.91
+0.56
+0.62
+0.77
41.28
+0.31
40.29
+0.40
+0.56
RANKSIM
Table 8: Accuracy
MMDB-WIKI-BE For baselines, directly use the reported performance in their paper and therefore do not have error bars.
MAIL
1
Manry
Medliam
Many
Medium
Many
Mialian
VANILLA (40
158.06
108.77
566.09
964.92
8:06
7.23
15.11
26.33
4.57
4.17
10.39
20:46
VAE
137.98
103.62
361.74
964.87
8.04
7.20
15.05
36.30
4.57
4.22
10.56
20.72
103.81
10.61
INPER Nosso
143.2
112.26
373.19
961.97
7.36
15.23
36.20
4.33
10.65
20.31
SMOTER
13875
111.55
$46.09
935.99
H.14
7,42
14.15
25:28
4.64
4.30
9:05
19.46
SMOGIN
1
136.09
102.15
$19.09
944.20
808
2.30
14.02
25.09
4.63
4.30
8.74
20.12
SQtNv
30
111.23
7.34
7.25
15.10
DEE 0
13381
107.51
332.90
916.18
7.85
2.18
13.35
24.12
4.47
4.18
8.18
15.18
LnS 50
131.65
102.04
298.98
870.55 35
7.83
7.31
12.41
22.51
4.42
4.19
3.00
19.64
FDS +2
132.64
102.28
311.35
851.06
7.83
7.23
12.60
22.37
4.42
4.30
6.93
13.48
PDS
106.8
7.20
RANKSIM
125.30
102.68
777.48
7.50
693
12.09
21 68
1.97
6.63
13:28
LDS + FDS + DER L
120.56
97.75
207.64
873.10
7.24
6.64
11.87
23,44
3.98
3.60
6.64
16.00
VIR (OURS)
118942115
96.10390)
295.79:1
171.4723
7,19
6.56:200
11,81
20.66006
3.8g00
34350m
6,51gmm
12.281c00
OURS vs. VANHLA
+19.12
+12.6
+70.3
+193.45
+0,87
+0.67
63.31
+5.37
-0.72
-0.54
+4.08
-8.23
OURS VS. INTER Nossn
43468
+16.16
477.40
+190.50
+0.92
+0.30
43.42
45.33
+0.53
+0.70
+4.14
+3.66
OURS vs. FDS
+18.41
+10.42
+15.7
+40.35
+0.39
+0.64
+08
+1.33
+0.49
-0.30
OURS Vs. RANKSIM
+6.36
+6.58
+3.31
+6.01
+0.31
+0.37
40.28
+0.72
+0.34
+0.34
40.14
+1.66
the preprocessing procedures in DIR [49] Details for each datasets are to the Supplement, and details for label density distributions and levels of imbalance are discussed in DIR (4)
AgeDB-DIR. We use AgeDB-DIR constructed in DIR 49 which contains 12.2K images
for training and 2. images for validation and testing. The maximum age in this dataset is 101 and the minimum age is 0, and the number of images per bin varies between and 353. We use IMDB- WIKI-DIK (IW-DIR) constructed in DIR 49). which contains 191.5K training images and 11.0K validation and testing images. The maximum age is 186 and minimum age is O: the maximum bin density is 7149, and minimum bin density is 1.
IW-D/R:
STS-B-DIR: We use STS-B-DIR constructed in DIR [49], which contains 5.2K pairs of training sentences and 1.0K pairs for validation and testing. This dataset is collection of sentence pairs generated from news headlines, video captions, etc. Each pair is annotated by multiple annotators with a similarity score between 0 and 5.
NYUD2-DIR: We use NYUD2-DIR constructed in DIR [49], which contains 50K images for training and 654 images for testing, and to make the test set balanced 9357 test pixels for each bin are randomly selected. The depth maps have an upper bound of 10 meters, and we set the bin length as 0.1 meter.
Baselines, We use Re [17] (for AgeDB-DIR IMDB-WIKI-DE WIKI-DIR and NYUD2-DIR) and BiLSTM [19 (for STS-B-DIR) as our backbone networks, and moredetails for baseline are in the supplement. we describe the baselines below.
Vanilla: We use the term VANILLA to denote plain model without adding any approaches.
18

---

Table 9: Accuracy on STS-B-DIR. For baselines, we directly use the reported performance in their paper and therefore do not have error bars.
Meato
MAAE
Parren
Spourme
Misiun
Madiun
Fine
VANILL [1ng
0.514
0:251
1,530
0.750
01:883
1.433
0.311
0.943
0.720
0.57
0.392
0.341
01003
11:05
(1790
VAE
0.588
0333
1,511
1,KG
01752
0.321
1.043
07763
0.751
0.724
0.521
0.340
0.122
2.674
0.501
b
0.507
0.315
1.406
1852
00211
01:73
1164
6.000
0.96
0.75
0.50
0.791
0.111
0/66
0.901
0.76
DVEER Nouic
0.80
1468
0.78
6367
6711
0331
0.336
0342
2381
0356
0233
0.690
0366
07588
0.355
0753
0.30
0.55
0.687
0313
5650
0.51
0670
B8V [3]
1.009
0254
1482
1866
0.915
0.381
0.20
6723
0.00
0.25
0.322
0311
0812
0541
0714
1.000
0993
1366
1855
0.812
0.733
0.88
5320
1573
6.71
0.96
030
6371
D47
0510
67W
0.84
0312
1339
1955
0.755
0.73
0.53
6172
0776
0.56
0.36
0.62
6763
67H
R556
826
6331
0.71
098
RANKSIM
0.403
0564
6911
0.399
0.89
LDS FDS DAR
0
1007
0560
1.55
1/06
0.81
0.357
1865
5.44
0.799
0.14
0.35
0.31
0.330
D/60
0.5%
0/90
(OuR))
8892
8.759
JANL 8209
0.7767711 100
(740
8.779 Jo
8.708 JUR:
BT76UASH K755
0.545
8.71
8.761000
0.566 Un: 0.61 1000
Chua 15. VASILA
+0.82
+1.55
+0.33
+8.83
-U164
40/34
6L354
+1863
+1534
-8J03
+ERSS
4URS
+0831
+0.55
+0.81
-8.13
Ouas vs Isrre Nover
+5.852
+0.58
+8.16
+0165
660
+1813
+8828
-8061
+UR9
40833
+8.15
+8.01
-6.14
Quas vs. DEN
+0.09)
-0.177
+0.43
40073
48366
68200
+154
-814
.8220
46.00
40844
18844
+8367
-8113
Oress RANESIM
+0.111
+0.133
+8303
-1.05
-8316
40017
+8.631
+9.83
-8.012
Table 10: Accuracy on NYUD2-DIR.
1.27
0.311
0.93
3155
1.80
E700
0.02
0.50
0.53
0.77
0.03
0.51
0.90
0.35
0.50
07.0
0.59
0.50
0,751
14.%
0758
3.35
1.81
1180
0.89
1463
13100
0.501
0.90
0.44
0111
0331
1855
1390
0.91
1.855
1.43
3.81
3140
0.00
0.96
0.10
0,89
0661
8.50
0.89
0.51
0554
3.99
0355
Ld0
1237
1854
1.09
0873
1472
5.53
636
0202
6.53
0174
4576
0554
0.92
1300
1963
1.01
110
1:50
5100
1953
1198
1370
1170
5134
1.03
0111
0.90
0344
0.8)
1490
Vil (Do0)
134
0.59
1.01
174
8.55
0.00
0.64
0/43
6122
6.35
6.08
6323
6.65
036
4508
080
0,80
(37
Osarss Iuns Mose
-8155
-8:85
1355
+037%
-8813
-088
-3035
-8.00
48005
+0133
-8455
40817
+8000
83
48.00
-8413
-8.55
-030
-0833
-MAT
+8.35
.8.34
-018
-064
40802
4.801
4600
.8.8
Sonthetic-Sapple Bastee-BBBdddaaa Methods: Various existing imbalanced regression methods are also included as baselines; these include Deep Ensemble 277. Infer Noise (2), SMOTER [40] and SMOGN [S].
Cost-Sensitive Reweighting: As shown in DIR 49), the square-root weighting variant
(SQINV) baseline (i.e. (EVeS k(yb. yv )p(w)) 1/2, always outperforms Vanilla. There- fore, for simplicity and fair comparison, all experiments (for both baselines and VIR) use SQINV weighting. To use SQINV in VIR, one simply needs to use the symmetric kernel k((,-) .) described in the Method section of the main paper. To use SQINV in DER, we replace the final layer in DIR 49 with the DER layer [ to produce the predictive distributions.
for data sample x; with its label y; which falls into the target bins b;. we divide the label space into three disjoint subsets: many-shot region {h; E B y: E b, & |/:] > 100}, medium-shot region {6, B y, E b, & 20 < y/l < 1000, and few-shot region {b; E B y/ E by 13/ < 20). where - denotes the cardinality of the set. We report results on the overall test set and these subsets with the accuracy metries discussed above.
Evaluation Process. Following 28 49
Implementation Details. We use ResNet-50 17 for all experiments in AgeDB-DIR and IMDB- WIKI-DIR. For all the experiments in STS-B-DIS DIR, we use 500-dimensions Glo word embeddings (840B Common Crawl version) BT (following (4 and a two-layer, di00-dimensional (per direc- tion) BiLSTM [9 with max pooling to encode the paired sentences into independent vectors u and v. and then pass [u; v; [u - =/s ue) to a regressor. We use the Adam optimizer (24 to train all models for 100 epochs, with same learning rate and decay by 0.1 and the 60-th and 90-th epoch, respectively. In order to determine the optimal batch size for training, we try different batch sizes and corroborate the conclusion from [49]. i.e., the optimal batch size is 256 when other hyperparameters are fixed. Therefore, we stick to the batch size of 256 for all the experiments in the paper. We also use the same configuration as in DIR 49 for other hyperparameters.
We use PyTorch to implement our method. For fair comparison, we implemented a Py PoToch version for the official TensorFlow implementation of [ER[I] To make sure we can obtain the reasonable uncertainty estimations, we restrict the range for a to [1.5, oo) instead of [1.0, oc) in DER. Besides, in the activation function SoftPlus, we set the hyperparameter beta to 0.1. As discussed in the main paper, we implement layer which produces the parameters n. V 4. We assign 2 as the minimum number for n, and use the same hyperparameter settings for activation function for DER layer.
19

---

N, 1= 2t" we need compute the Table 11: Uncertainty NYUD2-DIR.
Metries
N
AUSE 4
Shot
Many
Medium
Few
All
Many
Medium
Fow
DEEP
5.054
3.640
3.856
5.335
0.782
0.658
0.604
INPER NOISE
4.542
3.120
3.634
5.028
0.764
0.643
0.566
0.408
DER I
4.169
2,913
3.011
4,777
0.713
0.623
0.535
0,382
LDS FDS + DER 1
4.175
2.987
2.976
4,686
0.715
0.629
0,511
0,366
VIR (OURS)
3,866
2,815
2.727
4.113
0,690
0.603
0,493
0.335
OURS DER
+0.303
+0.098
+0.284
+0.664
+0.023
+0.020
+0.044
+0.047
OURS vs. LDS DER
+0.309
+0.172
+0.249
+0.573
40.025
40.018
+0.031
Table 12: Ablation study on À on AgeDB-DIR.
Metrics
Shol
All
Many
Medium
Few
All
Many
Medium
Few
All
Many
Medium
Few
10.0
104.31
91.01
116.43
196.35
7.88
7.38
8.42
11.13
3.827
3.733
4.140
4.407
196.12
86.28
76.87
101.57
132.90
7.19
6.75
7.97
9.19
3.785
3.694
3,963
4.151
0.01
86.86
76.58
99.95
147.82
7.12
6.69
7.72
9.59
3.887
3.797
4.007
4.401
0.001
87.25
74.13
104.78
162.64
7.13
6.64
7.92
9.63
3.980
3.868
4.161
4,546
, To search for a combination hyperparameters of prior distribution (%o, Pg. Op. 303 for NIG, we combine grid search method and random search method [2] to select the best hyperparameters. first intuitively assign a value and proper range with some step sizes which correspond to hyperparameters, then, apply grid search to search for the best combination for the hyperparameters on prior distributions. After locating a smaller range for each hyperparameters, we use random search to search for better combinations, if it exists. In the end, find our best hyperparameter combinations for NIG prior distributions.
C Complete Results
We include the complete results for all the experiments in AgeDB-Dii -IR, IMDB- WIKI-DIR, STS- B-DIR and NYUD2-DIR in T Table Table 9 and Table I0 These results demonstrate the superiority of our methods. Note that did not select report the baseline for DIR + DEEP ENS. since in DER paper [1, it has been showed that DER is better than Deep Ensemble method, therefore select report the baseline DIR+DER rather than DIR DEEP ENS..
D Discussions
D.1 Why We Need Bins
Throughout method, we need compute the statistics (i.e., the mean and variance) and the "statistics statistics" of data points (Line 164-165); computing these statistics (e.g., the mean) requires group of data points. Therefore, For example, in the equations from Line 176- 777, e.g., 1 statistics of bin b, which contains N₂ data points in the bin.
need to partition the continuous label size into B bins.
is also worth noting in the extreme case where (i) each data point has a different label y and (ii) we very small bin size, each bin will then contain exactly only data point.
D.2 Equal-Interval Bins versus Equal-Size Bins
Note that since our smoothing kernel function is based on labels (i.e., k(y, v/ )) it more reasonable to use equal-interval bins rather than equal-size bins.
20

---

Table 13: Comparison for different numbers of Bins. "Med." is short for "Medium".
Metries
Bins
MAEL
GM 4
Sbot
All
Many
Med.
Many
Many
Med.
RANKSIM
100
83.51
71.99
99.14
149.05
7.02
6.49
7.84
9.68
4.53
4.13
5.37
6.89
VIR (Ocas)
100
81.76
70.61
91.47
142.36
6,99
6.39
7.47
9.51
4.41
4.07
5.05
6.23
109.45
187.13
6,94
VIR (OURS)
33
84.77
77.29
95.66
125.33
7.01
6.70
7.45
8.74
4.36
4.20
4.73
4.94
RANKSIM
20
98.71
84.38
107.89
171.04
7.32
6.78
8.35
10.57
5.33
4.51
5.69
7.92
VIR (Ouas)
20
84.05
72.12
100.49
151.25
7.06
6.50
7.90
10.06
4.49
4.05
5.34
7.28
For example, if we use the iqual-interval bins [0. (1,))),,,)..... VIR will naturally compute
k(y, v') for y = 1, 2, 3, 4, 5,... and y/ 1, 2, 4, 5,.....
In contrast, if we use equal-size bins, VIR may end up with large intervals and may lead inaccurate kernel values for kyy, y/). To this, consider the kernel value k(y, 3/) between bins [2, 3.1) )) and [3.1, 8.9) k(2, 3.1). which is very inaccurate since 3.1 is very far away from the mean of the bin [3.1, 8.9) (i.e., 6). Using small and equal- bins can naturally address such issues.
where equal-size bins
are [0, 1). [1. 2). [2. (3.1,)8
D.3 The Number of Bins
Our preliminary results indicate that the performance of our VIR remains consistent regardless of the number bins, as shown in the Sec. | of the Supplement. Thus our paper, we chose to use the same number of bins the imbalanced regression literature [13] 49) for fair comparison with prior work. For example, in the AgeDB dataset where the regression labels are people's ' "age" in the range of 99, we use 100 bins, with each year one bin.
D.4 Reweighting Methods and Stronger Augmentations
Our method focus reweighting methods, and using augmentation (e.g.. the SimCLR pipeline (TO an orthogonal direction our work. However, we expect that data augmentation could further could perform data augmentation only
improve VIR's performance. This is because minority data to improve accuracy in the minority group, but this is sub-optimal; the reason that one could potentially further perform data augmentation majority group without sacrificing too much accuracy in the minority group. However, performing data augmentation on both minority and majority groups does not transform imbalanced dataset to an balanced dataset. This is why our VIR is still necessary; VIR could be used on top of any data augmentation techniques address the imbalance issue and further improve accuracy.
majority data to improve accuracy in the
D.5 Discussion on I.I.D. and N.L.D. Assumptions
Generalization Error, Bias, and Variance. We could analyze the generalization error of our VIR by bounding the generalization with the sum of three terms: (a) the bias of our estimator, (2) the variance of our estimator, (3) model complexity. Essentially VIR uses the N.I.D, assumption increases our estimator's s bias, but significantly reduces its variance in the imbalanced setting. Since the model complexity is kept the same (using the same backbone neural network) as the baselines, N.L.D. will lead to lower generalization error.
Variance of Estimators in Imbalanced Settings. In the imbalanced setting, one typically use inverse weighting (i.e., the IPS estimator in Definition AA.4 produced an unbiased estimator (i.e., making the first term of the aforementioned bound zero). However, for data with extremely low density, its inverse would be extremely large, therefore leading to a very large variance for the estimator. Our VIR replaces I.I.D. with N.I.D. to "smooth out" such singularity, and therefore significantly lowers the variance of the estimator (i.e., making the second term of the aforementioned bound smaller), and ultimately lowers the generalization error.
21

---

D.6 Why We Need Statisties of Statistics for Smoothing
Compared with DIR [49] which only considers the statisties for deterministic representations, our VIR considers the statisties of statisties for probabilistic representations, this is because the requirement to perform feature smoothing to get the representation 2; necessitates the computation of mean and variance of 2,'s neighboring data (i.e., data with neighboring labels). Here 2; contains the statistics of neighboring data. In contrast, our VIR also needs to generate uncertainty estimation, which requires a stochastic representation for 2;, e.g., the mean and variance of z; (note that 2; itself is already form of statistics). This motivates the hierarchical structure of the statistics of statistics. Here the variance measures the uncertainty of the representation.
D.7 The Choice of Kernel Function
The DIR paper shows that a simple Gaussian kernel with inverse square-root weighting (i.e., SQINV) achieves the best performance. Therefore, we use exactly the same parameter configuration as the DIR paper [49]. Specifically, we set a = 2; for label in bin b, we define neighboring labels as labels yo such that lyv y/ 2, i.e., B contains 5 bins. For example, if yb 23, its neighboring labels are 21, 22, 23, 24, and 25.
Besides, preliminary results also show that the performance is not very sensitive to the choice of kernels, as long as the kernel k(a, b) reflects the distance between a and b, i.c., larger distance between a and b leads to smaller k(a,, b)-
D.8
Why VIR Solves the Imbalanced Regression Problem
Our training objective function (Eqn.5 in the main paper) the negative log likelihood for the Normal Inverse Gaussian (NIG) distribution, and each posterior parameter wyfnt aç) of the NIG distribution is reweighted by importance weights, thereby assigning higher weights to minority data during training and allowing minority data points to benefit more from their neighboring information.
Take 1i as an example. Assume a minority data point (I;; Va) that belongs to bin b, i.e., its label Y/ = yo- Note that there is a loss term (U: - 7 ) in Eqn.5, where 7f is the model prediction, y; the label, and of is the importance weight for this data point.
Here D/- = vo + (Eveg kwy; yo (wo))((w)) () n, where n, represents the pseudo-count for the NIG distribution. Since (Iz, Mi) is minority data point, data from its neighboring bins has smaller frequency p(ys-) and therefore smaller Σ ΣvEB k(yb,, yv )(())), leading to a larger importance weight v for this minority data point in Eqn.5.
This allows VIR naturally put more focus the minority data, thereby alleviating the imbalance problem.
D.9 Difference from DIR, VAE and DER
From a technical perspective, VIR is substantially different from any combinations of DIR 44 VAE 225, and DER [ Specifically,
VIR is a deep generative model to define how imbalanced data are generated, which is learned by a principled variational inference algorithm. In contrast, DIR is a simply diseriminative model (without any principled generative model formulation) that directly predict the labels from input. is more prone to overfitting.
DIR uses deterministic representations, with one vector as the final representation for each data point. In contrast, our VIR uses probabilistic representations, with one vector as the mean of the representation and another vector as the variance of the representation. Such dual representation is more robust to noise and therefore leads better prediction performance.
DIR is a deterministic model, while our VIR is a Bayesian model. Essentially VIR equivalent to sampling infinitely many predictions for each input data point and averaging these predictions. Therefore intuitively makes sense that VIR could lead to better prediction performance.
22

---

Different from VAE and DIR, VIR introduces reweighting mechanism naturally through the pseudo-count formulation in the NIG distribution (discussed in the paragraphs Intuition of Pseudo-Counts for VIR and From Pseudo-Coune to Balanced Predictive Distribution in the paper). Note that such a reweighting mechanism more natural and powerful than DIR since it is rooted in the probabilistic formulation.
Unlike for the standard VAE, the optimal prior (that maximizes ELBO) is known to be the aggregated posterior, our optimal prior is neighbor- version of aggregated posterior: for standard VAE, different data points contribute independently to the aggregated posterior; in contrast, for our VIR, the importance of each data point with respect to the aggregated posterior is affected by data points with neighboring labels.
It is also worth noting that DIR cannot produce uncertainty estimation since it is deter- ministic model. In contrast, Our VIR formulates a probabilistic deep generative model for imbalanced data, and therefore naturally produce both more accurate predictions compared to DIR and better uncertainty estimation compared to DER.
D.10 Bayesian Neural Networks (BNNs)
We do not consider other BNNs in this work because:
Weights in Bayesian Neural Networks (BNNs) are extremely nigh-dimensional; therefore BNNs have several limitations, including the intractability of directly inferring the posterior distribution of the weights given data, the requirement and computational expense of sampling during inference, and the question of how to choose weight prior [.. In contrast, evidential regression does not have these challenges.
In our preliminary experiments, we found that typical BNN methods suffer from computa- tional inefficiency and would require at least two to three times more computational time and memory usage. In contrast, evidential regression does not involve such computation and memory overhead: its overhead only involves the last (few) layers. and is therefore minimal. Additionally, as demonstrated in (27), Deep Ensemble typically performs as well as or even better than BNNs. Our method outperforms Deep Ensemble, therefore suggesting its superiority over typical BNN methods.
E
Additional Experiment Results
E.1
Ablation Study on VIR
In this section, we include ablation studies to verify that VIR can outperform its counterparts in DIR (i.e., smoothing on the latent space) and DER (i.e., NIG distribution layers).
Ablation Study on g(2, {x; { ). To verify the effectiveness of VIR's ' encoder g(z [x; J )
Table 14: Ablation study for Accuracy.
we replace VIR's predictor p(y: (2,) with lin-
Monse
Skot
Mary Medi
ear layer (as in DIR). Table 14 shows that compared to its counterpart, FDS 49), our encoder-only VIR leads to considerable im- provements even without generating the NIG distribution. Both verify the effectiveness of our VIR's 's's |(z;|<x,,
FDS
109.38 $3.99 124.55
2169778112
EsCuOER-OND
121.78
309.20/111
PASDICTOR (oens)
363.76 7.28
LDS-PDS
$4.00
111.20
200.27
1255
7X01
K:34 10.79
LDS PREDICTOR-ONLY VJR (OuRs)
57.48
71.72
307.64
151,99
2.17
566
8.06
9.30
LDS ENCODER-OSES VLR (oUBS)
86:72
302.55
171.52
7.51
7.05
7.93
1045
VJR (OURS)
81.76
7861
91.47
142.36
6.00
6.39
7.47
9.51
Ablation Study on p(y 2..) To verify the effec- tiveness of VIR's predictor p(y (2,). we replace VIR's 's encoder q(z, If {x, IN with simple de- terministic encoder as in DER [.. Table 14 and Table [5] show that compared to DER, the coun- terpart of VIR's predictor, our VIR's predictor still outperforms than DER, demonstrating its effectiveness. Both verifies our claim that directly reweighting DER breaks NIG and leads to poor performance.
Table 15: Ablation study for Uncertainty.
Shoe
All Mary Med Fowr All
Many Mad. Fiw
4.304
L.DE PHRDICIOR-ONI VIR (or usj
N.22 N.I N.21 4.7000.441 CUIST OLTTA C.IT
VIR COURS)
3.70 3,5M 3,505 4.190 9.456 0.324 0.414
23

---

E.2 Ablation Study on A.
In this section, we include ablation studies on the in our objective function. For À {10.0, 1.0, 0.1, 0.01, 0.001), we run our VIR model on the AgeDB dataset. Table 12 shows the results. We can observe that our model achieves the best performance when 0.1.
E.3 Ablation Study Number of Bins
Table 16: Calibration Uncertainty AgeDB-DIR.
In this section, we include ablation studies on the number of bins in our settings. For the cases with 100/1 = 100, 100/3 R 33, and 100/5 20 bins, we run our VIR model on the AgeDB dataset. Table shows the results. We can observe that our VIR remains consistent regardless of the number of bins.
Mesics
NLL
AUSE
Mory
May
[CALBRATER]] Doap Ens.
64022
9,32
(3506
0.306
ENDER NoisE [
Hols 4411
4,506
5:542
(448
0.48
I.457
14466
[Catomatro]] Duree Noves N
1:70 4:83
1637/0/00
0.676
0:410 D415
IER
5311
3,510
4/3310.933
0.609
0:466
[CALBRATRH]
3636
3.00
48012
14450
E.4 Calibrated Uncertainty
[CALIRATED) VIR (OLBs)
VIR (00RS)
1.001 3.99 3.90
3.577 3493 3,96
4.196/2/446
3.866
0.359 0.332 0.376 0.266
1:006
11:00 11416
In bid enhance the evaluation of our model's uncertainty, used our validation set to apply calibration techniques (specifically, variants of temperature scaling (14) on different methods. We focused on each the output distribution parameters 2/, a, 3 discussed our main paper, introducing individual scalar weights for each parameter to calibrate uncertainty estimation. Upon deriving each weight Wy, Wa w from the validation set, we incorporated them into the test dataset ascertain the final performance. The data outlined in Table [o] infidiiaess that following re-calibration, the uncertainty was further optimized. Notwithstanding such uncertainty calibration, our model persists in demonstrating superior performance compared to other benchmark methods.
E.5 Error bars on Accuracy
In order further underscore the superiority of methodology, also included error bars for proposed method (i.e., VIR) which are generated from five independent runs. Note that we did not report eиrоr bars for those second/third best baselines (e.g., RankSim, LDS+FDS) since we directly use the reported performance from their papers, and in NYUD2-DIR, all the error bars on VIR are approximately 0.001. Due to the width constraints of the paper, these are included in Table I0 Results in Table Table g and Table Table demonstrate the effectiveness of our approach.
24


Reviews:
[{"review_id": "KcDxVaWwZN", "reviewer": "Reviewer_cFEB", "summary": "In this work, authors recognize that although the existing regression models for the imbalanced datasets have been mainly developed to improve the prediction accuracy, they overlooked the quality of the uncertainty estimation. In this context, authors propose a deep probabilistic regression framework, to improve the uncertainty estimation performance as well by combining the idea of [1] and [2]. \n\nSpecifically, authors first consider multiple bins, splitted on the range of labels to get statistic of the labels. Then, they revise the latent statistic of VAE for the imbalanced datasets, by smoothing these features based on the statistic on each bins and then applying probabilistic whitening and recoloring procedure. Next, they use the revised latent features to get the posterior distribution of NIG distribution, which acts as the pseudo counts that alleviates the issue of imbalanced sets. Last, they employ these parameters for prediction and training.   \n\nEmpirically, authors demonstrate that the proposed approach can improve the performance of the prediction accuracy and uncertainty estimation on various datasets.\n\n[1] Delving into Deep Imbalanced Regression - ICML 21\n\n[2] Deep Evidential Regression - NeurIPS 20", "questions": "* Is the performance of the proposed method consistent up to the number of bins? How can performance vary with the number of bins?\n\n* Why did the authors employ the stochastic latent feature through VAE, that is different approach to the [1] ? \n Does the stochastic feature improve the performance as comparing when the deterministic statistic is used to update the parameters of the NIG posterior distribution ?\n\n* Did the authors consider the BNNs with DIR or the Deep ensemble approach [2] with DIR to solve the imbalanced regression problem ? \n\n\n[1] Delving into Deep Imbalanced Regression - ICML 21\n\n[2] Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles - NeurIPS 17", "limitations": "See above Weaknesses and Questions.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This work extends the DIR of [1], as the probabilistic model, to estimate the uncertainty.\n\n* This work considers to use the NIG posterior distribution, updated by the statistic of the stochastic latent feature. This seems to help balance the latent features of the imbalanced labels and thus yield the credible uncertainty for the imbalanced datasets. I believe that this is a novel part of this work as comparing [1] and [2].\n\n[1] Delving into Deep Imbalanced Regression - ICML 21\n\n[2] Deep Evidential Regression - NeurIPS 20", "weaknesses": "* Absence of ablation study\n> In experiment section, it seems that the proposed method improves the prediction accuracy and uncertainty estimation. However, the current work does not investigate (1) whether each trick of the proposed method, such as the use of the stochastic latent feature (VAE) and use of the posterior distribution (NIG), is effective and (2) whether the proposed method is consistent up to the number of bins. \n\n* Less explanation on why the evidential regression model is used along with DIR, instead of using other BNNs.\n> In general, BNNs is widely used to estimate the uncertainty. I believe that applying the BNNs with DIR could be a direct way to solve the targeted problem. However, author takes the evidential regression approach, without explaining the reason or its motivation. This seems to less persuade why the proposed method is reasonable. If authors provide the motivation of this approach or can demonstrate that the proposed method could outperform the results of the DIR, obtained by BNNs or other approach for uncertainty estimation, I believe that the contribution of this work would be more clear and strong.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and insightful questions. We are glad that you find our work ``\"novel\"``. Below we address your questions one by one. \n\n**Q1.1: \"...ablation study...not investigate (1) whether each trick ... the stochastic latent feature (VAE) ... the posterior distribution (NIG), is effective ...\"**\n\nThank you mentioning this. Actually we did perform ablation studies and the results have been included in the Supplementary Material (please see Tables 8 and 9). \n\nBesides, according to your suggestions, we have also performed additional ablation studies. The results in Tables 3.1 through 3.2 below (more results in **Table 1 of the one-page PDF file**) demonstrate the effectiveness of each element in our proposed method. **VIR w/o VAE** refers to VIR *without stochastic latent features*, i.e., \"the deterministic statistic is used to update the parameters of the NIG posterior distribution\". **VIR w/o NIG** refers to VIR without NIG. Note that **VIR w/o VAE & NIG** is equivalent to **LDS+FDS+DER** in our main paper; here we include its results in the table as a reference to further demonstrate the effectiveness of our method. \n\nTable 3.1: Ablation studies on AgeDB in terms of MSE\n\n| model | overall | many | median | few |\n| :---------: | :------: | :------: | :------: | :------: |\n|VIR w/o VAE & NIG| 112.62 | 94.21 | 140.03 | 210.72 |\n|VIR w/o NIG| 87.48 | 73.72 | 107.64 | 161.69 |\n|VIR w/o VAE| 96.46 | 86.72 | 102.56 | 171.52 |\n|VIR (Ours)| **81.76** | **70.61** | **91.47** | **142.36** |\n|||||\n\nTable 3.2: Ablation studies on AgeDB in terms of NLL\n\n| model | overall | many | median | few |\n| :---------: | :------: | :------: | :------: | :------: |\n|VIR w/o VAE & NIG| 3.787 | 3.689 | 3.912 | 4.234 |\n|VIR w/o NIG| 3.722 | 3.604 | 3.821 | 4.209 |\n|VIR w/o VAE| 3.784 | 3.685 | 3.866 | 4.218 |\n|VIR (Ours)| **3.703** | **3.598** | **3.805** | **4.196** |\n|||||\n\nIt's worth noting that we included similar ablation studies in Tables 8 and 9 of the Supplementary Material, where the \"Encoder-only VIR\" is equivalent to \"VIR w/o NIG & LDS\", and \"Predictor-only VIR\" is equivalent to \"VIR w/o VAE & LDS\". These results also verify the effectiveness of each element in our proposed method. \n\nwe will incorporate the additional ablation studies above into the Supplementary Materials in the revision as suggested. \n\n**Q1.2: \"...  method is consistent up to the number of bins.\" \" performance vary with the number of bins?\"**\n\nThis is a good question. Please refer to **Q1 in the Global Response**.\n\n**Q2: \"... why the evidential regression model is used along with DIR, instead of using other BNNs.\"**\n\nThis is a good point. We do not consider other BNNs in this work because: \n\n**(1)**  **Weights** in Bayesian Neural Networks (BNNs) are extremly high-dimensional; therefore BNNs have several limitations, including the intractability of directly inferring the posterior distribution of the **weights** given data, the requirement and computational expense of sampling during inference, and the question of how to choose a **weight** prior [3]. In contrast, evidential regression does not have these challenges. \n\n**(2)** In our preliminary experiments, we found that typical BNN methods suffer from computational inefficiency and would require at least two to three times more computational time and memory usage. In contrast, evidential regression does not involve such computation and memory overhead; its overhead only involves the last (few) layers, and is therefore minimal. \n\n**(3)** Additionally, as demonstrated in [2], Deep Ensemble typically performs as well as or even better than BNNs. Our method outperforms **Deep Ensemble** (Tables 1~6 in the paper, with more results in **Q4** below), therefore suggesting its superiority over typical BNN methods. \n\nWe appreciate your suggestion and will incorporate the discussion above into our revised paper. \n\n**Q3: \"Why ... employ the stochastic latent feature through VAE ...? Does the stochastic feature improve ...?\"**\n\nWe are sorry for the confusion. Please refer to **Q2 in the Global Response**.\n\n**Q4: \"... consider the BNNs with DIR or the Deep ensemble [2] with DIR ...?\"**\n\nAs mentioned **Q2**'s response, we did not include other BNNs due to their limitations in the context of our work. Furthermore, as demonstrated in [2], Deep Ensemble typically performs as well as or even better than BNNs. Our method outperforms Deep Ensemble (Tables 1~6 in the paper), therefore suggesting its superiority over typical BNN methods. \n\nAccording to your suggestion, we ran additional experiments on **combining Deep Ensemble and DIR** and report the results in Tables 4.1~4.2 below (more results in **Table 3 of the one-page PDF file**). These results show that while this combination allows DIR to produce uncertainties, our method can still outperform it by a large margin. \n\nThanks for your insightful question, and we will include these additional results and the discussion above in the revision. \n\nTable 4.1: Results for DIR + Deep Ensemble in terms of MSE\n\n| model | overall | many | median | few |\n| :---------: | :------: | :------: | :------: | :------: |\n|DIR + Deep Ensemble| 94.10 | 80.24 | 109.45 | 182.52 |\n|VIR (Ours)| **81.76** | **70.61** | **91.47** | **142.36** |\n|||||\n\nTable 4.2: Results for DIR + Deep Ensemble in terms of NLL\n\n| model | overall | many | median | few |\n| :---------: | :------: | :------: | :------: | :------: |\n|DIR + Deep Ensemble| 5.069 | 4.772 | 4.574 | 5.236 |\n|VIR (Ours)| **3.703** | **3.598** | **3.805** | **4.196** |\n|||||\n\n[1] Delving into Deep Imbalanced Regression.\n\n[2] Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles.\n\n[3] Deep Evidential Regression."}, {"review_id": "TaGamlhz54", "reviewer": "Reviewer_ivhk", "summary": "This paper proposes a variational imbalanced regression model by taking the Neighboring and Identically Distributed (N.I.D.) assumption to solve both imbalanced regression and uncertainty estimation problems. Experiments on four imbalanced datasets demonstrate the effectiveness of the proposed method.", "questions": "1) What are the new challenges of imbalanced regression compared with imbalanced classification?\n2) Why do you partition the label space into B equal-interval bins? Does the choice of B affect the performance of VIR?\n3) The choice of the kernel functions and experiments for the corresponding parameters selection should be given since it determines the importance weights.\n4) In section 4.1, the authors claim that N.I.D. will lead to a lower generalization error just by simple text description, without any rigorous computation or derivation, this is not convincing for me.", "limitations": "Yes, the authors have adequately addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) Compared with imbalanced classification,  imbalanced regression is underexplored and also an important topic.\n2) The Neighboring and Identically Distributed (N.I.D.) assumption seems more reasonable than the Independent and Identically Distributed (I.I.D.) assumption.\n3) The proposed method improves not only the performance of few-shot region, but also the performance of many-shot region.", "weaknesses": "1) The description of some parts of the paper is not clear, e.g., how to define the neighboring labels?  What is the detailed formulation of the importance weights in line 237?\n2) Some statements in the paper are somewhat subjective and lack support, e.g., in line 42-43, the authors claim that \"This allows the negative log likelihood to naturally put more focus on the minority data\", but I do not find any \"naturally\" thing, the important weights are mainly determined by the kernel functions, which needs manually selection; in line 273, the authors claim that \"Such dual representation is more robust to noise\", but they neither list any references nor conduct any experiments to support it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive and encouraging comments with insightful questions. We are glad that you find the problem ``\"important\"`` and our N.I.D. assumption ``\"reasonable\"``, and acknowledge that our method improves performance. Below we address your questions in detail one by one.\n\n**Q1: \"... how to define the neighboring labels? ... detailed formulation of the importance weights in line 237?\"**\n\nWe are sorry for the confusion and the typo.\n\nIn Line 237~241, it should be $(\\sum_{b'\\in B}k(y_b,y_{b'})p(y_{b'}))^{-\\frac{1}{2}}$ rather than $(\\sum_{b'\\in B}k(y_b,y_{b'}))^{-\\frac{1}{2}}$. Here $B$ is the set of bin $b$'s neighboring bins, $k(y_b,y_{b'})$ is a kernel measuring the distance between labels $y_b$ and $y_{b'}$, and $p(y_{b'})$ is the frequency (density) of label $y_{b'}$ in the dataset. We use a **Gaussian kernel** $k(a, b)=\\exp({-\\frac{(a - b)^{2}}{2\\sigma^2}})$. \n\nFor fair comparison, we use exactly the same parameter configuration as the DIR paper [1]. Specifically, we set $\\sigma=2$; for label $y_b$ in bin $b$, we define neighboring labels as labels $y_{b'}$ such that $|y_{b'}-y_b|\\leq 2$, i.e., $B$ contains $5$ bins. For example, if $y_b=23$, its neighboring labels are $21$, $22$, $23$, $24$, and $25$. \n\nWith the definition above, we can see that data in the *minority* neighborhood (smaller $p(y_{b'})$) has *larger* importance weights $(\\sum_{b'\\in B}k(y_b,y_{b'})p(y_{b'}))^{-\\frac{1}{2}}$ in the training objective function. \n\nWe will include the details above in the revision as suggested. \n\n**Q2.1: \"... allows the negative log likelihood to naturally put more focus on the minority data...\"**\n\nWe are sorry for the confusion. Please refer to **Q3 in the Global Response**.\n\n**Q2.2: \"...needs manually selection;...\" \"The choice of the kernel functions...should be given...\"**\n\nThis is a good question. Please refer to **Q1 above** and **Q4 in the Global Response** .\n\n**Q2.3: \"...\"Such dual representation is more robust to noise\"...references...experiments...\"**\n\nWe are sorry for the confusion. Please refer to **Q2 in the Global Response**.\n\n**Q3: \"...new challenges of imbalanced regression compared with imbalanced classification?\"** \n\nThis is a good question. \n\nAs shown in **Figure 1 of the 1-page PDF**, we used two datasets, CIFAR-100, a 100-class classification dataset [2] and IMDB-WIKI [3], an age estimation dataset with labels in the range 0~99, to compare imbalanced data challenges in classification vs. regression. We adjusted their label ranges for consistency and simulated data imbalance, ensuring identical label density distribution, as seen in Fig. 1(a) in our global-response PDF. A ResNet-50 model trained on these datasets highlighted differences in test error distributions.\n\nResults from CIFAR-100 showed a negative correlation between test error and label density ($-0.76$), which is expected since classes with more samples often have lower errors. However, IMDB-WIKI, despite having the same label density as CIFAR-100, had a more uniform error distribution that **did not align as closely with label density ($−0.47$)**, as shown in Fig. 1(b).\n\n*This distinction highlights unique challenges in imbalanced regression*. Traditional imbalanced learning methods, which address the imbalance in the **empirical** label density, work for classification but might falter for regression with continuous labels. The challenge grows in *probabilistic* imbalanced regression, where both accuracy and *uncertainty estimation* matter. Addressing these challenges are therefore the focus of our paper. \n\n**Q4: \"Why...partition...into B equal-interval bins? Does the choice of B affect the performance of VIR?\"**\n\nThis is a good question. Please refer to **Q1 in the Global Response**.\n\n**Q5: \"In section 4.1, the authors claim that N.I.D. will lead to a lower generalization error just by simple text description, without any rigorous computation or derivation, this is not convincing for me.\"**\n\nWe are sorry for the confusion. We meant to use Section 4.1 to discuss *intuition* of our methods. We have prepared a rigorous proof and will include it in the Supplementary Material of our revision. Below we discuss the main idea of the proof. \n\nIn general, the generalization error is bounded with probability $1-\\eta$ by: test error $\\leq$ training error + bias + variance, where \n\nbias $=\\frac{\\Delta}{N} \\sum_{(x,y)} |1 - \\frac{P_{y}}{\\hat{P}_{y}}|$,\n\nvariance $=\\frac{\\Delta}{N} \\sqrt{\\frac{\\log ( 2 |{H}| / \\eta)}{2}} \\sqrt{\\sum_{(x,y)}(\\hat{P}_{y})^{-2}}$.\n\nHere\n\n(1) $\\hat{P}_{y}$ is the smoothed label distribution used in our VIR's objective function,\n\n(2) $P_{y}$ is the label distribution,\n\n(3) $N$ is the number of data points,\n\n(4) $\\Delta=y_{max}-y_{min}$,  where $y_{max}$ and $y_{min}$ are the maximum and minimum labels in the dataset, respectively, and\n\n(5) $\\mathcal{H}$ is the finite hypothesis space of prediction models.\n\nWe can see that if one directly uses the original label distribution in the training objective function, i.e., $\\hat{P}_{y}=P_y$:\n\n(a) The \"bias\" term will be $0$.\n\n(b) However, the \"variance\" term will be extremely large for minority data because $\\hat{P}_{y}$ is very close to $0$.\n\nIn contrast, under N.I.D., $\\hat{P}_{y}$ used in the training objective function will be smoothed. Therefore:  \n\n(a) The minority data's label density $\\hat{P}_{y}$ will be smoothed out by its neighbors and becomes larger (compared to the original $P_y$), leading to smaller \"variance\" in the generalization error bound.\n\n(b) Note that $\\hat{P}_{y}\\neq P_y$, VIR (with N.I.D.) essentially increases bias, but significantly reduces its variance in the imbalanced setting, thereby leading to a lower generalization error.\n\nWe will include the discussion above and our proof in the revision as suggested.\n\n[1] Delving into Deep Imbalanced Regression.\n\n[2] Learning multiple layers of features from tiny images.\n\n[3] Deep Expectation of Real and Apparent Age from a Single Image Without Facial Landmarks."}, {"review_id": "KGrEkma0Yk", "reviewer": "Reviewer_9XGJ", "summary": "The authors propose a variational regression model for imbalanced data, which (1) borrows data with similar regression labels for variational distribution (neighboring and identically distributed: N.I.D.) and (2) utilize the conjugate distributions to impose probabilistic reweighting on the imbalanced data to give better uncertainty estimation. Experiments show the proposed model achieves the SOTA on several datasets.", "questions": "Here are my questions and comments.\n\n- [Question] Just out of curiosity. What is the optimal prior for VIR? For the standard VAE, the optimal prior (that maximizes ELBO) is known to be the aggregated posterior. This might lead to an interesting future work.\n\n- [Comment (minor)] \"NIG\" on L.137 should be spelled out here, rather than on L. 213, or define NIG on L. 40.\n\n- [Comment (super-minor)] (1), (2), ... in the main text might sometimes be confused with the number of equations. Possible alternatives are: (i), (ii),..., (I), (II), ..., (a), (b), ..., (A), (B), ..., 1), 2), ..., etc.\n\n- [Question] Where does the idea of the hierarchical structure of the statistics (L. 166--195) (statistics of statistics for smoothing) come from? What motivated this idea? This is very interesting.\n\n- [Comment (minor)] L. 191: cross -> across?\n\n- [Comment (minor)] L. 212: correspond -> corresponding?\n\n- [Question (major)] At first glance, I could not fully understand why Eqn. 5 alleviate the imbalance problem. Could you clarify the underlying mechanism more?\n\n- [Question (major)] Is the performance improvement (accuracy and uncertainty) \"significant\"? If so, in what sense? (Sorry for the vague question, but this is an important point when evaluating model's performance and paper's quality because a marginal improvement is sometimes meaningless.)", "limitations": "Limitations are included in Section 5.3.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Imbalanced regression is a research area that is yet to be explored, although its real-world application is important.\n\n- In addition, the proposed model can provide uncertainty of the prediction, which is important for real-world applications.\n\n- The uncertainty estimation on imbalanced datasets is also an interesting field but is yet to be explored.\n\n- Overall, the present topic is very relevant in the community. I encourage the authors to develop and explore this direction more, in view of the nice performance of the model.\n\n- The performance of the proposed model is excellent. Both of the accuracy and uncertainty estimation outperform the SOTA models.\n\n- The present paper is easy to follow and well-written. I enjoyed reading it. I found some nice \"road signs\" for the readers to follow the logic and story.", "weaknesses": "- The code for reproduction is not available. I would like to see and use your code. I could not find any other major weakness.\n\n- See Questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive and encouraging comments as well as the insightful questions. We are glad that you find the problem we address ``\"important\"``/``\"interesting\"``, our performance ``\"excellent\"``, and our paper ``\"easy to follow\"``/``\"well-written\"``. Below we address your questions one by one. \n\n**Q1: \"The code for reproduction is not available. I would like to see and use your code. I could not find any other major weakness.\"**\n\nThank you very much for your interest. We have finished cleaning up the source code and will release it after the paper is accepted to facilitate further research in the community. \n\n**Q2: \"What is the optimal prior for VIR? For the standard VAE, the optimal prior (that maximizes ELBO) is known to be the aggregated posterior. This might lead to an interesting future work\"**\n\nThanks for the insightful question. Our optimal prior is a *neighbor-weighted* version of aggregated posterior: for stanard VAE, different data points contirbute **independetly** to the aggregated posterior; in contrast, for our VIR, the importance of each data point with repspect to the aggregated posterior is **affected by** data points with neighboring labels. We will include the discussion above in the revision to provide more insight into the difference between VAE and VIR. \n\n**Q3: \"\"NIG\" on L.137 should be spelled out here, rather than on L. 213, or define NIG on L. 40.\"**\n\nThank you for your suggestion. We will fix this in the revision.\n\n**Q4: \"... in the main text might sometimes be confused with the number of equations...\"**\n\nThank you for your suggestion. We will revise our paper accordingly in the revision.\n\n**Q5: \"Where does the idea of the hierarchical structure of the statistics (L. 166--195) (statistics of statistics for smoothing) come from? What motivated this idea? This is very interesting.\"**\n\nThis is a good question and thank you for your interest. \n\n**(1) Statistics.** The requirement to perform feature smoothing to get the representation $z_i$ necessitates the computation of mean and variance of $z_i$'s neiboring data (i.e., data with neighboring labels). Here $z_i$ contains the **statistics** of neighboring data. \n\n**(2) Statistics of Statistics.** Furthermore, uncertainty estimation requires a stochastic representation for $z_i$, e.g., the mean and variance of $z_i$ (note that $z_i$ itself is already a form of statistics). This motivates the hierarchical structure of the statistics, i.e., **statistics of statistics**. Here the variance measures the uncertainty of the representation. We appreciate your suggestion and will incorporate this discussion into the revised version of our paper. \n\n**Q6: \"L. 191: cross -> across? L. 212: correspond -> corresponding?\"**\n\nWe are sorry for the typos, and will fix them in the revision. \n\n**Q7: \"At first glance, I could not fully understand why Eqn. 5 alleviate the imbalance problem. Could you clarify the underlying mechanism more?\"**\n\nWe are sorry for the confusion. Please refer to **Q2 in the Global Response**.\n\n**Q8: \"Is the performance improvement (accuracy and uncertainty) \"significant\"? If so, in what sense? (Sorry for the vague question, but this is an important point when evaluating model's performance and paper's quality because a marginal improvement is sometimes meaningless.)\"**\n\nThank you for mentioning this. According to your suggestion, we ran the corresponding hypothesis tests, and the p values are in the range of $(9.08 \\times 10^{-7}, 3.38 \\times 10^{-4})$, much lower than the threshold of $0.05$ and therefore verifying the significance of VIR's performance improvement. We will include these results in the revision to strengthen the paper as suggested. "}, {"review_id": "uJHSQXU6jM", "reviewer": "Reviewer_C6MR", "summary": "This paper simultaneously addresses the label imbalance problem and uncertainty qualification capability in regression.\nThe authors propose to enhance the reweighting technique dealing with the imbalance problem in (Yang et al., 2021) to be applicable to VAE and combine the method with the output distribution and the corresponding loss in (Amini et al., 2020), which provides uncertainty qualification capability.\nExperimental results on several real-world datasets demonstrate that the proposed method performs better than state-of-the-art imbalanced regression methods in terms of both accuracy and uncertainty estimation.", "questions": "NA", "limitations": "They discussed that the exact computation of variance of the variances is challenging in Section 5.3.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Originality:\n- The authors simultaneously address the label imbalance problem and uncertainty estimation capability in regression, which is a novel problem setting.\n\n### Quality:\n- The combination of the reweighting technique in (Yang et al., 2021) and the output distribution and the corresponding loss in (Amini et al., 2020) is non-trivial and works well.\n\n- The authors showed the superiority of the proposed method experimentally in terms of both accuracy and uncertainty estimation, where they used multiple public datasets.\n\n### Clarity:\n- The presentation is clear.", "weaknesses": "- Good combination of the SOTA, but the originality can be limited because of that.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive and encouraging comments as well as the insightful questions. We are glad that you find the problem we address ``\"novel\"``, our method ``\"non-trivial\"``/``\"works well\"``, our presentation ``\"clear\"``, and that experiments show our method's ``\"superiority\"`` ``\"in terms of both accuracy and uncertainty estimation\"``. Below we address your question. \n\n**Q1: \"Good combination of the SOTA, but the originality can be limited because of that.\"**\n\nThank you for acknowledging our contribution and our SOTA performance. We should have better highlighted our originality in the main paper. Specifically, our VIR not only combines existing SOTA methods (e.g., DIR), as you mentioned, but also presents substantial divergence from them (see a more in-depth discussion of these differences in Section 1.2 of the Supplementary Material):\n\n**(1)** VIR is a deep generative model to define how imbalanced data are generated, which is learned by a principled variational inference algorithm. In contrast, DIR is a simply discriminative model (without any principled generative model formulation) that directly predicts the labels from input. It is more prone to overfitting.\n\n**(2)** DIR uses deterministic representations, with one vector as the final representation for each data point. In contrast, our VIR uses probabilistic representations, with one vector as the mean of the representation and another vector as the variance of the representation. Such dual representation is more robust to noise and therefore leads to better prediction performance.\n\n**(3)** DIR is a deterministic model, while our VIR is a Bayesian model. Essentially VIR is equivalent to sampling infinitely many predictions for each input data point and averaging these predictions. Therefore intuitively it makes sense that VIR could lead to better prediction performance.\n\n**(4)** Different from VAE and DIR, VIR introduces a reweighting mechanism naturally through the **pseudo-count formulation** in the NIG distribution (discussed in the paragraphs Intuition of Pseudo-Counts for VIR and From Pseudo-Counts to Balanced Predictive Distribution of the paper). Note that such a reweighting mechanism is more natural and powerful than DIR since it is rooted in the probabilistic formulation.\n\nBesides methodological originality, our other contributions include: \n\n**(a)** We identify the problem of probabilistic deep imbalanced regression as well as two desiderata, balanced accuracy and uncertainty estimation, for the problem.\n\n**(b)** As a byproduct, we also provide strong baselines for benchmarking high-quality uncertainty estimation and promising prediction performance on imbalanced datasets. \n\nWe will include the discussion above in the main paper of our revision to better highlight our originality as suggested. \n"}]

Author rebuttal:
We thank all reviewers for their encouraging and constructive comments. We are glad that they found the problems we identified ``"important"``/``"novel"`` (ivhk, 9XGJ, C6MR), our idea/method ``"novel"``/``"non-trivial"``/``"reasonable"`` (cFEB, C6MR, ivhk), our paper ``"easy to follow"``/``"clear"`` (9XGJ, C6MR), and that our method has ``"excellent"`` performance (9XGJ) and ``"superiority"`` (C6MR) over SOTA methods in terms of both ``"accuracy"`` and ``"uncertainty estimation"`` (C6MR, ivhk). Below we address the reviewers' questions one by one. 

Due to the space constraint, i.e., 6000 characters per reviewer, we cannot cover every question in the rebuttal, but we promise to address all questions, cite all related references, and **include all discussions/results below in our revision**.

**[cFEB, ivhk] Q1. The Number of Bins.**

Our preliminary results indicate that the performance of our VIR (as compared to the SOTA baseline, i.e., Ranksim [2]) remains consistent regardless of the number of bins, as shown in the included tables below (more results in **Table 2 of the one-page PDF file**). Here we report results for the cases with $100/1=100$, $100/3\approx 33$, and $100/5=20$ bins. 

Table 1: Ablation studies on the number of bins in terms of MSE

| model | bins | overall | many | median | few |
| :---------: | :------: | :------: | :------: | :------: | :------: |
| Ranksim | 100 | 83.51 | 71.99 | 99.14 | 149.05 |
| VIR (Ours) | 100 | **81.76** | **70.61** | **91.47** | **142.36** |
| Ranksim | 33 | 109.45 | 91.78 | 128.10 | 187.13 |
| VIR (Ours) | 33 | **84.77** | **77.29** | **95.66** | **125.33** |
| Ranksim | 20 | 98.71 | 84.38 | 107.89 | 171.04 |
| VIR (Ours) | 20 | **84.05** | **72.12** | **100.49** | **151.25** |
||||||

In our paper, we chose to use the same number of bins as the imbalanced regression literature [1, 2] for fair comparison with prior work. For example, in the AgeDB dataset where the regression labels are people's "age" in the range of 0~99, we use 100 bins, with each year as one bin. We will include the discussion and results above in the revised paper.

**[cFEB, ivhk] Q2: Advantage of Stochastic Latent Features / Dual Representation.**

That is a good question. Yes, stochastic latent features (i.e., dual representation) indeed improve the performance. We did conduct ablation studies in Table 8 and Table 9 of the Supplementary Material to verify this claim. In these tables, "Predictor-only VIR" is equivalent to our VIR without stochastic latent features and LDS. These results verify the effectiveness of such stochastic latent features.  

We have also conducted additional ablation studies to further support our claim. The results in Tables 2.1 and 2.2 below (more results in **Table 1 of the one-page PDF file**) demonstrate the importance of stochastic latent features (i.e., dual representation) in our proposed method. 

In the tables, **VIR w/o VAE** refers to VIR without stochastic latent features. These results verify that the stochastic latent features can indeed improve the performance.

Table 2.1: Ablation studies on AgeDB in terms of MSE

| model | overall | many | median | few |
| :---------: | :------: | :------: | :------: | :------: |
|VIR w/o VAE| 96.46 | 86.72 | 102.56 | 171.52 |
|VIR (Ours)| **81.76** | **70.61** | **91.47** | **142.36** |
|||||

Table 2.2: Ablation studies on AgeDB in terms of NLL

| model | overall | many | median | few |
| :---------: | :------: | :------: | :------: | :------: |
|VIR w/o VAE| 3.784 | 3.685 | 3.866 | 4.218 |
|VIR (Ours)| **3.703** | **3.598** | **3.805** | **4.196** |
|||||

**[ivhk, 9XGJ] Q3: Why Eqn. 5 Focuses on Minority Data (Line 42-43) and Alleviates the Imbalance Problem.**

We are sorry for the confusion. To see why, note that Eqn. 5 is the *negative log likelihood* for the Normal Inverse Gaussian (NIG) distribution. 

Specifically, each posterior parameter ($\nu_i^*, \gamma_i^*,  \alpha_i^*$) of the NIG distribution is reweighted by importance weights, thereby assigning higher weights to minority data during training and allowing minority data points to benefit more from their neighboring information.

Take $\nu_i^*$ as an example. Assume a minority data point $(x_i,y_i)$ that belongs to bin $b$, i.e., its label $y_i=y_b$. Note that there is **a loss term** $(y_i-\gamma_i^*)^2\nu_i^*$ in **Eqn. 5**, where $\gamma_i^*$ is the model prediction, $y_i$ is the label, and $\nu^{*}_{i}$ is the "importance weight" for this data point. 

Here $\nu_i^* = \nu_0 + (\sum_{b' \in \mathcal{B}} k (y_b, y_{b'}) p(y_{b'}))^{-1/2} \cdot n_i$ where $n_i$ represents the pseudo-count for the NIG distribution. Since $(x_i,y_i)$ is a minority data point, data from its neighboring bins has smaller frequency $p(y_{b'})$ and therefore smaller $\sum_{b' \in \mathcal{B}} k (y_{b}, y_{b'}) p(y_{b'})$, leading to **a larger "importance weight"** $\nu_i^*$ **for this minority data point in Eqn. 5**. 
This "allows the negative log likelihood (Eqn. 5) to naturally put more focus on the minority data" (Line 42-43), thereby alleviating the imbalance problem. 

**[ivhk] Q4: The Choice of the Kernel Functions.**

The DIR paper [1] shows that a simple Gaussian kernel with inverse square-root weighting (i.e., SQINV) achieves the best performance. We therefore use SQINV thoughout our paper for fair comparison (more details in Line 307 of the main paper). 

Besides, our preliminary results also show that the performance is not very sensitive to the choice of kernels, as long as the kernel $k(a,b)$ reflects the distance between $a$ and $b$, i.e., larger distance between $a$ and $b$ leads to smaller $k(a,b)$. 

[1] Delving into Deep Imbalanced Regression.

[2] RankSim: Ranking Similarity Regularization for Deep Imbalanced Regression.

Meta-review: The paper proposes a varaitional (Bayesian variant of) imbalanced regression, based on the neighboring and identically distributed assumption.   Experiments show sota results in terms of accuracy and uncertainty estimation.

Reviewers acknowledged strengths,

- Novel idea for an important and little-explored topic
- Assumption reasonable
- Good performance in both accuracy and uncertainty estimation, outperforming baselines.
- Well written,

while they raised several concerns,

- Ablation study missing? (actually in Appendix)
- Comparison with BNN, instead of DIR, missing.
- Clarity issues
- Subjective statements
- Code is not available.
- Limited novelty (combination of sota)

The authors' rebuttal addressed all reviewer's concerns, and all reviewers recommend acceptance.

Typos should be fixed.  For example, LHS and RHS don't match in the equation in Line 112.
Decision: Accept (poster)