Title: Variational Imbalanced Regression: Fair Uncertainty Quantification via Probabilistic Smoothing
Venue: NeurIPS 2023
Accepted: yes
Keywords: probabilistic methods

imbalanced regression

variational inference
Abstract: Existing regression models tend to fall short in both accuracy and uncertainty estimation when the label distribution is imbalanced. In this paper, we propose a probabilistic deep learning model, dubbed variational imbalanced regression (VIR), which not only performs well in imbalanced regression but naturally produces reasonable uncertainty estimation as a byproduct. 
Different from typical variational autoencoders assuming I.I.D. representations (a data point's representation is not directly affected by other data points), our VIR borrows data with similar regression labels to compute the latent representation's variational distribution; furthermore, different from deterministic regression models producing point estimates, VIR predicts the entire normal-inverse-gamma distributions and modulates the associated conjugate distributions to impose probabilistic reweighting on the imbalanced data, thereby providing better uncertainty estimation. Experiments in several real-world datasets show that our VIR can outperform state-of-the-art imbalanced regression models in terms of both accuracy and uncertainty estimation. Code will soon be available at https://github.com/Wang-ML-Lab/variational-imbalanced-regression.

Variational Imbalanced Regression: Fair Uncertainty Quantification via Probabilistic Smoothing

Abstract

Existing regression models tend to fall short in both accuracy and uncertainty estimation when the label distribution is imbalanced. In this paper, we propose a probabilistic deep learning model, dubbed variational imbalanced regression (VIR), which not only performs well in imbalanced regression but naturally produces reasonable uncertainty estimation as a byproduct. Different from typical variational autoencoders assuming I.I.D. representations (a data point's representation is not directly affected by other data points), our VIR borrows data with similar regression labels to compute the latent representation's variational distribution; furthermore, different from deterministic regression models producing point estimates, VIR predicts the entire normal-inverse-gamma distributions and modulates the associated conjugate distributions to impose probabilistic reweighting on the imbalanced data, thereby providing better uncertainty estimation. Experiments in several real-world datasets show that our VIR can outperform state-of-the-art imbalanced regression models in terms of both accuracy and uncertainty estimation. Code will soon be available at https: //github.com/Wang-ML-Lab/variational-imbalanced-regression .

1 Introduction

Deep regression models are currently the state of the art in making predictions in a continuous label space and have a wide range of successful applications in computer vision [50], natural language processing [22], healthcare [43, 45], recommender systems [16, 42], etc. However, these models fail however when the label distribution in training data is imbalanced. For example, in visual age estimation [30], where a model infers the age of a person given her visual appearance, models are typically trained on imbalanced datasets with overwhelmingly more images of younger adults, leading to poor regression accuracy for images of children or elderly people [48, 49]. Such unreliability in imbalanced regression settings motivates the need for both improving performance for the minority in the presence of imbalanced data and, more importantly, providing reasonable uncertainty estimation to inform practitioners on how reliable the predictions are (especially for the minority where accuracy is lower).

Existing methods for deep imbalanced regression (DIR) only focus on improving the accuracy of deep regression models by smoothing the label distribution and reweighting data with different labels [48, 49]. On the other hand, methods that provide uncertainty estimation for deep regression models operates under the balance-data assumption and therefore do not work well in the imbalanced setting [1, 8, 29].

To simultaneously cover these two desiderata, we propose a probabilistic deep imbalanced regression model, dubbed variational imbalanced regression (VIR). Different from typical variational autoencoders assuming I.I.D. representations (a data point's representation is not directly affected by other data points), our VIR assumes Neighboring and Identically Distributed (N.I.D.) and borrows

Hao Wang

data with similar regression labels to compute the latent representation's variational distribution. Specifically, VIR first encodes a data point into a probabilistic representation and then mix it with neighboring representations (i.e., representations from data with similar regression labels) to produce its final probabilistic representation; VIR is therefore particularly useful for minority data as it can borrow probabilistic representations from data with similar labels (and naturally weigh them using our probabilistic model) to counteract data sparsity. Furthermore, different from deterministic regression models producing point estimates, VIR predicts the entire Normal Inverse Gamma (NIG) distributions and modulates the associated conjugate distributions by the importance weight computed from the smoothed label distribution to impose probabilistic reweighting on the imbalanced data. This allows the negative log likelihood to naturally put more focus on the minority data, thereby balancing the accuracy for data with different regression labels. Our VIR framework is compatible with any deep regression models and can be trained end to end.

We summarize our contributions as below:

We identify the problem of probabilistic deep imbalanced regression as well as two desiderata, balanced accuracy and uncertainty estimation, for the problem.

We propose VIR to simultaneously cover these two desiderata and achieve state-of-the-art performance compared to existing methods.

As a byproduct, we also provide strong baselines for benchmarking high-quality uncertainty estimation and promising prediction performance on imbalanced datasets.

2 Related Work

Variational Autoencoder. Variational autoencoder (VAE) [25] is an unsupervised learning model that aims to infer probabilistic representations from data. However, as shown in Figure 1, V AE typically assumes I.I.D. representations, where a data point's representation is not directly affected by other data points. In contrast, our VIR borrows data with similar regression labels to compute the latent representation's variational distribution.

Imbalanced Regression. Imbalanced regression is under-explored in the machine learning community. Most existing methods for imbalanced regression are direct extensions of the SMOTE algorithm [9], a commonly

y

100

0

VAE

VIR (Ours)

...

𝒙𝟏

𝒙𝟑

𝒙𝟒

𝒙𝟓

𝒙𝑵

𝒛𝟏

𝒛𝟐

𝒛𝟑

𝒛𝟒

𝒛𝟓

𝒛𝑵

...

𝒙𝟏

𝒙𝟑

𝒙𝟒

𝒙𝟓

𝒛𝟏

𝒛𝟐

𝒛𝟑

𝒛𝟒

𝒛𝟓

𝒙𝑵

𝒛𝑵

used algorithm for imbalanced classification, where data from the minority classes is over-sampled. These algorithms usually synthesize augmented data for the minority regression labels by either interpolating both inputs and labels [40] or adding Gaussian noise [5, 6] (more discussion on augmentation-based methods in the Appendix).

Such algorithms fail to measure the distance in continuous label space and fall short in handling high-dimensional data (e.g., images and text). Recently, DIR [49] addresses these issues by applying kernel density estimation to smooth and reweight data on the continuous label distribution, achieving state-of-the-art performance. However, DIR only focuses on improving the accuracy, especially for the data with minority labels, and therefore does not provide uncertainty estimation, which is crucial to assess the predictions' reliability. [32] focuses on re-balancing the mean squared error (MSE) loss for imbalanced regression, and [13] introduces ranking similarity for improving deep imbalanced regression. In contrast, our VIR provides a principled probabilistic approach to simultaneously achieve these two desiderata, not only improving upon DIR in terms of performance but also producing reasonable uncertainty estimation as a much-needed byproduct to assess model reliability. There is also related work on imbalanced classification [11], which is related to our work but focusing on classification rather than regression.

Uncertainty Estimation in Regression. There has been renewed interest in uncertainty estimation in the context of deep regression models [1, 12, 18, 23, 26, 29, 36-38, 51]. Most existing methods directly predict the variance of the output distribution as the estimated uncertainty [1, 23, 52], rely on post-hoc confidence interval calibration [26, 37, 51], or using Bayesian neural networks [44, 46, 47]; there are also training-free approaches, such as Infer Noise and Infer Dropout [29], which produce multiple predictions from different perturbed neurons and compute their variance as uncertainty estimation. Closest to our work is Deep Evidential Regression (DER) [1], which attempts to estimate both aleatoric and epistemic uncertainty [20, 23] on regression tasks by training the neural networks to directly infer the parameters of the evidential distribution, thereby producing uncertainty measures. DER [1] is designed for the data-rich regime and therefore fails to reasonably estimate the uncertainty if the data is imbalanced; for data with minority labels, DER [1] tends produce unstable distribution parameters, leading to poor uncertainty estimation (as shown in Sec. 5). In contrast, our proposed VIR explicitly handles data imbalance in the continuous label space to avoid such instability; VIR does so by modulating both the representations and the output conjugate distribution parameters according to the imbalanced label distribution, allowing training/inference to proceed as if the data is balance and leading to better performance as well as uncertainty estimation (as shown in Sec. 5).

3 Method

In this section we introduce the notation and problem setting, provide an overview of our VIR, and then describe details on each of VIR's key components.

3.1 Notation and Problem Setting

Assuming an imbalanced dataset in continuous space { x i , y i } N i =1 where N is the total number of data points, x i ∈ R d is the input, and y i ∈ Y ⊂ R is the corresponding label from a continuous label space Y . In practice, Y is partitioned into B equal-interval bins [ y (0) , y (1) ) , [ y (1) , y (2) ) , ..., [ y ( B -1) , y ( B ) ) , with slight notation overload. To directly compare with baselines, we use the same grouping index for target value b ∈ B as in [49].

We denote representations as z i , and use ( ˜ z µ i , ˜ z Σ i ) = q ϕ ( z | x i ) to denote the probabilistic representations for input x i generated by a probabilistic encoder parameterized by ϕ ; furthermore, we denote ¯ z as the mean of representation z i in each bin, i.e., ¯ z = 1 N b ∑ N b i =1 z i for a bin with N b data points. Similarly we use ( ̂ y i , ̂ s i ) to denote the mean and variance of the predictive distribution generated by a probabilistic predictor p θ ( y i | z ) .

3.2 Method Overview

In order to achieve both desiderata in probabilistic deep imbalanced regression (i.e., performance improvement and uncertainty estimation),

𝑁

𝒙

𝒛

𝑦

𝜃

𝑁

𝒙

𝒛

𝜙

𝑁

our proposed variational imbalanced regression (VIR) operates on both the encoder q ϕ ( z i |{ x i } N i =1 ) and the predictor p θ ( y i | z i ) .

Typical VAE [25] lower-bounds input x i 's marginal likelihood; in contrast, VIR lower-bounds the marginal likelihood of input x i and labels y i :

Note that our variational distribution q ϕ ( z i |{ x i } N i =1 ) (1) does not condition on labels y i , since the task is to predict y i and (2) conditions on all (neighboring) inputs { x i } N i =1 rather than just x i . The second term L ( θ, ϕ ; x i , y i ) is VIR's evidence lower bound (ELBO), which is defined as:

where the p θ ( z i ) is the standard Gaussian prior N ( 0 , I ) , following typical VAE [25], and the expectation is taken over q ϕ ( z i |{ x i } N i =1 ) , which infers z i by borrowing data with similar regression labels to produce the balanced probabilistic representations, which is beneficial especially for the minority (see Sec. 3.3 for details).

Different from typical regression models which produce only point estimates for y i , our VIR's predictor, p θ ( y i | z i ) , directly produces the parameters of the entire NIG distribution for y i and further imposes probabilistic reweighting on the imbalanced data, thereby producing balanced predictive distributions (more details in Sec. 3.4).

3.3 Constructing q ( z i |{ x i } N i =1 )

To cover both desiderata, one needs to (1) produce balanced representations to improve performance for the data with minority labels and (2) produce probabilistic representations to naturally obtain reasonable uncertainty estimation for each model prediction. To learn such balanced probabilistic representations , we construct the encoder of our VIR (i.e., q ϕ ( z i |{ x i } N i =1 ) ) by (1) first encoding a data point into a probabilistic representation , (2) computing probabilistic statistics from neighboring representations (i.e., representations from data with similar regression labels), and (3) producing the final representations via probabilistic whitening and recoloring using the obtained statistics.

Intuition on Using Probabilistic Representation. DIR uses deterministic representations, with one vector as the final representation for each data point. In contrast, our VIR uses probabilistic representations, with one vector as the mean of the representation and another vector as the variance of the representation. Such dual representation is more robust to noise and therefore leads to better prediction performance. Therefore, We first encode each data point into a probabilistic representation. Note that this is in contrast to existing work [49] that uses deterministic representations. We assume that each encoding z i is a Gaussian distribution with parameters { z µ i , z Σ i } , which are generated from the last layer in the deep neural network.

From I.I.D. to Neighboring and Identically Distributed (N.I.D.). Typical VAE [25] is an unsupervised learning model that aims to learn a variational representation from latent space to reconstruct the original inputs under the I.I.D. assumption; that is, in V AE, the latent value (i.e., z i ) is generated from its own input x i . This I.I.D. assumption works well for data with majority labels, but significantly harms performance for data with minority labels. To address this problem, we replace the I.I.D. assumption with the N.I.D. assumption; specifically, VIR's variational latent representations still follow Gaussian distributions (i.e., N ( z µ i , z Σ i ), but these distributions will be first calibrated using data with neighboring labels. For a data point ( x i , y i ) where y i is in the b 'th bin, i.e., y i ∈ [ y ( b -1) , y ( b ) ) , we compute q ( z i |{ x i } N i =1 ) ≜ N ( z i ; ˜ z µ i , ˜ z Σ i ) with the following four steps.

˜

˜

˜

˜

where the details of functions I ( · ) , A ( · ) , S ( · ) , and F ( · ) are described below.

(1) Function I ( · ) : From Deterministic to Probabilistic Statistics. Different from deterministic statistics in [49], our VIR's encoder uses probabilistic statistics , i.e., statistics of statistics . Specifically, VIR treats z i as a distribution with the mean and covariance ( z µ i , z Σ i ) = I ( x i ) rather than a deterministic vector.

As a result, all the deterministic statistics for bin b , µ b , Σ b , ˜ µ b , and ˜ Σ b are replaced by distributions with the means and covariances, ( µ µ b , µ Σ b ) , ( Σ µ b , Σ Σ b ) , ( ˜ µ µ b , ˜ µ Σ b ) , and ( ˜ Σ µ b , ˜ Σ Σ b ) , respectively (more details in the following three paragraphs on A ( · ) , S ( · ) , and F ( · ) ).

(2) Function A ( · ) : Statistics of the Current Bin b 's Statistics. In VIR, the deterministic overall mean for bin b (with N b data points), µ b = ¯ z = 1 N b ∑ N b i =1 z i , becomes the probabilistic overall mean , i.e., a distribution of µ b with the mean µ µ b and covariance µ Σ b (assuming diagonal covariance) as

follows:

Similarly, the deterministic overall covariance for bin b , Σ b = 1 N b ∑ N b i =1 ( z i -¯ z ) 2 , becomes the probabilistic overall covariance , i.e., a matrix-variate distribution [15] with the mean:

since E [¯ z ] = µ µ b and V [¯ z ] = µ Σ b . Note that the covariance of Σ b , i.e., Σ Σ b ≜ V [ Σ b ] , involves computing the fourth-order moments, which is computationally prohibitive. Therefore in practice, we directly set Σ Σ b to zero for simplicity; empirically we observe that such simplified treatment already achieves promising performance improvement upon the state of the art. More discussions on the idea of the hierarchical structure of the statistics of statistics for smoothing are in the Appendix.

(3) Function S ( · ) : Neighboring Data and Smoothed Statistics. Next, we can borrow data from neighboring label bins b ′ to compute the smoothed statistics of the current bin b by applying a symmetric kernel k ( · , · ) (e.g., Gaussian, Laplacian, and Triangular kernels). Specifically, the probabilistic smoothed mean and covariance are (assuming diagonal covariance):

(4) Function F ( · ) : Probabilistic Whitening and Recoloring. We develop a probabilistic version of the whitening and re-coloring procedure in [39, 49]. Specifically, we produce the final probabilistic representation { ˜ z µ i , ˜ z Σ i } for each data point as:

During training, we keep updating the probabilistic overall statistics, { µ µ b , µ Σ b , Σ µ b } , and the probabilistic smoothed statistics, { ˜ µ µ b , ˜ µ Σ b , ˜ Σ µ b } , across different epochs. The probabilistic representation { ˜ z µ i , ˜ z Σ i } are then re-parameterized [25] into the final representation z i , and passed into the final layer (discussed in Sec. 3.4) to generate the prediction and uncertainty estimation. Note that the computation of statistics from multiple x 's is only needed during training. During testing, VIR directly uses these statistics and therefore does not need to re-compute them.

3.4 Constructing p ( y i | z i )

Our VIR's predictor p ( y i | z i ) ≜ N ( y i ; ̂ y i , ̂ s i ) predicts both the mean and variance for y i by first predicting the NIG distribution and then marginalizing out the latent variables. It is motivated by the following observations on label distribution smoothing (LDS) in [49] and deep evidental regression (DER) in [1], as well as intuitions on effective counts in conjugate distributions.

LDS's Limitations in Our Probabilistic Imbalanced Regression Setting. The motivation of LDS [49] is that the empirical label distribution can not reflect the real label distribution in an imbalanced dataset with a continuous label space; consequently, reweighting methods for imbalanced regression fail due to these inaccurate label densities. By applying a smoothing kernel on the empirical label distribution, LDS tries to recover the effective label distribution, with which reweighting methods can obtain 'better' weights to improve imbalanced regression. However, in our probabilistic imbalanced regression, one needs to consider both (1) prediction accuracy for the data with minority labels and (2) uncertainty estimation for each model. Unfortunately, LDS only focuses on improving the accuracy, especially for the data with minority labels, and therefore does not provide uncertainty estimation, which is crucial to assess the predictions' reliability.

DER's Limitations in Our Probabilistic Imbalanced Regression Setting. In DER [1], the predicted labels with their corresponding uncertainties are represented by the approximate posterior parameters ( γ, ν, α, β ) of the NIG distribution NIG ( γ, ν, α, β ) . A DER model is trained via minimizing the negative log-likelihood (NLL) of a Student-t distribution:

where Ω = 2 β (1 + ν ) . It is therefore nontrivial to properly incorporate a reweighting mechanism into the NLL. One straightforward approach is to directly reweight L DER i for different data points ( x i , y i ) . However, this contradicts the formulation of NIG and often leads to poor performance, as we verify in Sec. 5.

Intuition of Pseudo-Counts for VIR. To properly incorporate different reweighting methods, our VIR relies on the intuition of pseudo-counts (pseudo-observations) in conjugate distributions [4]. Assuming Gaussian likelihood, the conjugate distributions would be an NIG distribution [4], i.e., ( µ, Σ) ∼ NIG ( γ, ν, α, β ) , which means:

where Γ -1 ( α, β ) is an inverse gamma distribution. With an NIG prior distribution NIG ( γ 0 , ν 0 , α 0 , β 0 ) , the posterior distribution of the NIG after observing n real data points { u i } n i =1 are:

where Ψ = ¯ u and Φ = 1 2 ( ∑ i u 2 i -γ 2 n ν n ) . Here ν 0 and α 0 can be interpreted as virtual observations, i.e., pseudo-counts or pseudo-observations that contribute to the posterior distribution. Overall, the mean of posterior distribution above can be interpreted as an estimation from (2 α 0 + n ) observations, with 2 α 0 virtual observations and n real observations. Similarly, the variance can be interpreted an estimation from ( ν 0 + n ) observations. This intuition is crucial in developing our VIR's predictor.

From Pseudo-Counts to Balanced Predictive Distributions. Based on the intuition above, we construct our predictor (i.e., p ( y i | z i ) ) by (1) generating the parameters in the posterior distribution of NIG, (2) computing re-weighted parameters by imposing the importance weights obtained from LDS, and (3) producing the final prediction with corresponding uncertainty estimation.

Based on Eqn. 4, we feed the final representation { z i } N i =1 generated from the Sec. 3.3 (Eqn. 2) into a linear layer to output the intermediate parameters n i , Ψ i , Φ i for data point ( x i , y i ) :

We then apply the importance weights ( ∑ b ′ ∈B k ( y b , y b ′ ) p ( y b ′ ) ) -1 / 2 calculated from the smoothed label distribution to the pseudo-count n i to produce the re-weighted parameters of posterior distribution of NIG, where p ( y ) denotes the marginal distribution of y . Along with the pre-defined prior parameters ( γ 0 , ν 0 , α 0 , β 0 ) , we are able to compute the parameters of posterior distribution NIG ( γ i , ν i , α i , β i ) for ( x i , y i ) :

Based on the NIG posterior distribution, we can then compute final prediction and uncertainty estimation as

We use an objective function similar to Eqn. 3, but with different definitions of ( γ, ν, α, β ) , to optimize our VIR model:

where ω ∗ i = 2 β ∗ i (1 + ν ∗ i ) . Note that L P i is part of the ELBO in Eqn. 1. Similar to [1], we use an additional regularization term to achieve better accuracy :

L P i and L R i together constitute the objective function for learning the predictor p ( y i | z i ) .

3.5 Final Objective Function

Putting together Sec. 3.3 and Sec. 3.4, our final objective function (to minimize) for VIR is:

where L ( θ, ϕ ; x i , y i ) = L P i + L D i -L KL i is the ELBO in Eqn. 1. λ adjusts the importance of the additional regularizer and the ELBO, and thus lead to a better result both on accuracy and uncertainty estimation.

4 Theory

Notation. As mentioned in Sec. 3.1, we partitioned { Y j } N j =1 into |B| equal-interval bins (denote the set of bins as B ), and { Y j } N j =1 are sampled from the label space Y . In addition, We use the binary set { P i } |B| i =1 to represent the label distribution (frequency) for each bin i , i.e., P i ≜ P ( Y j ∈ B i ) . We also use the binary set { O i } N j =1 to represent whether the data point ( x j , y j ) is observed (i.e., O j ∼ P ( O j = 1) ∝ P B ( Y j ) , and E O [ O j ] ∼ P B ( Y j ) ), where B ( Y j ) represents the bin which ( x j , y j ) belongs to. For each bin i ∈ B , we denote the associated set of data points as

When the imbalanced dataset is partially observed, we denote the observation set as:

Definition 4.1 ( Expectation over Observability E O ) . We define the expectation over the observability variable O as E O [ · ] ≡ E O j ∼ P ( O j =1) [ · ] .

Definition 4.2 ( True Risk ) . Based on the previous definitions, the true risk is defined as:

where δ j ( Y, ̂ Y ) refers to some loss function (e.g. MAE, MSE). In this paper, we assume the loss is upper bounded by ∆ , i.e., 0 ≤ δ j ( Y, ̂ Y ) ≤ ∆ .

Below we define the Naive Estimator.

Definition 4.3 ( Naive Estimator ) . Given the observation set, the Naive Estimator is defined as:

̸

It is easy to verify that the expectation of this naive estimator is not equal to the true risk, as E O [ ̂ R NAIVE ( ̂ Y )] = R ( ̂ Y ) .

Considering an imbalanced dataset as a subset of observations from a balanced one, we contrast it with the Inverse Propensity Score (IPS) estimator [34].

Definition 4.4 ( Inverse Propensity Score Estimator ) . The inverse propensity score (IPS) estimator (an unbiased estimator) is defined as

The IPS estimator is an unbiased estimator, as we can verify by taking the expectation value over the observation set:

Finally, we define our VIR/DIR estimator below.

Definition 4.5 ( VIR Estimator ) . The VIR estimator, denoted by ̂ R VIR ( ̂ Y | ˜ P ) , is defined as:

where { ˜ P i } |B| i =1 represents the smoothed label distribution used in our VIR's objective function. It is important to note that our VIR estimator is biased.

For multiple predictions, we select the 'best' estimator according to the following definition.

Definition 4.6 ( Empirical Risk Minimizer ) . For a given hypothesis space H of predictions ̂ Y , the Empirical Risk Minimization (ERM) identifies the prediction ̂ Y ∈ H as

With all the aforementioned definitions, we can derive the generalization bound for the VIR estimator.

Theorem 4.1 (Generalization Bound of VIR) . In imbalanced regression with bins B , for any finite hypothesis space of predictions H = { ̂ Y 1 , . . . , ̂ Y H } , the transductive prediction error of the empirical risk minimizer ̂ Y ERM using the VIR estimator with estimated propensities ˜ P ( P i > 0 ) and given training observations O from Y with independent Bernoulli propensities P , is bounded by:

̸

Remark. The naive estimator (i.e., Definition 4.3) has large bias and large variance. If one directly uses the original label distribution in the training objective (i.e., Definition 4.4), i.e., ˜ P i = P i , the 'bias' term will be 0 . However, the 'variance' term will be extremely large for minority data because ˜ P i is very close to 0 . In contrast, under VIR's N.I.D., ˜ P i used in the training objective function will be smoothed. Therefore, the minority data's label density ˜ P i will be smoothed out by its neighbors and becomes larger (compared to the original P i ), leading to smaller 'variance' in the generalization error bound. Note that ˜ P i = P i , VIR (with N.I.D.) essentially increases bias, but significantly reduces its variance in the imbalanced setting, thereby leading to a lower generalization error.

5 Experiments

Datasets. We evaluate our methods in terms of prediction accuracy and uncertainty estimation on four imbalanced datasets 1 , AgeDB-DIR [30], IMDB-WIKI-DIR [33], STS-B-DIR [7], and NYUD2DIR [35]. Due to page limit, results for NYUD2-DIR [35] are moved to the Appendix. We follow the preprocessing procedures in DIR [49]. Details for each datasets are in the Appendix, and please refer to [49] for details on label density distributions and levels of imbalance.

Baselines. We use ResNet-50 [17] (for AgeDB-DIR and IMDB-WIKI-DIR) and BiLSTM [19] (for STS-B-DIR) as our backbone networks, and more details for baseline are in the Appendix. we describe the baselines below.

Vanilla : We use the term VANILLA to denote a plain model without adding any approaches.

Synthetic-Sample-Based Methods : Various existing imbalanced regression methods are also included as baselines; these include Deep Ensemble [27], Infer Noise [29], SMOTER [40], and SMOGN [5].

Cost-Sensitive Reweighting : As shown in DIR [49], the square-root weighting variant (SQINV) baseline (i.e., ( ∑ b ′ ∈B k ( y b , y b ′ ) p ( y b ′ ) ) -1 / 2 ) always outperforms Vanilla. Therefore, for fair comparison, all our experiments (for both baselines and VIR) use SQINV weighting.

Evaluation Metrics. Wefollow the evaluation metrics in [49] to evaluate the accuracy of our proposed methods; these include Mean Absolute Error (MAE), Mean Squared Error (MSE). Furthermore, for AgeDB-DIR and IMDB-WIKI-DIR, we use Geometric Mean (GM) to evaluate the accuracy; for STS-B-DIR, we use Pearson correlation and Spearman correlation. We use typical evaluation metrics for uncertainty estimation in regression problems to evaluate our produced uncertainty estimation; these include Negative Log Likelihood (NLL), Area Under Sparsification Error (AUSE). Eqn. 5 shows the formula for NLL, and more details regarding to AUSE can be found in [21]. We also include calibrated uncertainty results for VIR in the Appendix.

Evaluation Process. Following [28, 49], for a data sample x i with its label y i which falls into the target bins b i , we divide the label space into three disjoint subsets: many-shot region { b i ∈ B | y i ∈ b i & | y i | > 100 } , medium-shot region { b i ∈ B | y i ∈ b i & 20 ≤ | y i | ≤ 100 } , and fewshot region { b i ∈ B | y i ∈ b i & | y i | < 20 } , where | · | denotes the cardinality of the set. We report results on the overall test set and these subsets with the accuracy metrics and uncertainty metrics discussed above.

Implementation Details. We conducted five separate trials for our method using different random seeds. The error bars and other implementa-

tion details are included in the Appendix.

5.1 Imbalanced Regression Accuracy

We report the accuracy of different methods in Table 1, Table 2, and Table 3 for AgeDB-DIR, IMDBWIKI-DIR and STS-B-DIR, respectively. In all the tables, we can observe that our VIR consistently outperforms all baselines in all metrics.

As shown in the last four rows of all three tables, our proposed VIR compares favorably against strong baselines including DIR variants [49] and DER [1], Infer Noise [29], and RankSim [13], especially on the imbalanced data samples (i.e., in the few-shot columns). Notably, VIR improves

Metrics

NLL

↓

AUSE

↓

Shot

All

Many

Med

Few

All

Many

Med

Few

DEEP ENS. [27]

5.311

4.031

6.726

8.523

0.541

0.626

0.466

0.483

INFER NOISE [29]

4.616

4.413

4.866

5.842

0.465

0.458

0.457

0.496

DER [1]

3.918

3.741

3.919

4.432

0.523

0.464

0.449

0.486

LDS + FDS + DER [1]

3.787

3.689

3.912

4.234

0.451

0.460

0.399

0.565

VIR (OURS)

3.703

3.598

3.805

4.196

0.434

0.456

0.324

0.414

OURS

VS. DER

+0.215

+0.143

+0.114

+0.236

+0.089

+0.008

+0.125

+0.072

OURS

VS. LDS + FDS + DER

+0.084

+0.091

+0.107

+0.038

+0.017

+0.004

+0.075

+0.151

upon the state-of-the-art method RankSim by 9 . 6% and 7 . 9% on AgeDB-DIR and IMDB-WIKIDIR, respectively, in terms of few-shot GM. This verifies the effectiveness of our methods in terms of overall performance. More accuracy results on different metrics are included in the Appendix. Besides the main results, we also include ablation studies for VIR in the Appendix, showing the effectiveness of VIR's encoder and predictor.

5.2 Imbalanced Regression Uncertainty Estimation

Different from DIR [49] which only focuses on accuracy, we create a new benchmark for uncertainty estimation in imbalanced regression. Table 4, Table 5, and Table 6 show the results on uncertainty estimation for three datasets AgeDB-DIR, IMDB-WIKI-DIR, and STS-B-DIR, respectively. Note that most baselines from Table 1, Table 2, and Table 3 are deterministic methods (as opposed to probabilistic methods like ours) and cannot provide uncertainty estimation ; therefore they are not applicable here. To show the superiority of our VIR model, we create a strongest baseline by concatenating the DIR variants (LDS + FDS) with the DER [1].

Results show that our VIR consistently outperforms all baselines across different metrics, especially in the few-shot metrics. Note that our proposed methods mainly focus on the imbalanced setting, and therefore naturally places more emphasis on the few-shot metrics. Notably, on AgeDB-DIR, IMDBWIKI-DIR, and STS-B-DIR, our VIR improves upon the strongest baselines, by 14 . 2% ∼ 17 . 1% in terms of few-shot AUSE.

5.3 Limitations

Although our methods successfully improve both accuracy and uncertainty estimation on imbalanced regression, there are still several limitations. Exactly computing variance of the variances in Sec. 3.3 is challenging; we therefore resort to fixed variance as an approximation. Developing more accurate and efficient ap-

proximations would also be interesting future work.

6 Conclusion

We identify the problem of probabilistic deep imbalanced regression, which aims to both improve accuracy and obtain reasonable uncertainty estimation in imbalanced regression. We propose VIR, which can use any deep regression models as backbone networks. VIR borrows data with similar regression labels to produce the probabilistic representations and modulates the conjugate distributions to impose probabilistic reweighting on imbalanced data. Furthermore, we create new benchmarks with strong baselines for uncertainty estimation on imbalanced regression. Experiments show that our methods outperform state-of-the-art imbalanced regression models in terms of both accuracy and uncertainty estimation. Future work may include (1) improving VIR by better approximating variance of the variances in probability distributions, and (2) developing novel approaches that can achieve stable performance even on imbalanced data with limited sample size, and (3) exploring techniques such as mixture density networks [3] to enable multi-modality in the latent distribution, thereby further improving the performance.

References

A. Amini, W. Schwarting, A. Soleimany, and D. Rus. Deep evidential regression. Advances in Neural Information Processing Systems , 33:14927-14937, 2020.

J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of machine learning research , 13(2), 2012.

C. M. Bishop. Mixture density networks. 1994.

C. M. Bishop. Pattern recognition and machine learning . springer, 2006.

P. Branco, L. Torgo, and R. P. Ribeiro. Smogn: a pre-processing approach for imbalanced regression. In First international workshop on learning with imbalanced domains: Theory and applications , pages 36-50. PMLR, 2017.

P. Branco, L. Torgo, and R. P. Ribeiro. Rebagg: Resampled bagging for imbalanced regression. In Second International Workshop on Learning with Imbalanced Domains: Theory and Applications , pages 67-81. PMLR, 2018.

D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation , pages 1-14, 2017.

B. Charpentier, O. Borchert, D. Zügner, S. Geisler, and S. Günnemann. Natural posterior network: Deep bayesian predictive uncertainty for exponential family distributions. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research , 16:321-357, 2002.

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Z. Deng, H. Liu, Y. Wang, C. Wang, Z. Yu, and X. Sun. PML: progressive margin loss for long-tailed age classification. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , 2021.

Y. Gal and Z. Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016 , volume 48 of JMLRWorkshop and Conference Proceedings , pages 1050-1059. JMLR.org, 2016.

Y. Gong, G. Mori, and F. Tung. Ranksim: Ranking similarity regularization for deep imbalanced regression. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 7634-7649. PMLR, 2022.

C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

A. K. Gupta and D. K. Nagar. Matrix variate distributions , volume 104. CRC Press, 2018.

S. Gupta, H. Wang, Z. Lipton, and Y. Wang. Correcting exposure bias for link recommendation. In ICML , 2021.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 770-778. IEEE Computer Society, 2016.

J. Heiss, J. Weissteiner, H. Wutte, S. Seuken, and J. Teichmann. Nomu: Neural optimizationbased model uncertainty. arXiv preprint arXiv:2102.13640 , 2021.

Z. Huang, W. Xu, and K. Yu. Bidirectional LSTM-CRF models for sequence tagging. CoRR , 2015.

E. Hüllermeier and W. Waegeman. Aleatoric and epistemic uncertainty in machine learning: A tutorial introduction. CoRR , abs/1910.09457, 2019.

E. Ilg, Ö. c Ci c cek, S. Galesso, A. Klein, O. Makansi, F. Hutter, and T. Brox. Uncertainty estimates and multi-hypotheses networks for optical flow. In Computer Vision - ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VII , volume 11211 of Lecture Notes in Computer Science , pages 677-693. Springer, 2018.

H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and T. Zhao. SMART: robust and efficient finetuning for pre-trained natural language models through principled regularized optimization. In D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020 , pages 2177-2190. Association for Computational Linguistics, 2020.

A. Kendall and Y. Gal. What uncertainties do we need in bayesian deep learning for computer vision? arXiv preprint arXiv:1703.04977 , 2017.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

D. P. Kingma and M. Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014.

V. Kuleshov, N. Fenner, and S. Ermon. Accurate uncertainties for deep learning using calibrated regression. In International Conference on Machine Learning , pages 2796-2804. PMLR, 2018.

B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 6402-6413, 2017.

Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu. Large-scale long-tailed recognition in an open world. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019 , pages 2537-2546. Computer Vision Foundation / IEEE, 2019.

L. Mi, H. Wang, Y. Tian, and N. Shavit. Training-free uncertainty estimation for neural networks. In AAAI , 2022.

S. Moschoglou, A. Papaioannou, C. Sagonas, J. Deng, I. Kotsia, and S. Zafeiriou. Agedb: The first manually collected, in-the-wild age database. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2017, Honolulu, HI, USA, July 21-26, 2017 , pages 1997-2005, 2017.

J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pages 1532-1543, 2014.

J. Ren, M. Zhang, C. Yu, and Z. Liu. Balanced mse for imbalanced visual regression. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 7916-7925. IEEE, 2022.

R. Rothe, R. Timofte, and L. V. Gool. Deep expectation of real and apparent age from a single image without facial landmarks. Int. J. Comput. Vis. , 126(2-4):144-157, 2018.

T. Schnabel, A. Swaminathan, A. Singh, N. Chandak, and T. Joachims. Recommendations as treatments: Debiasing learning and evaluation. In international conference on machine learning , pages 1670-1679. PMLR, 2016.

N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor segmentation and support inference from rgbd images. ECCV (5) , 7576:746-760, 2012.

J. Snoek, Y. Ovadia, E. Fertig, B. Lakshminarayanan, S. Nowozin, D. Sculley, J. V. Dillon, J. Ren, and Z. Nado. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 13969-13980, 2019.

H. Song, T. Diethe, M. Kull, and P. Flach. Distribution calibration for regression. In International Conference on Machine Learning , pages 5897-5906. PMLR, 2019.

M. Stadler, B. Charpentier, S. Geisler, D. Zügner, and S. Günnemann. Graph posterior network: Bayesian predictive uncertainty for node classification. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 18033-18048, 2021.

B. Sun, J. Feng, and K. Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA , pages 2058-2065. AAAI Press, 2016.

L. Torgo, R. P. Ribeiro, B. Pfahringer, and P. Branco. SMOTE for regression. In L. Correia, L. P. Reis, and J. Cascalho, editors, Progress in Artificial Intelligence - 16th Portuguese Conference on Artificial Intelligence, EPIA 2013, Angra do Heroísmo, Azores, Portugal, September 912, 2013. Proceedings , volume 8154 of Lecture Notes in Computer Science , pages 378-389. Springer, 2013.

A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. EMNLP 2018 , page 353, 2018.

H. Wang, Y. Ma, H. Ding, and Y. Wang. Context uncertainty in contextual bandits with applications to recommender systems. In AAAI , 2022.

H. Wang, C. Mao, H. He, M. Zhao, T. S. Jaakkola, and D. Katabi. Bidirectional inference networks: A class of deep bayesian networks for health profiling. In AAAI , volume 33, pages 766-773, 2019.

H. Wang, S. Xingjian, and D.-Y. Yeung. Natural-parameter networks: A class of probabilistic neural networks. In NIPS , pages 118-126, 2016.

H. Wang and J. Yan. Self-interpretable time series prediction with counterfactual explanations. In ICML , 2023.

H. Wang and D.-Y. Yeung. Towards bayesian deep learning: A framework and some existing methods. TDKE , 28(12):3395-3408, 2016.

H. Wang and D.-Y. Yeung. A survey on bayesian deep learning. CSUR , 53(5):1-37, 2020.

Y. Yang, H. Wang, and D. Katabi. On multi-domain long-tailed recognition, generalization and beyond. In ECCV , 2022.

Y. Yang, K. Zha, Y. Chen, H. Wang, and D. Katabi. Delving into deep imbalanced regression. In International Conference on Machine Learning , pages 11842-11851. PMLR, 2021.

W. Yin, J. Zhang, O. Wang, S. Niklaus, L. Mai, S. Chen, and C. Shen. Learning to recover 3d scene shape from a single image. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , pages 204-213. Computer Vision Foundation / IEEE, 2021.

E. Zelikman, C. Healy, S. Zhou, and A. Avati. Crude: calibrating regression uncertainty distributions empirically. arXiv preprint arXiv:2005.12496 , 2020.

Z. Zhang, A. Romero, M. J. Muckley, P. Vincent, L. Yang, and M. Drozdzal. Reducing uncertainty in undersampled mri reconstruction with active acquisition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2049-2058, 2019.

A Theory

Notation. As defined in Sec.3 in main paper, we partitioned { Y j } N j =1 into |B| equal-interval bins (denote the set of bins as B ), and { Y j } N j =1 are sampled from the label space Y . In addition, We denote the binary set { P i } |B| i =1 as the label distribution (frequency) for each bin, i.e., P i ≜ P ( Y j ∈ B i ) . We also denote the binary set { O i } N j =1 to represent whether the data { x j , y j } are observed (i.e., O j ∼ P ( O j = 1) ∝ P B ( Y j ) , and E O [ O j ] ∼ P B ( Y j ) ), where B ( Y j ) represents the bin which ( x j , y j ) belongs to. For each bin i ∈ B , we denote the global set of samples as

When the imbalanced dataset is partially observed, we denote the observation set as:

Definition A.1 ( Expectation over Observability E O ) . We define the expectation over the observability variable O as E O [ · ] ≡ E O j ∼ P ( O j =1) [ · ]

Definition A.2 ( True Risk ) . Based on the previous definitions, the true risk for our model is defined as:

where δ j ( Y, ̂ Y ) refers to some loss function (e.g. MAE, MSE). In this paper we assume these loss is upper bounded by ∆ , i.e., 0 ≤ δ j ( Y, ̂ Y ) ≤ ∆ .

Then in next step we define the Naive Estimator.

Definition A.3 ( Naive Estimator ) . Given the observation set, the Naive Estimator is defined as:

It is easy to verify that the expectation of this naive estimator is not equal to the true risk, as E O [ ̂ R NAIVE ( ̂ Y )] = R ( ̂ Y ) .

̸

Considering an imbalanced dataset as a subset of observations from a balanced one, we contrast it with the Inverse Propensity Score (IPS) estimator [34], underscoring the superiorities of our approach. Definition A.4 ( Inverse Propensity Score Estimator ) . The inverse propensity score (IPS) estimator (an unbiased estimator) is defined as

IPS estimator is an unbiased estimator, as we can verify by taking the expectation value over the observation set:

Finally, we define our VIR/DIR estimator below.

Definition A.5 ( VIR Estimator ) . The VIR estimator, denoted by ̂ R VIR ( ̂ Y | ˜ P ) , is defined as:

where { ˜ P i } |B| i =1 represents the smoothed label distribution utilized in our VIR's objective function (see Eqn.5 in the main paper). It's important to note that our VIR estimator is biased.

For multiple predictions, we select the 'best' estimator according to the following definition.

Definition A.6 ( Empirical Risk Minimizer ) . For a given hypothesis space H of predictions ̂ Y , the Empirical Risk Minimization (ERM) identifies the prediction ̂ Y ∈ H as

Lemma A.1 ( Tail Bound for VIR Estimator ) . For any given ̂ Y and Y , with probability 1 -η , the VIR estimator ̂ R VIR ( ̂ Y | ˜ P ) does not deviate from its expectation E O [ ̂ R VIR ( ̂ Y ERM | ˜ P )] by more than

Proof. For independent bounded random variables X 1 , · · · , X n that takes values in intervals of sizes ρ 1 , · · · , ρ n with probability 1 , and for any ϵ > 0 ,

Consider the error term for each bin i in ̂ R VIR ( ̂ Y ERM | ˜ P ) as X i . Using Hoeffding's inequality, define P ( X i = δ j ( Y, ̂ Y ) ˜ P i ) = ˜ P i and P ( X i = 0) = 1 -˜ P i . Then, by setting ϵ 0 = |B| · ϵ , we are then able to show that:

We can then solve for ϵ , completing the proof.

With all the aforementioned definitions, we can derive the generalization bound for the VIR estimator. Theorem A.1 (Generalization Bound of VIR) . In imbalanced regression with bins B , for any finite hypothesis space of predictions H = { ̂ Y 1 , ..., ̂ Y H } , the transductive prediction error of the empirical risk minimizer ̂ Y ERM using the VIR estimator with estimated propensities ˜ P ( P i > 0 ) and given training observations O from Y with independent Bernoulli propensities P , is bounded by:

Proof. We first re-state the generalization bound for our VIR estimator: with probability 1 -η , we have

We start to prove it from the LHS:

Below we derive each term:

Variance Term. With probability 1 -η , the variance term is derived as

where inequality (10) is by Boole's inequality (Union bound), and inequality (11) holds by Lemma A.1. Then, by solving for ϵ , we can derive Variance Term that with probability 1 -η ,

Bias Term. By definition, we can derive:

concluding the proof for the Bias Term, hence completing the proof for the whole generalization bound.

B Details for Experiments

Datasets. In this work, we evaluate our methods in terms of prediction accuracy and uncertainty estimation on four imbalanced datasets 2 , AgeDB [30], IMDB-WIKI [33], STS-B [7], and NYUD2DIR [35]. Due to page limit, the results for NYUD2-DIR [35] are in the supplementary. We follow

the preprocessing procedures in DIR [49]. Details for each datasets are to the Supplement, and details for label density distributions and levels of imbalance are discussed in DIR [49].

AgeDB-DIR : We use AgeDB-DIR constructed in DIR [49], which contains 12.2K images for training and 2.1K images for validation and testing. The maximum age in this dataset is 101 and the minimum age is 0, and the number of images per bin varies between 1 and 353.

IW-DIR : We use IMDB-WIKI-DIR (IW-DIR) constructed in DIR [49], which contains 191.5K training images and 11.0K validation and testing images. The maximum age is 186 and minimum age is 0; the maximum bin density is 7149, and minimum bin density is 1.

STS-B-DIR : We use STS-B-DIR constructed in DIR [49], which contains 5.2K pairs of training sentences and 1.0K pairs for validation and testing. This dataset is a collection of sentence pairs generated from news headlines, video captions, etc. Each pair is annotated by multiple annotators with a similarity score between 0 and 5.

NYUD2-DIR : We use NYUD2-DIR constructed in DIR [49], which contains 50K images for training and 654 images for testing, and to make the test set balanced 9357 test pixels for each bin are randomly selected. The depth maps have an upper bound of 10 meters, and we set the bin length as 0.1 meter.

Baselines. We use ResNet-50 [17] (for AgeDB-DIR, IMDB-WIKI-DIR and NYUD2-DIR) and BiLSTM [19] (for STS-B-DIR) as our backbone networks, and moredetails for baseline are in the supplement. we describe the baselines below.

Vanilla : We use the term VANILLA to denote a plain model without adding any approaches.

Metrics

MSE

↓

MAE

↓

Pearson

↑

Spearman

↑

Shot

All

Many

Medium

Few

All

Many

Medium

Few

All

Many

Medium

Few

All

Many

Medium

Few

VANILLA [49]

0.974

0.851

1.520

0.984

0.794

0.740

1.043

0.771

0.742

0.720

0.627

0.752

0.744

0.688

0.505

0.750

VAE [25]

0.968

0.833

1.511

1.102

0.782

0.721

1.040

0.767

0.751

0.724

0.621

0.749

0.752

0.674

0.501

0.743

DEEP ENSEMBLE [27]

0.972

0.846

1.496

1.032

0.791

0.723

1.096

0.792

0.746

0.723

0.619

0.750

0.741

0.689

0.501

0.746

INFER NOISE [29]

0.954

0.980

1.408

0.967

0.795

0.745

0.977

0.741

0.747

0.711

0.631

0.756

0.742

0.681

0.508

0.753

SMOTER [40]

1.046

0.924

1.542

1.154

0.834

0.782

1.052

0.861

0.726

0.693

0.653

0.706

0.726

0.656

0.556

0.691

SMOGN [5]

0.990

0.896

1.327

1.175

0.798

0.755

0.967

0.848

0.732

0.704

0.655

0.692

0.732

0.670

0.551

0.670

INV [49]

1.005

0.894

1.482

1.046

0.805

0.761

1.016

0.780

0.728

0.703

0.625

0.732

0.731

0.672

0.541

0.714

DER [1]

1.001

0.912

1.368

1.055

0.812

0.772

0.989

0.809

0.732

0.711

0.646

0.742

0.731

0.672

0.519

0.739

LDS [49]

0.914

0.819

1.319

0.955

0.773

0.729

0.970

0.772

0.756

0.734

0.638

0.762

0.761

0.704

0.556

0.743

FDS [49]

0.927

0.851

1.225

1.012

0.771

0.740

0.914

0.756

0.750

0.724

0.667

0.742

0.752

0.692

0.552

0.748

LDS + FDS [49]

0.907

0.802

1.363

0.942

0.766

0.718

0.986

0.755

0.760

0.740

0.652

0.766

0.764

0.707

0.549

0.749

RANKSIM [13]

0.903

0.908

0.911

0.804

0.761

0.759

0.786

0.712

0.758

0.706

0.690

0.827

0.758

0.673

0.493

0.849

LDS + FDS + DER [1]

1.007

0.880

1.535

1.086

0.812

0.757

1.046

0.842

0.729

0.714

0.635

0.731

0.730

0.680

0.526

0.699

VIR (OURS)

0.892

±

0.002

0.795

±

0.002

0.899

±

0.002

0.781

±

0.003

0.740

±

0.002

0.706

±

0.001

0.779

±

0.002

0.708

±

0.002

0.776

±

0.004

0.752

±

0.003

0.696

±

0.005

0.845

±

0.006

0.775

±

0.003

0.716

±

0.003

0.586

±

0.005

0.861

±

0.007

OURS

VS. VANILLA

+0.082

+0.056

+0.621

+0.203

+0.054

+0.034

+0.264

+0.063

+0.034

+0.032

+0.069

+0.093

+0.031

+0.028

+0.081

+0.111

OURS

VS. INFER NOISE

+0.062

+0.185

+0.509

+0.186

+0.055

+0.039

+0.198

+0.033

+0.029

+0.041

+0.065

+0.089

+0.033

+0.035

+0.078

+0.108

OURS

VS. DER

+0.109

+0.117

+0.469

+0.274

+0.072

+0.066

+0.210

+0.101

+0.044

+0.041

+0.050

+0.103

+0.044

+0.044

+0.067

+0.122

OURS

VS. LDS + FDS

+0.015

+0.007

+0.464

+0.161

+0.026

+0.012

+0.207

+0.047

+0.016

+0.012

+0.044

+0.079

+0.011

+0.009

+0.037

+0.112

OURS

VS. RANKSIM

+0.011

+0.113

+0.012

+0.023

+0.021

+0.053

+0.007

+0.004

+0.018

+0.046

+0.006

+0.018

+0.017

+0.043

+0.093

+0.012

Metrics

RMSE

↓

log

10

↓

δ

1

↑

δ

2

↑

δ

3

↑

Shot

All

Many

Medium

Few

All

Many

Medium

Few

All

Many

Medium

Few

All

Many

Medium

Few

All

Many

Medium

Few

VANILLA [49]

1.477

0.591

0.952

2.123

0.086

0.066

0.082

0.107

0.677

0.777

0.693

0.570

0.899

0.956

0.906

0.840

0.969

0.990

0.975

0.946

VAE [25]

1.483

0.596

0.949

2.131

0.084

0.062

0.079

0.110

0.675

0.774

0.693

0.575

0.894

0.951

0.906

0.846

0.963

0.982

0.976

0.951

DEEP ENSEMBLE [27]

1.479

0.595

0.954

2.126

0.091

0.067

0.082

0.109

0.678

0.782

0.702

0.583

0.906

0.961

0.912

0.851

0.972

0.993

0.981

0.956

INFER NOISE [29]

1.480

0.594

0.959

2.125

0.088

0.069

0.089

0.111

0.672

0.768

0.688

0.566

0.894

0.949

0.902

0.834

0.963

0.983

0.970

0.941

DER [1]

1.483

0.615

0.961

2.142

0.098

0.089

0.091

0.110

0.597

0.647

0.657

0.525

0.880

0.904

0.894

0.851

0.964

0.974

0.959

0.955

LDS [49]

1.387

0.671

0.913

1.954

0.086

0.079

0.079

0.097

0.672

0.701

0.706

0.630

0.907

0.932

0.929

0.875

0.976

0.984

0.982

0.964

FDS [49]

1.442

0.615

0.940

2.059

0.084

0.069

0.080

0.101

0.681

0.760

0.695

0.596

0.903

0.952

0.918

0.849

0.975

0.989

0.976

0.960

LDS + FDS [49]

1.338

0.670

0.851

1.880

0.080

0.074

0.070

0.090

0.705

0.730

0.764

0.655

0.916

0.939

0.941

0.884

0.979

0.984

0.983

0.971

LDS + FDS + DER [1]

1.426

0.703

0.906

1.918

0.092

0.081

0.088

0.098

0.676

0.677

0.754

0.621

0.889

0.912

0.899

0.862

0.964

0.976

0.969

0.958

VIR (OURS)

1.305

0.589

0.831

1.749

0.075

0.060

0.064

0.082

0.722

0.781

0.793

0.688

0.929

0.966

0.961

0.910

0.985

0.993

0.989

0.979

OURS

VS. VANILLA

+0.172

+0.002

+0.121

+0.374

+0.011

+0.006

+0.018

+0.025

+0.045

+0.004

+0.100

+0.118

+0.003

+0.010

+0.055

+0.070

+0.016

+0.003

+0.014

+0.033

OURS

VS. INFER NOISE

+0.175

+0.005

+0.128

+0.376

+0.013

+0.009

+0.025

+0.029

+0.050

+0.013

+0.105

+0.122

+0.035

+0.017

+0.059

+0.076

+0.022

+0.010

+0.019

+0.038

OURS

VS. DER

+0.178

+0.026

+0.130

+0.393

+0.023

+0.029

+0.027

+0.028

+0.125

+0.134

+0.136

+0.163

+0.049

+0.062

+0.067

+0.059

+0.021

+0.019

+0.030

+0.024

OURS

VS. LDS + FDS

+0.033

+0.081

+0.020

+0.131

+0.005

+0.014

+0.006

+0.008

+0.017

+0.051

+0.029

+0.033

+0.013

+0.027

+0.020

+0.026

+0.006

+0.009

+0.006

+0.008

Synthetic-Sample-Based Methods : Various existing imbalanced regression methods are also included as baselines; these include Deep Ensemble [27], Infer Noise [29], SMOTER [40], and SMOGN [5].

Cost-Sensitive Reweighting : As shown in DIR [49], the square-root weighting variant (SQINV) baseline (i.e. ( ∑ b ′ ∈B k ( y b , y b ′ ) p ( y b ′ ) ) -1 / 2 ) always outperforms Vanilla. Therefore, for simplicity and fair comparison, all our experiments (for both baselines and VIR) use SQINV weighting. To use SQINV in VIR, one simply needs to use the symmetric kernel k ( · , · ) described in the Method section of the main paper. To use SQINV in DER, we replace the final layer in DIR [49] with the DER layer [1] to produce the predictive distributions.

Evaluation Process. Following [28, 49], for a data sample x i with its label y i which falls into the target bins b i , we divide the label space into three disjoint subsets: many-shot region { b i ∈ B | y i ∈ b i & | y i | > 100 } , medium-shot region { b i ∈ B | y i ∈ b i & 20 ≤ | y i | ≤ 100 } , and few-shot region { b i ∈ B | y i ∈ b i & | y i | < 20 } , where | · | denotes the cardinality of the set. We report results on the overall test set and these subsets with the accuracy metrics discussed above.

Implementation Details. We use ResNet-50 [17] for all experiments in AgeDB-DIR and IMDBWIKI-DIR. For all the experiments in STS-B-DIR, we use 300-dimensional GloVe word embeddings (840B Common Crawl version) [31] (following [41]) and a two-layer, 1500-dimensional (per direction) BiLSTM [19] with max pooling to encode the paired sentences into independent vectors u and v , and then pass [ u ; v ; | u -v | ; uv ] to a regressor. We use the Adam optimizer [24] to train all models for 100 epochs, with same learning rate and decay by 0.1 and the 60-th and 90-th epoch, respectively. In order to determine the optimal batch size for training, we try different batch sizes and corroborate the conclusion from [49], i.e., the optimal batch size is 256 when other hyperparameters are fixed. Therefore, we stick to the batch size of 256 for all the experiments in the paper. We also use the same configuration as in DIR [49] for other hyperparameters.

We use PyTorch to implement our method. For fair comparison, we implemented a PyTorch version for the official TensorFlow implementation of DER[1]. To make sure we can obtain the reasonable uncertainty estimations, we restrict the range for α to [1 . 5 , ∞ ) instead of [1 . 0 , ∞ ) in DER. Besides, in the activation function SoftPlus , we set the hyperparameter beta to 0.1. As discussed in the main paper, we implement a layer which produces the parameters n, Ψ , Φ . We assign 2 as the minimum number for n , and use the same hyperparameter settings for activation function for DER layer.

To search for a combination hyperparameters of prior distribution { γ 0 , ν 0 , α 0 , β 0 } for NIG, we combine grid search method and random search method [2] to select the best hyperparameters. We first intuitively assign a value and a proper range with some step sizes which correspond to the hyperparameters, then, we apply grid search to search for the best combination for the hyperparameters on prior distributions. After locating a smaller range for each hyperparameters, we use random search to search for better combinations, if it exists. In the end, we find our best hyperparameter combinations for NIG prior distributions.

C Complete Results

We include the complete results for all the experiments in AgeDB-DIR, IMDB-WIKI-DIR, STSB-DIR and NYUD2-DIR in Table 7, Table 8, Table 9 and Table 10. These results demonstrate the superiority of our methods. Note that we did not select to report the baseline for DIR + DEEP ENS. since in DER paper [1], it has been showed that DER is better than Deep Ensemble method, therefore we select to report the baseline DIR+DER rather than DIR + DEEP ENS. .

D Discussions

D.1 Why We Need Bins

Throughout our method, we need to compute the statistics (i.e., the mean and variance) and the "statistics of statistics" of data points (Line 164-165); computing these statistics (e.g., the mean) requires a group of data points. Therefore, we need to partition the continuous label size into B bins. For example, in the equations from Line 176-177, e.g., µ µ b = 1 N b ∑ N b i =1 z µ i , we need to compute the statistics of bin b , which contains N b data points in the bin.

It is also worth noting in the extreme case where (i) each data point has a different label y and (ii) we use a very small bin size, each bin will then contain exactly only one data point.

D.2 Equal-Interval Bins versus Equal-Size Bins

Note that since our smoothing kernel function is based on labels (i.e., k ( y, y ′ ) ), it is more reasonable to use equal-interval bins rather than equal-size bins.

For example, if we use the equal-interval bins [0 , 1) , [1 , 2) , ... , VIR will naturally compute k ( y, y ′ ) for y = 1 , 2 , 3 , 4 , 5 , ... and y ′ = 1 , 2 , 3 , 4 , 5 , ... .

In contrast, if we use equal-size bins, VIR may end up with large intervals and may lead to inaccurate kernel values for k ( y, y ′ ) . To see this, consider a case where equal-size bins are [0 , 1) , [1 , 2) , [2 , 3 . 1) , [3 . 1 , 8 . 9) , ... ; the kernel value k ( y, y ′ ) between bins [2 , 3 . 1) and [3 . 1 , 8 . 9) is k (2 , 3 . 1) , which is very inaccurate since 3 . 1 is very far away from the mean of the bin [3 . 1 , 8 . 9) (i.e., 6 ). Using small and equal-interval bins can naturally address such issues.

D.3 The Number of Bins

Our preliminary results indicate that the performance of our VIR remains consistent regardless of the number of bins, as shown in the Sec. E.3 of the Supplement. Thus in our paper, we chose to use the same number of bins as the imbalanced regression literature [13, 49] for fair comparison with prior work. For example, in the AgeDB dataset where the regression labels are people's "age" in the range of 0 99, we use 100 bins, with each year as one bin.

D.4 Reweighting Methods and Stronger Augmentations

Our method focus on reweighting methods, and using augmentation (e.g., the SimCLR pipeline [10]) is an orthogonal direction to our work. However, we expect that data augmentation could further improve our VIR's performance. This is because one could perform data augmentation only on minority data to improve accuracy in the minority group, but this is sub-optimal; the reason is that one could potentially further perform data augmentation on majority data to improve accuracy in the majority group without sacrificing too much accuracy in the minority group. However, performing data augmentation on both minority and majority groups does not transform an imbalanced dataset to an balanced dataset. This is why our VIR is still necessary; VIR could be used on top of any data augmentation techniques to address the imbalance issue and further improve accuracy.

D.5 Discussion on I.I.D. and N.I.D. Assumptions

Generalization Error, Bias, and Variance. We could analyze the generalization error of our VIR by bounding the generalization with the sum of three terms: (a) the bias of our estimator, (2) the variance of our estimator, (3) model complexity. Essentially VIR uses the N.I.D. assumption increases our estimator's bias, but significantly reduces its variance in the imbalanced setting. Since the model complexity is kept the same (using the same backbone neural network) as the baselines, N.I.D. will lead to a lower generalization error.

Variance of Estimators in Imbalanced Settings. In the imbalanced setting, one typically use inverse weighting (i.e., the IPS estimator in Definition A.4) to produced an unbiased estimator (i.e., making the first term of the aforementioned bound zero). However, for data with extremely low density, its inverse would be extremely large, therefore leading to a very large variance for the estimator. Our VIR replaces I.I.D. with N.I.D. to 'smooth out' such singularity, and therefore significantly lowers the variance of the estimator (i.e., making the second term of the aforementioned bound smaller), and ultimately lowers the generalization error.

D.6 Why We Need Statistics of Statistics for Smoothing

Compared with DIR [49], which only considers the statistics for deterministic representations , our VIR considers the statistics of statistics for probabilistic representations , this is because the requirement to perform feature smoothing to get the representation z i necessitates the computation of mean and variance of z i 's neighboring data (i.e., data with neighboring labels). Here z i contains the statistics of neighboring data. In contrast, our VIR also needs to generate uncertainty estimation, which requires a stochastic representation for z i , e.g., the mean and variance of z i (note that z i itself is already a form of statistics). This motivates the hierarchical structure of the statistics of statistics . Here the variance measures the uncertainty of the representation.

D.7 The Choice of Kernel Function

The DIR paper shows that a simple Gaussian kernel with inverse square-root weighting (i.e., SQINV) achieves the best performance. Therefore, we use exactly the same parameter configuration as the DIR paper [49]. Specifically, we set σ = 2 ; for label y b in bin b , we define neighboring labels as labels y b ′ such that | y b ′ -y b | ≤ 2 , i.e., B contains 5 bins. For example, if y b = 23 , its neighboring labels are 21 , 22 , 23 , 24 , and 25 .

Besides, our preliminary results also show that the performance is not very sensitive to the choice of kernels, as long as the kernel k ( a, b ) reflects the distance between a and b , i.e., larger distance between a and b leads to smaller k ( a, b ) .

D.8 Why VIR Solves the Imbalanced Regression Problem

Our training objective function (Eqn.5 in the main paper) is the negative log likelihood for the Normal Inverse Gaussian (NIG) distribution, and each posterior parameter ( ν ∗ i , γ ∗ i , α ∗ i ) of the NIG distribution is reweighted by importance weights, thereby assigning higher weights to minority data during training and allowing minority data points to benefit more from their neighboring information.

Take ν ∗ i as an example. Assume a minority data point ( x i , y i ) that belongs to bin b , i.e., its label y i = y b . Note that there is a loss term ( y i -γ ∗ i ) 2 ν ∗ i in Eqn.5 , where γ ∗ i is the model prediction, y i is the label, and ν ∗ i is the importance weight for this data point.

Here ν ∗ i = ν 0 + ( ∑ b ′ ∈B k ( y b , y b ′ ) p ( y b ′ )) -1 / 2 · n i where n i represents the pseudo-count for the NIG distribution. Since ( x i , y i ) is a minority data point, data from its neighboring bins has smaller frequency p ( y b ′ ) and therefore smaller ∑ b ′ ∈B k ( y b , y b ′ ) p ( y b ′ ) , leading to a larger importance weight ν ∗ i for this minority data point in Eqn.5 .

This allows VIR to naturally put more focus on the minority data, thereby alleviating the imbalance problem.

D.9 Difference from DIR, VAE and DER

From a technical perspective, VIR is substantially different from any combinations of DIR [49], VAE [25], and DER [1]. Specifically,

VIR is a deep generative model to define how imbalanced data are generated, which is learned by a principled variational inference algorithm. In contrast, DIR is a simply discriminative model (without any principled generative model formulation) that directly predict the labels from input. It is more prone to overfitting.

DIR uses deterministic representations, with one vector as the final representation for each data point. In contrast, our VIR uses probabilistic representations, with one vector as the mean of the representation and another vector as the variance of the representation. Such dual representation is more robust to noise and therefore leads to better prediction performance.

DIR is a deterministic model, while our VIR is a Bayesian model. Essentially VIR is equivalent to sampling infinitely many predictions for each input data point and averaging these predictions. Therefore intuitively it makes sense that VIR could lead to better prediction performance.

Different from VAE and DIR, VIR introduces a reweighting mechanism naturally through the pseudo-count formulation in the NIG distribution (discussed in the paragraphs Intuition of Pseudo-Counts for VIR and From Pseudo-Counts to Balanced Predictive Distribution in the paper). Note that such a reweighting mechanism is more natural and powerful than DIR since it is rooted in the probabilistic formulation.

Unlike for the standard VAE, the optimal prior (that maximizes ELBO) is known to be the aggregated posterior, our optimal prior is a neighbor-weighted version of aggregated posterior: for standard V AE, different data points contribute independently to the aggregated posterior; in contrast, for our VIR, the importance of each data point with respect to the aggregated posterior is affected by data points with neighboring labels.

It is also worth noting that DIR cannot produce uncertainty estimation since it is a deterministic model. In contrast, Our VIR formulates a probabilistic deep generative model for imbalanced data, and therefore can naturally produce both more accurate predictions compared to DIR and better uncertainty estimation compared to DER.

D.10 Bayesian Neural Networks (BNNs)

We do not consider other BNNs in this work because:

Weights in Bayesian Neural Networks (BNNs) are extremely high-dimensional; therefore BNNs have several limitations, including the intractability of directly inferring the posterior distribution of the weights given data, the requirement and computational expense of sampling during inference, and the question of how to choose a weight prior [1]. In contrast, evidential regression does not have these challenges.

In our preliminary experiments, we found that typical BNN methods suffer from computational inefficiency and would require at least two to three times more computational time and memory usage. In contrast, evidential regression does not involve such computation and memory overhead; its overhead only involves the last (few) layers, and is therefore minimal.

Additionally, as demonstrated in [27], Deep Ensemble typically performs as well as or even better than BNNs. Our method outperforms Deep Ensemble, therefore suggesting its superiority over typical BNN methods.

E Additional Experiment Results

E.1 Ablation Study on VIR

In this section, we include ablation studies to verify that our VIR can outperform its counterparts in DIR (i.e., smoothing on the latent space) and DER (i.e., NIG distribution layers).

Ablation Study on q ( z i |{ x i } N i =1 ) . To verify the effectiveness of VIR's encoder q ( z i |{ x i } N i =1 ) , we replace VIR's predictor p ( y i | z i ) with a linear layer (as in DIR). Table 14 shows that compared to its counterpart, FDS [49], our encoder-only VIR leads to a considerable improvements even without generating the NIG distribution. Both verify the effectiveness of our VIR's q ( z i |{ x i } N i =1 ) .

Ablation Study on p ( y i | z i ) . To verify the effectiveness of VIR's predictor p ( y i | z i ) , we replace VIR's encoder q ( z i |{ x i } N i =1 ) with a simple deterministic encoder as in DER [1]. Table 14 and Table 15 show that compared to DER, the counterpart of VIR's predictor, our VIR's predictor still outperforms than DER, demonstrating its

effectiveness. Both verifies our claim that directly reweighting DER breaks NIG and leads to poor performance.

E.2 Ablation Study on λ .

In this section, we include ablation studies on the λ in our objective function. For λ ∈ { 10 . 0 , 1 . 0 , 0 . 1 , 0 . 01 , 0 . 001 } , we run our VIR model on the AgeDB dataset. Table 12 shows the results. We can observe that our model achieves the best performance when λ = 0 . 1 .

E.3 Ablation Study on Number of Bins

In this section, we include ablation studies on the number of bins in our settings. For the cases with 100 / 1 = 100 , 100 / 3 ≈ 33 , and 100 / 5 = 20 bins, we run our VIR model on the AgeDB dataset. Table 13 shows the results. We can observe that our VIR remains consistent regardless of the number of bins.

E.4 Calibrated Uncertainty

In a bid to enhance the evaluation of our model's uncertainty, we used our validation set to apply calibration techniques (specifically, variants of temperature scaling [14]) on different methods. We focused on each of the output distribution parameters ν, α, β as discussed in our main paper, introducing individual scalar weights for each parameter to calibrate uncertainty estimation. Upon deriving each weight w ν , w α , w β from the validation set, we incorporated them into the test dataset to ascertain the final performance. The data outlined in Table 16 indicates that following re-calibration, the uncertainty was further optimized. Notwithstanding such uncertainty calibration, our model persists in demonstrating superior performance compared to other benchmark methods.

E.5 Error bars on Accuracy

In order to further underscore the superiority of our methodology, we also included error bars for our proposed method (i.e., VIR) which are generated from five independent runs. Note that we did not report error bars for those second/third best baselines (e.g., RankSim, LDS+FDS) since we directly use the reported performance from their papers, and in NYUD2-DIR, all the error bars on VIR are approximately 0.001. Due to the width constraints of the paper, these are not included in Table 10. Results in Table 7, Table 8, and Table 9 demonstrate the effectiveness of our approach.

Peer reviews:
comments: SUMMARY:
This paper simultaneously addresses the label imbalance problem and uncertainty qualification capability in regression.
The authors propose to enhance the reweighting technique dealing with the imbalance problem in (Yang et al., 2021) to be applicable to VAE and combine the method with the output distribution and the corresponding loss in (Amini et al., 2020), which provides uncertainty qualification capability.
Experimental results on several real-world datasets demonstrate that the proposed method performs better than state-of-the-art imbalanced regression methods in terms of both accuracy and uncertainty estimation.

STRENGTHS:
### Originality:
- The authors simultaneously address the label imbalance problem and uncertainty estimation capability in regression, which is a novel problem setting.

### Quality:
- The combination of the reweighting technique in (Yang et al., 2021) and the output distribution and the corresponding loss in (Amini et al., 2020) is non-trivial and works well.

- The authors showed the superiority of the proposed method experimentally in terms of both accuracy and uncertainty estimation, where they used multiple public datasets.

### Clarity:
- The presentation is clear.

WEAKNESSES:
- Good combination of the SOTA, but the originality can be limited because of that.

QUESTIONS:
NA

LIMITATIONS:
They discussed that the exact computation of variance of the variances is challenging in Section 5.3.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a variational regression model for imbalanced data, which (1) borrows data with similar regression labels for variational distribution (neighboring and identically distributed: N.I.D.) and (2) utilize the conjugate distributions to impose probabilistic reweighting on the imbalanced data to give better uncertainty estimation. Experiments show the proposed model achieves the SOTA on several datasets.\n\nSTRENGTHS:\n- Imbalanced regression is a research area that is yet to be explored, although its real-world application is important.\n\n- In addition, the proposed model can provide uncertainty of the prediction, which is important for real-world applications.\n\n- The uncertainty estimation on imbalanced datasets is also an interesting field but is yet to be explored.\n\n- Overall, the present topic is very relevant in the community. I encourage the authors to develop and explore this direction more, in view of the nice performance of the model.\n\n- The performance of the proposed model is excellent. Both of the accuracy and uncertainty estimation outperform the SOTA models.\n\n- The present paper is easy to follow and well-written. I enjoyed reading it. I found some nice \"road signs\" for the readers to follow the logic and story.\n\nWEAKNESSES:\n- The code for reproduction is not available. I would like to see and use your code. I could not find any other major weakness.\n\n- See Questions below.\n\nQUESTIONS:\nHere are my questions and comments.\n\n- [Question] Just out of curiosity. What is the optimal prior for VIR? For the standard VAE, the optimal prior (that maximizes ELBO) is known to be the aggregated posterior. This might lead to an interesting future work.\n\n- [Comment (minor)] \"NIG\" on L.137 should be spelled out here, rather than on L. 213, or define NIG on L. 40.\n\n- [Comment (super-minor)] (1), (2), ... in the main text might sometimes be confused with the number of equations. Possible alternatives are: (i), (ii),..., (I), (II), ..., (a), (b), ..., (A), (B), ..., 1), 2), ..., etc.\n\n- [Question] Where does the idea of the hierarchical structure of the statistics (L. 166--195) (statistics of statistics for smoothing) come from? What motivated this idea? This is very interesting.\n\n- [Comment (minor)] L. 191: cross -> across?\n\n- [Comment (minor)] L. 212: correspond -> corresponding?\n\n- [Question (major)] At first glance, I could not fully understand why Eqn. 5 alleviate the imbalance problem. Could you clarify the underlying mechanism more?\n\n- [Question (major)] Is the performance improvement (accuracy and uncertainty) \"significant\"? If so, in what sense? (Sorry for the vague question, but this is an important point when evaluating model's performance and paper's quality because a marginal improvement is sometimes meaningless.)\n\nLIMITATIONS:\nLimitations are included in Section 5.3.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a variational imbalanced regression model by taking the Neighboring and Identically Distributed (N.I.D.) assumption to solve both imbalanced regression and uncertainty estimation problems. Experiments on four imbalanced datasets demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1) Compared with imbalanced classification,  imbalanced regression is underexplored and also an important topic.\n2) The Neighboring and Identically Distributed (N.I.D.) assumption seems more reasonable than the Independent and Identically Distributed (I.I.D.) assumption.\n3) The proposed method improves not only the performance of few-shot region, but also the performance of many-shot region.\n\nWEAKNESSES:\n1) The description of some parts of the paper is not clear, e.g., how to define the neighboring labels?  What is the detailed formulation of the importance weights in line 237?\n2) Some statements in the paper are somewhat subjective and lack support, e.g., in line 42-43, the authors claim that \"This allows the negative log likelihood to naturally put more focus on the minority data\", but I do not find any \"naturally\" thing, the important weights are mainly determined by the kernel functions, which needs manually selection; in line 273, the authors claim that \"Such dual representation is more robust to noise\", but they neither list any references nor conduct any experiments to support it.\n\nQUESTIONS:\n1) What are the new challenges of imbalanced regression compared with imbalanced classification?\n2) Why do you partition the label space into B equal-interval bins? Does the choice of B affect the performance of VIR?\n3) The choice of the kernel functions and experiments for the corresponding parameters selection should be given since it determines the importance weights.\n4) In section 4.1, the authors claim that N.I.D. will lead to a lower generalization error just by simple text description, without any rigorous computation or derivation, this is not convincing for me.\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, authors recognize that although the existing regression models for the imbalanced datasets have been mainly developed to improve the prediction accuracy, they overlooked the quality of the uncertainty estimation. In this context, authors propose a deep probabilistic regression framework, to improve the uncertainty estimation performance as well by combining the idea of [1] and [2]. \n\nSpecifically, authors first consider multiple bins, splitted on the range of labels to get statistic of the labels. Then, they revise the latent statistic of VAE for the imbalanced datasets, by smoothing these features based on the statistic on each bins and then applying probabilistic whitening and recoloring procedure. Next, they use the revised latent features to get the posterior distribution of NIG distribution, which acts as the pseudo counts that alleviates the issue of imbalanced sets. Last, they employ these parameters for prediction and training.   \n\nEmpirically, authors demonstrate that the proposed approach can improve the performance of the prediction accuracy and uncertainty estimation on various datasets.\n\n[1] Delving into Deep Imbalanced Regression - ICML 21\n\n[2] Deep Evidential Regression - NeurIPS 20\n\nSTRENGTHS:\n* This work extends the DIR of [1], as the probabilistic model, to estimate the uncertainty.\n\n* This work considers to use the NIG posterior distribution, updated by the statistic of the stochastic latent feature. This seems to help balance the latent features of the imbalanced labels and thus yield the credible uncertainty for the imbalanced datasets. I believe that this is a novel part of this work as comparing [1] and [2].\n\n[1] Delving into Deep Imbalanced Regression - ICML 21\n\n[2] Deep Evidential Regression - NeurIPS 20\n\nWEAKNESSES:\n* Absence of ablation study\n> In experiment section, it seems that the proposed method improves the prediction accuracy and uncertainty estimation. However, the current work does not investigate (1) whether each trick of the proposed method, such as the use of the stochastic latent feature (VAE) and use of the posterior distribution (NIG), is effective and (2) whether the proposed method is consistent up to the number of bins. \n\n* Less explanation on why the evidential regression model is used along with DIR, instead of using other BNNs.\n> In general, BNNs is widely used to estimate the uncertainty. I believe that applying the BNNs with DIR could be a direct way to solve the targeted problem. However, author takes the evidential regression approach, without explaining the reason or its motivation. This seems to less persuade why the proposed method is reasonable. If authors provide the motivation of this approach or can demonstrate that the proposed method could outperform the results of the DIR, obtained by BNNs or other approach for uncertainty estimation, I believe that the contribution of this work would be more clear and strong.\n\nQUESTIONS:\n* Is the performance of the proposed method consistent up to the number of bins? How can performance vary with the number of bins?\n\n* Why did the authors employ the stochastic latent feature through VAE, that is different approach to the [1] ? \n Does the stochastic feature improve the performance as comparing when the deterministic statistic is used to update the parameters of the NIG posterior distribution ?\n\n* Did the authors consider the BNNs with DIR or the Deep ensemble approach [2] with DIR to solve the imbalanced regression problem ? \n\n\n[1] Delving into Deep Imbalanced Regression - ICML 21\n\n[2] Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles - NeurIPS 17\n\nLIMITATIONS:\nSee above Weaknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}