SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- AgeDB-DIR: We use AgeDB-DIR constructed in DIR [49], which contains 12.2K images for training and 2.1K images for validation and testing. The maximum age in this dataset is 101 and the minimum age is 0, and the number of images per bin varies between 1 and 353.
- IW-DIR: We use IMDB-WIKI-DIR (IW-DIR) constructed in DIR [49], which contains 191.5K training images and 11.0K validation and testing images. The maximum age is 186 and minimum age is 0; the maximum bin density is 7149, and minimum bin density is 1.
- STS-B-DIR: We use STS-B-DIR constructed in DIR [49], which contains 5.2K pairs of training sentences and 1.0K pairs for validation and testing. This dataset is a collection of sentence pairs generated from news headlines, video captions, etc. Each pair is annotated by multiple annotators with a similarity score between 0 and 5.
- NYUD2-DIR: We use NYUD2-DIR constructed in DIR [49], which contains 50K images for training and 654 images for testing, and to make the test set balanced 9357 test pixels for each bin are randomly selected. The depth maps have an upper bound of 10 meters, and we set the bin length as 0.1 meter.
Baselines. We use ResNet-50 [17] (for AgeDB-DIR, IMDB-WIKI-DIR and NYUD2-DIR) and BiLSTM [19] (for STS-B-DIR) as our backbone networks, and moredetails for baseline are in the supplement. we describe the baselines below.
- Vanilla: We use the term VANILLA to denote a plain model without adding any approaches.
Table 9: Accuracy on STS-B-DIR. For baselines, we directly use the reported performance in their paper and therefore do not have error bars.
Metrics	MSE↓	MAE↓	Pearson ↑	Spearman ↑
Shot	All	Many	Medium	Few	All	Many	Medium	Few	All	Many	Medium	Few	All	Many	Medium	Few
VANILLA [49]	0.974	0.851	1.520	0.984	0.794	0.740	1.043	0.771	0.742	0.720	0.627	0.752	0.744	0.688	0.505	0.750
VAE [25]	0.968	0.833	1.511	1.102	0.782	0.721	1.040	0.767	0.751	0.724	0.621	0.749	0.752	0.674	0.501	0.743
DEEP ENSEMBLE [27]	0.972	0.846	1.496	1.032	0.791	0.723	1.096	0.792	0.746	0.723	0.619	0.750	0.741	0.689	0.501	0.746
INFER NOISE [29]	0.954	0.980	1.408	0.967	0.795	0.745	0.977	0.741	0.747	0.711	0.631	0.756	0.742	0.681	0.508	0.753
SMOTER [40]	1.046	0.924	1.542	1.154	0.834	0.782	1.052	0.861	0.726	0.693	0.653	0.706	0.726	0.656	0.556	0.691
SMOGN [5]	0.990	0.896	1.327	1.175	0.798	0.755	0.967	0.848	0.732	0.704	0.655	0.692	0.732	0.670	0.551	0.670
INV [49]	1.005	0.894	1.482	1.046	0.805	0.761	1.016	0.780	0.728	0.703	0.625	0.732	0.731	0.672	0.541	0.714
DER [1]	1.001	0.912	1.368	1.055	0.812	0.772	0.989	0.809	0.732	0.711	0.646	0.742	0.731	0.672	0.519	0.739
LDS [49]	0.914	0.819	1.319	0.955	0.773	0.729	0.970	0.772	0.756	0.734	0.638	0.762	0.761	0.704	0.556	0.743
FDS [49]	0.927	0.851	1.225	1.012	0.771	0.740	0.914	0.756	0.750	0.724	0.667	0.742	0.752	0.692	0.552	0.748
LDS + FDS [49]	0.907	0.802	1.363	0.942	0.766	0.718	0.986	0.755	0.760	0.740	0.652	0.766	0.764	0.707	0.549	0.749
RANKSIM [13]	0.903	0.908	0.911	0.804	0.761	0.759	0.786	0.712	0.758	0.706	0.690	0.827	0.758	0.673	0.493	0.849
LDS + FDS + DER [1]	1.007	0.880	1.535	1.086	0.812	0.757	1.046	0.842	0.729	0.714	0.635	0.731	0.730	0.680	0.526	0.699
VIR (OURS)	0.892±0.002	0.795±0.002	0.899±0.002	0.781±0.000	0.740±0.002	0.706±0.001	0.779±0.002	0.708±0.002	0.776±0.004	0.752±0.003	0.696±0.005	0.845±0.006	0.775±0.003	0.716±0.003	0.586±0.005	0.861±0.007
OURS vs. VANILLA	+0.082	+0.056	+0.621	+0.203	+0.054	+0.034	+0.264	+0.063	+0.034	+0.032	+0.069	+0.093	+0.031	+0.028	+0.081	+0.111
OURS vs. INFER NOISE	+0.062	+0.185	+0.509	+0.186	+0.055	+0.039	+0.198	+0.033	+0.029	+0.041	+0.065	+0.089	+0.033	+0.035	+0.078	+0.108
OURS vs. DER	+0.109	+0.117	+0.469	+0.274	+0.072	+0.066	+0.210	+0.101	+0.044	+0.041	+0.050	+0.103	+0.044	+0.044	+0.067	+0.122
OURS vs. LDS + FDS	+0.015	+0.007	+0.464	+0.161	+0.026	+0.012	+0.207	+0.047	+0.016	+0.012	+0.044	+0.079	+0.011	+0.009	+0.037	+0.112
OURS vs. RANKSIM	+0.011	+0.113	+0.012	+0.023	+0.021	+0.053	+0.007	+0.004	+0.018	+0.046	+0.006	+0.018	+0.017	+0.043	+0.093	+0.012
Table 10: Accuracy on NYUD2-DIR.
Metrics	RMSE↓	log10↓	δ1↑	δ2↑	δ3↑
Shot	All	Many	Medium	Few	All	Many	Medium	Few	All	Many	Medium	Few	All	Many	Medium	Few	All	Many	Medium	Few
VANILLA [49]	1.477	0.591	0.952	2.123	0.086	0.066	0.082	0.107	0.677	0.777	0.693	0.570	0.899	0.956	0.906	0.840	0.969	0.990	0.975	0.946
VAE [25]	1.483	0.596	0.949	2.131	0.084	0.062	0.079	0.110	0.675	0.774	0.693	0.575	0.894	0.951	0.906	0.846	0.963	0.982	0.976	0.951
DEEP ENSEMBLE [27]	1.479	0.595	0.954	2.126	0.091	0.067	0.082	0.109	0.678	0.782	0.702	0.583	0.906	0.961	0.912	0.851	0.972	0.993	0.981	0.956
INFER NOISE [29]	1.480	0.594	0.959	2.125	0.088	0.069	0.089	0.111	0.672	0.768	0.688	0.566	0.894	0.949	0.902	0.834	0.963	0.983	0.970	0.941
DER [1]	1.483	0.615	0.961	2.142	0.098	0.089	0.091	0.110	0.597	0.647	0.657	0.525	0.880	0.904	0.894	0.851	0.964	0.974	0.959	0.955
LDS [49]	1.387	0.671	0.913	1.954	0.086	0.079	0.079	0.097	0.672	0.701	0.706	0.630	0.907	0.932	0.929	0.875	0.976	0.984	0.982	0.964
FDS [49]	1.442	0.615	0.940	2.059	0.084	0.069	0.080	0.101	0.681	0.760	0.695	0.596	0.903	0.952	0.918	0.849	0.975	0.989	0.976	0.960
LDS + FDS [49]	1.338	0.670	0.851	1.880	0.080	0.074	0.070	0.090	0.705	0.730	0.764	0.655	0.916	0.939	0.941	0.884	0.979	0.984	0.983	0.971
LDS + FDS + DER [1]	1.426	0.703	0.906	1.918	0.092	0.081	0.088	0.098	0.676	0.677	0.754	0.621	0.889	0.912	0.899	0.862	0.964	0.976	0.969	0.958
VIR (OURS)	1.305	0.589	0.831	1.749	0.075	0.060	0.064	0.082	0.722	0.781	0.793	0.688	0.929	0.966	0.961	0.910	0.985	0.993	0.989	0.979
OURS vs. VANILLA	+0.172	+0.002	+0.121	+0.374	+0.011	+0.006	+0.018	+0.025	+0.045	+0.004	+0.100	+0.118	+0.003	+0.010	+0.055	+0.070	+0.016	+0.003	+0.014	+0.033
OURS vs. INFER NOISE	+0.175	+0.005	+0.128	+0.376	+0.013	+0.009	+0.025	+0.029	+0.050	+0.013	+0.105	+0.122	+0.035	+0.017	+0.059	+0.076	+0.022	+0.010	+0.019	+0.038
OURS vs. DER	+0.178	+0.026	+0.130	+0.393	+0.023	+0.029	+0.027	+0.028	+0.125	+0.134	+0.136	+0.163	+0.049	+0.062	+0.067	+0.059	+0.021	+0.019	+0.030	+0.024
OURS vs. LDS + FDS	+0.033	+0.081	+0.020	+0.131	+0.005	+0.014	+0.006	+0.088	+0.017	+0.051	+0.029	+0.033	+0.013	+0.027	+0.020	+0.026	+0.006	+0.009	+0.006	+0.008
- Synthetic-Sample-Based Methods: Various existing imbalanced regression methods are also included as baselines; these include Deep Ensemble [27], Infer Noise [29], SMOTER [40], and SMOGN [5].
- Cost-Sensitive Reweighting: As shown in DIR [49], the square-root weighting variant (SQINV) baseline (i.e.  $\left(\sum_{b' \in \mathcal{B}} k(y_b, y_{b'}) p(y_{b'})\right)^{-1/2}$ ) always outperforms Vanilla. Therefore, for simplicity and fair comparison, all our experiments (for both baselines and VIR) use SQINV weighting. To use SQINV in VIR, one simply needs to use the symmetric kernel  $k(\cdot, \cdot)$  described in the Method section of the main paper. To use SQINV in DER, we replace the final layer in DIR [49] with the DER layer [1] to produce the predictive distributions.
Evaluation Process. Following [28, 49], for a data sample  $x_{i}$  with its label  $y_{i}$  which falls into the target bins  $b_{i}$ , we divide the label space into three disjoint subsets: many-shot region  $\{b_{i} \in \mathcal{B} \mid y_{i} \in b_{i} \& |y_{i}| > 100\}$ , medium-shot region  $\{b_{i} \in \mathcal{B} \mid y_{i} \in b_{i} \& 20 \leq |y_{i}| \leq 100\}$ , and few-shot region  $\{b_{i} \in \mathcal{B} \mid y_{i} \in b_{i} \& |y_{i}| < 20\}$ , where  $|\cdot|$  denotes the cardinality of the set. We report results on the overall test set and these subsets with the accuracy metrics discussed above.
Implementation Details. We use ResNet-50 [17] for all experiments in AgeDB-DIR and IMDB-WIKI-DIR. For all the experiments in STS-B-DIR, we use 300-dimensional GloVe word embeddings (840B Common Crawl version) [31] (following [41]) and a two-layer, 1500-dimensional (per direction) BiLSTM [19] with max pooling to encode the paired sentences into independent vectors  $u$  and  $v$ , and then pass  $[u; v; |u - v|; uv]$  to a regressor. We use the Adam optimizer [24] to train all models for 100 epochs, with same learning rate and decay by 0.1 and the 60-th and 90-th epoch, respectively. In order to determine the optimal batch size for training, we try different batch sizes and corroborate the conclusion from [49], i.e., the optimal batch size is 256 when other hyperparameters are fixed. Therefore, we stick to the batch size of 256 for all the experiments in the paper. We also use the same configuration as in DIR [49] for other hyperparameters.
We use PyTorch to implement our method. For fair comparison, we implemented a PyTorch version for the official TensorFlow implementation of DER[1]. To make sure we can obtain the reasonable uncertainty estimations, we restrict the range for  $\alpha$  to  $[1.5,\infty)$  instead of  $[1.0,\infty)$  in DER. Besides, in the activation function SoftPlus, we set the hyperparameter beta to 0.1. As discussed in the main paper, we implement a layer which produces the parameters  $n$ ,  $\Psi$ ,  $\Phi$ . We assign 2 as the minimum number for  $n$ , and use the same hyperparameter settings for activation function for DER layer.
Table 11: Uncertainty on NYUD2-DIR.
Metrics	NLL ↓	AUSE ↓
Shot	All	Many	Medium	Few	All	Many	Medium	Few
DEEP ENSEMBLE [27]	5.054	3.640	3.856	5.335	0.782	0.658	0.604	0.583
INFER NOISE [29]	4.542	3.120	3.634	5.028	0.764	0.643	0.566	0.408
DER [1]	4.169	2.913	3.011	4.777	0.713	0.623	0.535	0.382
LDS + FDS + DER [1]	4.175	2.987	2.976	4.686	0.715	0.629	0.511	0.366
VIR (OURS)	3.866	2.815	2.727	4.113	0.690	0.603	0.493	0.335
OURS VS. DER	+0.303	+0.098	+0.284	+0.664	+0.023	+0.020	+0.042	+0.047
OURS VS. LDS + FDS + DER	+0.309	+0.172	+0.249	+0.573	+0.025	+0.026	+0.018	+0.031
Table 12: Ablation study on  $\lambda$  on AgeDB-DIR.
Metrics	MSE ↓	MAE ↓	NLL ↓
Shot	All	Many	Medium	Few	All	Many	Medium	Few	All	Many	Medium	Few
λ = 10.0	104.31	91.01	116.43	196.35	7.88	7.38	8.42	11.13	3.827	3.733	4.140	4.407
λ = 1.0	104.10	87.28	128.26	196.12	7.83	7.21	8.81	10.89	3.848	3.738	4.041	4.356
λ = 0.1	86.28	76.87	101.57	132.90	7.19	6.75	7.97	9.19	3.785	3.694	3.963	4.151
λ = 0.01	86.86	76.58	99.95	147.82	7.12	6.69	7.72	9.59	3.887	3.797	4.007	4.401
λ = 0.001	87.25	74.13	104.78	162.64	7.13	6.64	7.92	9.63	3.980	3.868	4.161	4.546
To search for a combination hyperparameters of prior distribution  $\{\gamma_0,\nu_0,\alpha_0,\beta_0\}$  for NIG, we combine grid search method and random search method [2] to select the best hyperparameters. We first intuitively assign a value and a proper range with some step sizes which correspond to the hyperparameters, then, we apply grid search to search for the best combination for the hyperparameters on prior distributions. After locating a smaller range for each hyperparameters, we use random search to search for better combinations, if it exists. In the end, we find our best hyperparameter combinations for NIG prior distributions.
C Complete Results
We include the complete results for all the experiments in AgeDB-DIR, IMDB-WIKI-DIR, STS-B-DIR and NYUD2-DIR in Table 7, Table 8, Table 9 and Table 10. These results demonstrate the superiority of our methods. Note that we did not select to report the baseline for DIR + DEEP ENS. since in DER paper [1], it has been showed that DER is better than Deep Ensemble method, therefore we select to report the baseline DIR+DER rather than DIR + DEEP ENS..
D.1 Why We Need Bins
Throughout our method, we need to compute the statistics (i.e., the mean and variance) and the "statistics of statistics" of data points (Line 164-165); computing these statistics (e.g., the mean) requires a group of data points. Therefore, we need to partition the continuous label size into  $\mathcal{B}$  bins. For example, in the equations from Line 176-177, e.g.,  $\mu_b^\mu = \frac{1}{N_b} \sum_{i=1}^{N_b} z_i^\mu$ , we need to compute the statistics of bin  $b$ , which contains  $N_b$  data points in the bin.
It is also worth noting in the extreme case where (i) each data point has a different label  $y$  and (ii) we use a very small bin size, each bin will then contain exactly only one data point.
D.2 Equal-Interval Bins versus Equal-Size Bins
Note that since our smoothing kernel function is based on labels (i.e.,  $k(y, y')$ ), it is more reasonable to use equal-interval bins rather than equal-size bins.
Table 13: Comparison for different numbers of Bins. "Med." is short for "Medium".
Metrics	Bins	MSE ↓	MAE ↓	GM ↓
Shot	#	All	Many	Med.	Few	All	Many	Med.	Few	All	Many	Med.	Few
RANKSIM	100	83.51	71.99	99.14	149.05	7.02	6.49	7.84	9.68	4.53	4.13	5.37	6.89
VIR (OURS)	100	81.76	70.61	91.47	142.36	6.99	6.39	7.47	9.51	4.41	4.07	5.05	6.23
RANKSIM	33	109.45	91.78	128.10	187.13	7.46	6.94	8.42	10.66	5.13	4.70	5.23	8.21
VIR (OURS)	33	84.77	77.29	95.66	125.33	7.01	6.70	7.45	8.74	4.36	4.20	4.73	4.94
RANKSIM	20	98.71	84.38	107.89	171.04	7.32	6.78	8.35	10.57	5.33	4.51	5.69	7.92
VIR (OURS)	20	84.05	72.12	100.49	151.25	7.06	6.50	7.90	10.06	4.49	4.05	5.34	7.28
- For example, if we use the equal-interval bins  $[0,1), [1,2), \dots, \text{VIR}$  will naturally compute  $k(y,y')$  for  $y = 1,2,3,4,5,\ldots$  and  $y' = 1,2,3,4,5,\ldots$ .
- In contrast, if we use equal-size bins, VIR may end up with large intervals and may lead to inaccurate kernel values for  $k(y, y')$ . To see this, consider a case where equal-size bins are [0, 1), [1, 2), [2, 3.1), [3.1, 8.9), ..., the kernel value  $k(y, y')$  between bins [2, 3.1) and [3.1, 8.9) is  $k(2, 3.1)$ , which is very inaccurate since 3.1 is very far away from the mean of the bin [3.1, 8.9) (i.e., 6). Using small and equal-interval bins can naturally address such issues.
D.3 The Number of Bins
Our preliminary results indicate that the performance of our VIR remains consistent regardless of the number of bins, as shown in the Sec. E.3 of the Supplement. Thus in our paper, we chose to use the same number of bins as the imbalanced regression literature [13, 49] for fair comparison with prior work. For example, in the AgeDB dataset where the regression labels are people's "age" in the range of 0.99, we use 100 bins, with each year as one bin.
D.4 Reweighting Methods and Stronger Augmentations
Our method focuses on reweighting methods, and using augmentation (e.g., the SimCLR pipeline [10]) is an orthogonal direction to our work. However, we expect that data augmentation could further improve our VIR's performance. This is because one could perform data augmentation only on minority data to improve accuracy in the minority group, but this is sub-optimal; the reason is that one could potentially further perform data augmentation on majority data to improve accuracy in the majority group without sacrificing too much accuracy in the minority group. However, performing data augmentation on both minority and majority groups does not transform an imbalanced dataset to an balanced dataset. This is why our VIR is still necessary; VIR could be used on top of any data augmentation techniques to address the imbalance issue and further improve accuracy.
D.5 Discussion on I.I.D. and N.I.D. Assumptions
Generalization Error, Bias, and Variance. We could analyze the generalization error of our VIR by bounding the generalization with the sum of three terms: (a) the bias of our estimator, (2) the variance of our estimator, (3) model complexity. Essentially VIR uses the N.I.D. assumption increases our estimator's bias, but significantly reduces its variance in the imbalanced setting. Since the model complexity is kept the same (using the same backbone neural network) as the baselines, N.I.D. will lead to a lower generalization error.
Variance of Estimators in Imbalanced Settings. In the imbalanced setting, one typically use inverse weighting (i.e., the IPS estimator in Definition A.4) to produce an unbiased estimator (i.e., making the first term of the aforementioned bound zero). However, for data with extremely low density, its inverse would be extremely large, therefore leading to a very large variance for the estimator. Our VIR replaces I.I.D. with N.I.D. to "smooth out" such singularity, and therefore significantly lowers the variance of the estimator (i.e., making the second term of the aforementioned bound smaller), and ultimately lowers the generalization error.
D.6 Why We Need Statistics of Statistics for Smoothing
Compared with DIR [49], which only considers the statistics for deterministic representations, our VIR considers the statistics of statistics for probabilistic representations, this is because the requirement to perform feature smoothing to get the representation  $z_{i}$  necessitates the computation of mean and variance of  $z_{i}$ 's neighboring data (i.e., data with neighboring labels). Here  $z_{i}$  contains the statistics of neighboring data. In contrast, our VIR also needs to generate uncertainty estimation, which requires a stochastic representation for  $z_{i}$ , e.g., the mean and variance of  $z_{i}$  (note that  $z_{i}$  itself is already a form of statistics). This motivates the hierarchical structure of the statistics of statistics. Here the variance measures the uncertainty of the representation.
D.7 The Choice of Kernel Function
The DIR paper shows that a simple Gaussian kernel with inverse square-root weighting (i.e., SQINV) achieves the best performance. Therefore, we use exactly the same parameter configuration as the DIR paper [49]. Specifically, we set  $\sigma = 2$ ; for label  $y_{b}$  in bin  $b$ , we define neighboring labels as labels  $y_{b'}$  such that  $|y_{b'} - y_{b}| \leq 2$ , i.e.,  $B$  contains 5 bins. For example, if  $y_{b} = 23$ , its neighboring labels are 21, 22, 23, 24, and 25.
Besides, our preliminary results also show that the performance is not very sensitive to the choice of kernels, as long as the kernel  $k(a,b)$  reflects the distance between  $a$  and  $b$ , i.e., larger distance between  $a$  and  $b$  leads to smaller  $k(a,b)$ .
D.8 Why VIR Solves the Imbalanced Regression Problem
Our training objective function (Eqn.5 in the main paper) is the negative log likelihood for the Normal Inverse Gaussian (NIG) distribution, and each posterior parameter  $(\nu_{i}^{*},\gamma_{i}^{*},\alpha_{i}^{*})$  of the NIG distribution is reweighted by importance weights, thereby assigning higher weights to minority data during training and allowing minority data points to benefit more from their neighboring information.
Take  $\nu_{i}^{*}$  as an example. Assume a minority data point  $(x_{i},y_{i})$  that belongs to bin  $b$ , i.e., its label  $y_{i} = y_{b}$ . Note that there is a loss term  $(y_{i} - \gamma_{i}^{*})^{2}\nu_{i}^{*}$  in Eqn.5, where  $\gamma_{i}^{*}$  is the model prediction,  $y_{i}$  is the label, and  $\nu_{i}^{*}$  is the importance weight for this data point.
Here  $\nu_{i}^{*} = \nu_{0} + (\sum_{b^{\prime}\in \mathcal{B}}k(y_{b},y_{b^{\prime}})p(y_{b^{\prime}}))^{-1 / 2}\cdot n_{i}$  where  $n_i$  represents the pseudo-count for the NIG distribution. Since  $(x_{i},y_{i})$  is a minority data point, data from its neighboring bins has smaller frequency  $p(y_{b^{\prime}})$  and therefore smaller  $\sum_{b^{\prime}\in \mathcal{B}}k(y_b,y_{b^{\prime}})p(y_{b^{\prime}})$ , leading to a larger importance weight  $\nu_{i}^{*}$  for this minority data point in Eqn.5.
This allows VIR to naturally put more focus on the minority data, thereby alleviating the imbalance problem.
D.9 Difference from DIR, VAE and DER
From a technical perspective, VIR is substantially different from any combinations of DIR [49], VAE [25], and DER [1]. Specifically,
- VIR is a deep generative model to define how imbalanced data are generated, which is learned by a principled variational inference algorithm. In contrast, DIR is a simply discriminative model (without any principled generative model formulation) that directly predict the labels from input. It is more prone to overfitting.
- DIR uses deterministic representations, with one vector as the final representation for each data point. In contrast, our VIR uses probabilistic representations, with one vector as the mean of the representation and another vector as the variance of the representation. Such dual representation is more robust to noise and therefore leads to better prediction performance.
- DIR is a deterministic model, while our VIR is a Bayesian model. Essentially VIR is equivalent to sampling infinitely many predictions for each input data point and averaging these predictions. Therefore intuitively it makes sense that VIR could lead to better prediction performance.
- Different from VAE and DIR, VIR introduces a reweighting mechanism naturally through the pseudo-count formulation in the NIG distribution (discussed in the paragraphs Intuition of Pseudo-Counts for VIR and From Pseudo-Counts to Balanced Predictive Distribution in the paper). Note that such a reweighting mechanism is more natural and powerful than DIR since it is rooted in the probabilistic formulation.
- Unlike for the standard VAE, the optimal prior (that maximizes ELBO) is known to be the aggregated posterior, our optimal prior is a neighbor-weighted version of aggregated posterior: for standard VAE, different data points contribute independently to the aggregated posterior; in contrast, for our VIR, the importance of each data point with respect to the aggregated posterior is affected by data points with neighboring labels.
- It is also worth noting that DIR cannot produce uncertainty estimation since it is a deterministic model. In contrast, Our VIR formulates a probabilistic deep generative model for imbalanced data, and therefore can naturally produce both more accurate predictions compared to DIR and better uncertainty estimation compared to DER.
D.10 Bayesian Neural Networks (BNNs)
We do not consider other BNNs in this work because:
- Weights in Bayesian Neural Networks (BNNs) are extremely high-dimensional; therefore BNNs have several limitations, including the intractability of directly inferring the posterior distribution of the weights given data, the requirement and computational expense of sampling during inference, and the question of how to choose a weight prior [1]. In contrast, evidential regression does not have these challenges.
- In our preliminary experiments, we found that typical BNN methods suffer from computational inefficiency and would require at least two to three times more computational time and memory usage. In contrast, evidential regression does not involve such computation and memory overhead; its overhead only involves the last (few) layers, and is therefore minimal.
- Additionally, as demonstrated in [27], Deep Ensemble typically performs as well as or even better than BNNs. Our method outperforms Deep Ensemble, therefore suggesting its superiority over typical BNN methods.
E.1 Ablation Study on VIR
In this section, we include ablation studies to verify that our VIR can outperform its counterparts in DIR (i.e., smoothing on the latent space) and DER (i.e., NIG distribution layers).
Ablation Study on  $q(\mathbf{z}_i|\{\mathbf{x}_i\}_{i=1}^N)$ . To verify the effectiveness of VIR's encoder  $q(\mathbf{z}_i|\{\mathbf{x}_i\}_{i=1}^N)$ , we replace VIR's predictor  $p(y_i|\mathbf{z}_i)$  with a linear layer (as in DIR). Table 14 shows that compared to its counterpart, FDS [49], our encoder-only VIR leads to a considerable improvements even without generating the NIG distribution. Both verify the effectiveness of our VIR's  $q(\mathbf{z}_i|\{\mathbf{x}_i\}_{i=1}^N)$ .
Ablation Study on  $p(y_{i}|\mathbf{z}_{i})$ . To verify the effectiveness of VIR's predictor  $p(y_{i}|\mathbf{z}_{i})$ , we replace VIR's encoder  $q(\mathbf{z}_i|\{\mathbf{x}_i\}_{i = 1}^N)$  with a simple deterministic encoder as in DER [1]. Table 14 and Table 15 show that compared to DER, the counterpart of VIR's predictor, our VIR's predictor still outperforms than DER, demonstrating its