SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Although our methods successfully improve both accuracy and uncertainty estimation on imbalanced regression, there are still several limitations. Exactly computing variance of the variances in Sec. 3.3 is challenging; we therefore resort to fixed variance as an approximation. Developing more accurate and efficient approximations would also be interesting
Table 6: Uncertainty estimation on STS-B-DIR.
Metrics	NLL ↓	AUSE ↓
Shot	All	Many	Medium	Few	All	Many	Medium	Few
DEEP ENS. [27]	3.913	3.911	4.223	4.106	0.709	0.621	0.676	0.663
INFER NOISE [29]	3.748	3.753	3.755	3.688	0.673	0.631	0.644	0.639
DER [1]	2.667	2.601	3.013	2.401	0.682	0.583	0.613	0.624
LDS + FDS + DER [1]	2.561	2.514	2.880	2.358	0.672	0.581	0.609	0.615
VIR (OURS)	1.996	1.810	2.754	2.152	0.591	0.575	0.602	0.510
OURS VS. DER	+0.671	+0.791	+0.259	+0.249	+0.091	+0.008	+0.011	+0.114
OURS VS. LDS + FDS + DER	+0.565	+0.704	+0.126	+0.206	+0.081	+0.006	+0.007	+0.105
We identify the problem of probabilistic deep imbalanced regression, which aims to both improve accuracy and obtain reasonable uncertainty estimation in imbalanced regression. We propose VIR, which can use any deep regression models as backbone networks. VIR borrows data with similar regression labels to produce the probabilistic representations and modulates the conjugate distributions to impose probabilistic reweighting on imbalanced data. Furthermore, we create new benchmarks with strong baselines for uncertainty estimation on imbalanced regression. Experiments show that our methods outperform state-of-the-art imbalanced regression models in terms of both accuracy and uncertainty estimation. Future work may include (1) improving VIR by better approximating variance of the variances in probability distributions, and (2) developing novel approaches that can achieve stable performance even on imbalanced data with limited sample size, and (3) exploring techniques such as mixture density networks [3] to enable multi-modality in the latent distribution, thereby further improving the performance.
The authors thank Pei Wu for help with figures, the reviewers/AC for the constructive comments to improve the paper, and Amazon Web Service for providing cloud computing credit. Most work is done when ZW is a master student at UMich. HW is partially supported by NSF Grant IIS-2127918 and an Amazon Faculty Research Award. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the sponsors.
[1] A. Amini, W. Schwarting, A. Soleimany, and D. Rus. Deep evidential regression. Advances in Neural Information Processing Systems, 33:14927-14937, 2020.
[2] J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of machine learning research, 13(2), 2012.
[3] C. M. Bishop. Mixture density networks. 1994.
[4] C. M. Bishop. Pattern recognition and machine learning. Springer, 2006.
[5] P. Branco, L. Torgo, and R. P. Ribeiro. Smogn: a pre-processing approach for imbalanced regression. In First international workshop on learning with imbalanced domains: Theory and applications, pages 36-50. PMLR, 2017.
[6] P. Branco, L. Torgo, and R. P. Ribeiro. Rebagg: Resampled bagging for imbalanced regression. In Second International Workshop on Learning with Imbalanced Domains: Theory and Applications, pages 67-81. PMLR, 2018.
[7] D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation, pages 1–14, 2017.
[8] B. Charpentier, O. Borchert, D. Zügner, S. Geisler, and S. Gunnemann. Natural posterior network: Deep bayesian predictive uncertainty for exponential family distributions. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[9] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
[10] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[11] Z. Deng, H. Liu, Y. Wang, C. Wang, Z. Yu, and X. Sun. PML: progressive margin loss for long-tailed age classification. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, 2021.
[12] Y. Gal and Z. Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1050-1059. JMLR.org, 2016.
[13] Y. Gong, G. Mori, and F. Tung. Ranksim: Ranking similarity regularization for deep imbalanced regression. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 7634-7649. PMLR, 2022.
[14] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321-1330. PMLR, 2017.
[15] A. K. Gupta and D. K. Nagar. Matrix variate distributions, volume 104. CRC Press, 2018.
[50] W. Yin, J. Zhang, O. Wang, S. Niklaus, L. Mai, S. Chen, and C. Shen. Learning to recover 3d scene shape from a single image. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 204-213. Computer Vision Foundation / IEEE, 2021.
[51] E. Zelikman, C. Healy, S. Zhou, and A. Avati. Crude: calibrating regression uncertainty distributions empirically. arXiv preprint arXiv:2005.12496, 2020.
[52] Z. Zhang, A. Romero, M. J. Muckley, P. Vincent, L. Yang, and M. Drozdzal. Reducing uncertainty in undersampled mri reconstruction with active acquisition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2049-2058, 2019.
Notation. As defined in Sec.3 in main paper, we partitioned  $\{Y_j\}_{j=1}^N$  into  $|\mathcal{B}|$  equal-interval bins (denote the set of bins as  $\mathcal{B}$ ), and  $\{Y_j\}_{j=1}^N$  are sampled from the label space  $\mathcal{V}$ . In addition, We denote the binary set  $\{P_i\}_{i=1}^{|\mathcal{B}|}$  as the label distribution (frequency) for each bin, i.e.,  $P_i \triangleq \mathbb{P}(Y_j \in \mathcal{B}_i)$ . We also denote the binary set  $\{O_i\}_{i=1}^N$  to represent whether the data  $\{\mathbf{x}_j, y_j\}$  are observed (i.e.,  $O_j \sim \mathbb{P}(O_j = 1) \propto P_{B(Y_j)}$ , and  $\mathbb{E}_O[O_j] \sim P_{B(Y_j)}$ ), where  $B(Y_j)$  represents the bin which  $(x_j, y_j)$  belongs to. For each bin  $i \in \mathcal{B}$ , we denote the global set of samples as
$$
\mathcal {S} _ {i} \triangleq \{j: O _ {j} = 1 \& B (Y _ {j}) = i \}.
$$
Definition A.1 (Expectation over Observability  $\mathbb{E}_O$ ). We define the expectation over the observability variable  $O$  as  $\mathbb{E}_O[\cdot ]\equiv \mathbb{E}_{O_j\sim \mathbb{P}(O_j = 1)}[\cdot ]$
Definition A.2 (True Risk). Based on the previous definitions, the true risk for our model is defined as:
where  $\delta_j(Y, \widehat{Y})$  refers to some loss function (e.g. MAE, MSE). In this paper we assume these loss is upper bounded by  $\Delta$ , i.e.,  $0 \leq \delta_j(Y, \widehat{Y}) \leq \Delta$ .
Then in next step we define the Naive Estimator.
Definition A.3 (Naive Estimator). Given the observation set, the Naive Estimator is defined as:
$$
\widehat {R} _ {\text {N A I V E}} (\widehat {Y}) = \frac {1}{\sum_ {i = 1} ^ {| \mathcal {B} |} | \mathcal {S} _ {i} |} \sum_ {i = 1} ^ {| \mathcal {B} |} \sum_ {j \in \mathcal {S} _ {i}} \delta_ {j} (Y, \widehat {Y})
$$
Considering an imbalanced dataset as a subset of observations from a balanced one, we contrast it with the Inverse Propensity Score (IPS) estimator [34], underscoring the superiorities of our approach.
Definition A.4 (Inverse Propensity Score Estimator). The inverse propensity score (IPS) estimator (an unbiased estimator) is defined as
IPS estimator is an unbiased estimator, as we can verify by taking the expectation value over the observation set:
$$
\begin{array}{l} \mathbb {E} _ {O} \left[ \widehat {R} _ {\mathrm {I P S}} (\widehat {Y} | P) \right] = \frac {1}{| \mathcal {B} |} \sum_ {i = 1} ^ {| \mathcal {B} |} \frac {1}{| \mathcal {U} _ {i} |} \sum_ {j \in \mathcal {U} _ {i}} \frac {\delta_ {j} (Y , \widehat {Y})}{P _ {i}} \cdot \mathbb {E} _ {O} [ O _ {j} ] \\ = \frac {1}{| \mathcal {B} |} \sum_ {i = 1} ^ {| \mathcal {B} |} \frac {1}{| \mathcal {U} _ {i} |} \sum_ {j \in \mathcal {U} _ {i}} \delta_ {j} (Y, \widehat {Y}) = R (\widehat {Y}). \\ \end{array}
$$
Definition A.5 (VIR Estimator). The VIR estimator, denoted by  $\widehat{R}_{\mathrm{VIR}}(\widehat{Y}|\widetilde{P})$ , is defined as:
$$
\widehat {R} _ {\mathrm {V I R}} (\widehat {Y} | \widetilde {P}) = \frac {1}{| \mathcal {B} |} \sum_ {i = 1} ^ {| \mathcal {B} |} \frac {1}{| \mathcal {U} _ {i} |} \sum_ {j \in \mathcal {S} _ {i}} \frac {\delta_ {j} (Y , \widehat {Y})}{\widetilde {P} _ {i}}, \tag {8}
$$
where  $\{\widetilde{P}_i\}_{i=1}^{|B|}$  represents the smoothed label distribution utilized in our VIR's objective function (see Eqn.5 in the main paper). It's important to note that our VIR estimator is biased.
Definition A.6 (Empirical Risk Minimizer). For a given hypothesis space  $\mathcal{H}$  of predictions  $\widehat{Y}$ , the Empirical Risk Minimization (ERM) identifies the prediction  $\widehat{Y} \in \mathcal{H}$  as
$$
\widehat {Y} ^ {\mathrm {E R M}} = \operatorname {a r g m i n} _ {\widehat {Y} \in \mathcal {H}} \left\{\widehat {R} _ {\mathrm {V I R}} (\widehat {Y} | \widetilde {P}) \right\}
$$
Lemma A.1 (Tail Bound for VIR Estimator). For any given  $\widehat{Y}$  and  $Y$ , with probability  $1 - \eta$ , the VIR estimator  $\widehat{R}_{\mathrm{VIR}}(\widehat{Y}|\widetilde{P})$  does not deviate from its expectation  $\mathbb{E}_O[\widehat{R}_{\mathrm{VIR}}(\widehat{Y}^{ERM}|\widetilde{P})]$  by more than
$$
\left| \widehat {R} _ {\mathrm {V I R}} (\widehat {Y} ^ {E R M} | \widetilde {P}) - \mathbb {E} _ {O} [ \widehat {R} _ {\mathrm {V I R}} (\widehat {Y} ^ {E R M} | \widetilde {P}) ] \right| \leq \frac {\Delta}{| \mathcal {B} |} \sqrt {\frac {\log (2 | \mathcal {H} | / \eta)}{2}} \sqrt {\sum_ {i = 1} ^ {| \mathcal {B} |} \frac {1}{\widetilde {P} _ {i} ^ {2}}}.
$$
Proof. For independent bounded random variables  $X_{1},\dots ,X_{n}$  that takes values in intervals of sizes  $\rho_1,\dots ,\rho_n$  with probability 1, and for any  $\epsilon >0$
$$
\mathbb {P} \left(\left| \sum_ {i} ^ {n} X _ {i} - \mathbb {E} \left[ \sum_ {i} ^ {n} X _ {i} \right] \right| \geq \epsilon\right) \leq 2 \exp \left(\frac {- 2 \epsilon^ {2}}{\sum_ {i} ^ {n} \rho_ {i} ^ {2}}\right)
$$
Consider the error term for each bin  $i$  in  $\widehat{R}_{\mathrm{VIR}}(\widehat{Y}^{ERM}|\widetilde{P})$  as  $X_{i}$ . Using Hoeffding's inequality, define  $\mathbb{P}(X_i = \frac{\delta_j(Y,\widehat{Y})}{\widetilde{P}_i}) = \widetilde{P}_i$  and  $\mathbb{P}(X_i = 0) = 1 - \widetilde{P}_i$ . Then, by setting  $\epsilon_0 = |\mathcal{B}| \cdot \epsilon$ , we are then able to show that:
$$
\begin{array}{l} \mathbb {P} \left(\left| | \mathcal {B} | \cdot \widehat {R} _ {\mathrm {V I R}} (\widehat {Y} ^ {E R M} | \widetilde {P}) - | \mathcal {B} | \cdot \mathbb {E} _ {O} [ \widehat {R} _ {\mathrm {V I R}} (\widehat {Y} ^ {E R M} | \widetilde {P}) ] \right| \geq \epsilon_ {0}\right) \leq 2 \exp \left(\frac {- 2 \epsilon_ {0} ^ {2}}{\Delta^ {2} \sum_ {i = 1} ^ {| \mathcal {B} |} \frac {1}{| \mathcal {U} _ {i} |} \sum_ {j \in \mathcal {U} _ {i}} \frac {1}{\widetilde {P} _ {i} ^ {2}}}\right) \\ \Longleftrightarrow \mathbb {P} \left(\left| \widehat {R} _ {\mathrm {V I R}} (\widehat {Y} ^ {E R M} | \widetilde {P}) - \mathbb {E} _ {O} [ \widehat {R} _ {\mathrm {V I R}} (\widehat {Y} ^ {E R M} | \widetilde {P}) ] \right| \geq \epsilon\right) \leq 2 \exp \left(\frac {- 2 \epsilon^ {2} | \mathcal {B} | ^ {2}}{\Delta^ {2} \sum_ {i = 1} ^ {| \mathcal {B} |} \frac {1}{\widetilde {P} _ {i} ^ {2}}}\right). \\ \end{array}
$$
We can then solve for  $\epsilon$ , completing the proof.
Theorem A.1 (Generalization Bound of VIR). In imbalanced regression with bins  $\mathcal{B}$ , for any finite hypothesis space of predictions  $\mathcal{H} = \{\widehat{Y}_1, \dots, \widehat{Y}_{\mathcal{H}}\}$ , the transductive prediction error of the empirical risk minimizer  $\widehat{Y}^{ERM}$  using the VIR estimator with estimated propensities  $\widetilde{P}(P_i > 0)$  and given training observations  $O$  from  $\mathcal{V}$  with independent Bernoulli propensities  $P$ , is bounded by:
$$
R \left(\widehat {Y} ^ {E R M}\right) \leq \widehat {R} _ {\mathrm {V I R}} \left(\widehat {Y} ^ {E R M} | \widetilde {P}\right) + \frac {\Delta}{| \mathcal {B} |} \sum_ {i = 1} ^ {| \mathcal {B} |} \left| 1 - \frac {P _ {i}}{\widetilde {P} _ {i}} \right| + \frac {\Delta}{| \mathcal {B} |} \sqrt {\frac {\log (2 | \mathcal {H} | / \eta)}{2}} \sqrt {\sum_ {i = 1} ^ {| \mathcal {B} |} \frac {1}{\widetilde {P} _ {i} ^ {2}}} \tag {9}
$$
Proof. We first re-state the generalization bound for our VIR estimator: with probability  $1 - \eta$ , we have
$$
R (\widehat {Y} ^ {E R M}) \leq \widehat {R} _ {\mathrm {V I R}} (\widehat {Y} ^ {E R M} | \widetilde {P}) + \frac {\Delta}{| \mathcal {B} |} \sum_ {i = 1} ^ {| \mathcal {B} |} \left| 1 - \frac {P _ {i}}{\widetilde {P} _ {i}} \right| + \frac {\Delta}{| \mathcal {B} |} \sqrt {\frac {\log (2 | \mathcal {H} | / \eta)}{2}} \sqrt {\sum_ {i = 1} ^ {| \mathcal {B} |} \frac {1}{\widetilde {P} _ {i} ^ {2}}}
$$
We start to prove it from the LHS:
$$
\begin{array}{l} R (\widehat {Y} ^ {E R M}) = R (\widehat {Y} ^ {E R M}) - \mathbb {E} _ {O} [ \widehat {R} _ {\mathrm {V I R}} (\widehat {Y} ^ {E R M} | \widetilde {P}) ] + \mathbb {E} _ {O} [ \widehat {R} _ {\mathrm {V I R}} (\widehat {Y} ^ {E R M} | \widetilde {P}) ] \\ = \underbrace {\operatorname {b i a s} (\widehat {R} _ {\text {V I R}} (\widehat {Y} ^ {E R M} | \widetilde {P}))} _ {\text {B i a s T e r m}} + \underbrace {\mathbb {E} _ {O} [ \widehat {R} _ {\text {V I R}} (\widehat {Y} ^ {E R M} | \widetilde {P}) ] - \widehat {R} _ {\text {V I R}} (\widehat {Y} ^ {E R M} | \widetilde {P})} _ {\text {V a r i a n c e T e r m}} + \widehat {R} _ {\text {V I R}} (\widehat {Y} ^ {E R M} | \widetilde {P}) \\ = \underbrace{\text{bias}(\widehat{R}_{\text{VIR}}(\widehat{Y}^{ERM}|\widetilde{P}))}_{\text{BiasTerm}} + \Bigg|\underbrace{\widehat{R}_{\text{VIR}}(\widehat{Y}^{ERM}|\widetilde{P}) - \mathbb{E}_{O}[\widehat{R}_{\text{VIR}}(\widehat{Y}^{ERM}|\widetilde{P})]}_{\text{VarianceTerm}}\Bigg| + \widehat{R}_{\text{VIR}}(\widehat{Y}^{ERM}|\widetilde{P}) \\ \end{array}
$$
Below we derive each term:
Variance Term. With probability  $1 - \eta$ , the variance term is derived as
$$
\begin{array}{l} \mathbb {P} \left(\left| \widehat {R} _ {\mathrm {V I R}} \left(\widehat {Y} ^ {E R M} | \widetilde {P}\right) - \mathbb {E} _ {O} \left[ \widehat {R} _ {\mathrm {V I R}} \left(\widehat {Y} ^ {E R M} | \widetilde {P}\right) \right] \right| \leq \epsilon\right) \geq 1 - \eta \\ \Longleftrightarrow \mathbb {P} \left(\max  _ {\widehat {Y} _ {j}} \left| \widehat {R} _ {\mathrm {V I R}} (\widehat {Y} | \widetilde {P}) - \mathbb {E} _ {O} [ \widehat {R} _ {\mathrm {V I R}} (\widehat {Y} | \widetilde {P}) ] \right| \leq \epsilon\right) \geq 1 - \eta \\ \Longleftrightarrow \mathbb {P} \left(\bigvee_ {\widehat {Y} _ {j}} \left| \widehat {R} _ {\mathrm {V I R}} (\widehat {Y} | \widetilde {P}) - \mathbb {E} _ {O} [ \widehat {R} _ {\mathrm {V I R}} (\widehat {Y} | \widetilde {P}) ] \right| \geq \epsilon\right) <   \eta \\ \Longleftrightarrow \mathbb {P} \left(\bigcup_ {\widehat {Y} _ {j}} \left| \widehat {R} _ {\mathrm {V I R}} (\widehat {Y} | \widetilde {P}) - \mathbb {E} _ {O} [ \widehat {R} _ {\mathrm {V I R}} (\widehat {Y} | \widetilde {P}) ] \right| \geq \epsilon\right) <   \eta \\ \Longleftarrow \sum_ {j} ^ {| \mathcal {H} |} \mathbb {P} \left(\left| \widehat {R} _ {\mathrm {V I R}} (\widehat {Y} | \widetilde {P}) - \mathbb {E} _ {O} [ \widehat {R} _ {\mathrm {V I R}} (\widehat {Y} | \widetilde {P}) ] \right| \geq \epsilon\right) <   \eta (10) \\ \Longleftarrow | \mathcal {H} | \cdot 2 \exp \left(\frac {- 2 \epsilon^ {2} | \mathcal {B} | ^ {2}}{\Delta^ {2} \sum_ {i = 1} ^ {| \mathcal {B} |} \frac {1}{| \mathcal {U} _ {i} |} \sum_ {j \in \mathcal {U} _ {i}} \frac {1}{\widetilde {P} _ {i} ^ {2}}}\right) <   \eta (11) \\ \Longleftrightarrow | \mathcal {H} | \cdot 2 \exp (\frac {- 2 \epsilon^ {2} | \mathcal {B} | ^ {2}}{\Delta^ {2} \sum_ {i = 1} ^ {| \mathcal {B} |} \frac {1}{\widetilde {P _ {i} ^ {2}}}}) <   \eta , \\ \end{array}
$$
where inequality (10) is by Boole's inequality (Union bound), and inequality (11) holds by Lemma A.1. Then, by solving for  $\epsilon$ , we can derive Variance Term that with probability  $1 - \eta$ ,
$$
\begin{array}{l} \mathbb {E} _ {O} \left[ \widehat {R} _ {\mathrm {V I R}} \left(\widehat {Y} ^ {E R M} | \widetilde {P}\right) \right] - \widehat {R} _ {\mathrm {V I R}} \left(\widehat {Y} ^ {E R M} | \widetilde {P}\right) \leq \left| \widehat {R} _ {\mathrm {V I R}} \left(\widehat {Y} ^ {E R M} | \widetilde {P}\right) - \mathbb {E} _ {O} \left[ \widehat {R} _ {\mathrm {V I R}} \left(\widehat {Y} ^ {E R M} | \widetilde {P}\right) \right] \right| \\ \leq \frac {\Delta}{| \mathcal {B} |} \sqrt {\frac {\log (2 | \mathcal {H} | / \eta)}{2}} \sqrt {\sum_ {i = 1} ^ {| \mathcal {B} |} \frac {1}{\widetilde {P} _ {i} ^ {2}}}. \\ \end{array}
$$
Bias Term. By definition, we can derive:
$$
\begin{array}{l} \operatorname {b i a s} \left(\widehat {R} _ {\mathrm {V I R}} \left(\widehat {Y} ^ {E R M} | \widetilde {P}\right)\right) = R \left(\widehat {Y} ^ {E R M}\right) - \mathbb {E} _ {O} \left[ \widehat {R} _ {\mathrm {V I R}} \left(\widehat {Y} ^ {E R M} | \widetilde {P}\right) \right] \\ = \frac {1}{| \mathcal {B} |} \sum_ {i = 1} ^ {| \mathcal {B} |} \frac {1}{| \mathcal {U} _ {i} |} \sum_ {j \in \mathcal {U} _ {i}} \delta_ {j} (Y, \widehat {Y} ^ {E R M}) - \frac {1}{| \mathcal {B} |} \sum_ {i = 1} ^ {| \mathcal {B} |} \frac {1}{| \mathcal {U} _ {i} |} \sum_ {j \in \mathcal {U} _ {i}} \frac {P _ {i}}{\widetilde {P} _ {i}} \delta_ {j} (Y, \widehat {Y} ^ {E R M}) \\ \leq \frac {\Delta}{| \mathcal {B} |} \sum_ {i = 1} ^ {| \mathcal {B} |} \frac {1}{| \mathcal {U} _ {i} |} \sum_ {j \in \mathcal {U} _ {i}} \left| 1 - \frac {P _ {i}}{\widetilde {P} _ {i}} \right| \\ = \frac {\Delta}{| \mathcal {B} |} \sum_ {i = 1} ^ {| \mathcal {B} |} \left| 1 - \frac {P _ {i}}{\widetilde {P} _ {i}} \right|, \\ \end{array}
$$
concluding the proof for the Bias Term, hence completing the proof for the whole generalization bound.
B Details for Experiments
Datasets. In this work, we evaluate our methods in terms of prediction accuracy and uncertainty estimation on four imbalanced datasets², AgeDB [30], IMDB-WIKI [33], STS-B [7], and NYUD2-DIR [35]. Due to page limit, the results for NYUD2-DIR [35] are in the supplementary. We follow
2Among the five datasets proposed in [49], only four of them are publicly available.
Table 7: Accuracy on AgeDB-DIR. For baselines, we directly use the reported performance in their paper and therefore do not have error bars.
Metrics	MSE↓	MAE↓	GM↓
Shot	All	Many	Medium	Few	All	Many	Medium	Few	All	Many	Medium	Few
VANILLA [49]	101.60	78.40	138.52	253.74	7.77	6.62	9.55	13.67	5.05	4.23	7.01	10.75
VAE [25]	99.85	78.86	130.59	223.09	7.63	6.58	9.21	13.45	4.86	4.11	6.61	10.24
DEEPENSEMBLE [27]	100.94	79.3	129.95	249.18	7.73	6.62	9.37	13.90	4.87	4.37	6.50	11.35
INFER NOISE [29]	119.46	95.02	149.84	266.29	8.53	7.62	9.73	13.82	5.57	4.95	6.58	10.86
SMOTER [40]	114.34	93.35	129.89	244.57	8.16	7.39	8.65	12.28	5.21	4.65	5.69	8.49
SMOGN [5]	117.29	101.36	133.86	232.90	8.26	7.64	9.01	12.09	5.36	4.9	6.19	8.44
SQINV [49]	105.14	87.21	127.66	212.30	7.81	7.16	8.80	11.2	4.99	4.57	5.73	7.77
DER [1]	106.77	91.29	122.43	209.69	8.09	7.31	8.99	12.66	5.19	4.59	6.43	10.49
LDS [49]	102.22	83.62	128.73	204.64	7.67	6.98	8.86	10.89	4.85	4.39	5.8	7.45
FDS [49]	101.67	86.49	129.61	167.75	7.69	7.10	8.86	9.98	4.83	4.41	5.97	6.29
LDS + FDS [49]	99.46	84.10	112.20	209.27	7.55	7.01	8.24	10.79	4.72	4.36	5.45	6.79
RANKSIM [13]	83.51	71.99	99.14	149.05	7.02	6.49	7.84	9.68	4.53	4.13	5.37	6.89
LDS + FDS + DER [1]	112.62	94.21	140.03	210.72	8.18	7.44	9.52	11.45	5.30	4.75	6.74	7.68
VIR (OURS)	81.76±0.10	70.61±0.05	91.47±1.50	142.36±2.10	6.99 ±0.02	6.39±0.02	7.47±0.04	9.51±0.06	4.41±0.03	4.07±0.02	5.05±0.03	6.23±0.05
OURS VS. VANILLA	+19.84	+7.79	+47.05	+111.38	+0.78	+0.23	+2.08	+4.16	+0.64	+0.16	+1.96	+4.52
OURS VS. INFER NOISE	+37.70	+24.41	+58.37	+123.93	+1.54	+1.23	+2.26	+4.31	+1.16	+0.88	+1.53	+4.63
OURS VS. DER	+25.01	+20.68	+30.96	+67.33	+1.10	+0.92	+1.52	+3.15	+0.78	+0.52	+1.38	+4.26
OURS VS. LDS + FDS	+17.70	+13.49	+20.73	+66.91	+0.56	+0.62	+0.77	+1.28	+0.31	+0.29	+0.40	+0.56
OURS VS. RANKSIM	+1.75	+1.38	+7.67	+6.69	+0.03	+0.10	+0.37	+0.17	+0.12	+0.06	+0.32	+0.66
Table 8: Accuracy on IMDB-WIKI-DIR. For baselines, we directly use the reported performance in their paper and therefore do not have error bars.
Metrics	MSE↓	MAE↓	GM↓
Shot	All	Many	Medium	Few	All	Many	Medium	Few	All	Many	Medium	Few
VANILLA [49]	138.06	108.70	366.09	964.92	8.06	7.23	15.12	26.33	4.57	4.17	10.59	20.46
VAE [25]	137.98	108.62	361.74	964.87	8.04	7.20	15.05	26.30	4.57	4.22	10.56	20.72
DEEP ENSEMBLE [27]	138.02	108.83	365.76	962.88	8.08	7.31	15.09	26.47	4.59	4.26	10.61	21.13
INFER NOISE [29]	143.62	112.26	373.19	961.97	8.11	7.36	15.23	26.29	4.68	4.33	10.65	20.31
SMOTER [40]	138.75	111.55	346.09	935.89	8.14	7.42	14.15	25.28	4.64	4.30	9.05	19.46
SMOGN [5]	136.09	109.15	339.09	944.20	8.03	7.30	14.02	25.93	4.63	4.30	8.74	20.12
SQINV [49]	134.36	111.23	308.63	834.08	7.87	7.24	12.44	22.76	4.47	4.22	7.25	15.10
DER [1]	133.81	107.51	332.90	916.18	7.85	7.18	13.35	24.12	4.47	4.18	8.18	15.18
LDS [49]	131.65	109.04	298.98	829.35	7.83	7.31	12.43	22.51	4.42	4.19	7.00	13.94
FDS [49]	132.64	109.28	311.35	851.06	7.83	7.23	12.60	22.37	4.42	4.20	6.93	13.48
LDS + FDS [49]	129.35	106.52	311.49	811.82	7.78	7.20	12.61	22.19	4.37	4.12	7.39	12.61
RANKSIM [13]	125.30	102.68	299.10	777.48	7.50	6.93	12.09	21.68	4.19	3.97	6.65	13.28
LDS + FDS + DER [1]	120.86	97.75	297.64	873.10	7.24	6.64	11.87	23.44	3.93	3.69	6.64	16.00
VIR (OURS)	118.94±1.10	96.10±0.80	295.79±1.20	771.47±3.10	7.19±0.03	6.56±0.03	11.81±0.04	20.96±0.05	3.85±0.04	3.63±0.05	6.51±0.03	12.23±0.03
OURS VS. VANILLA	+19.12	+12.6	+70.3	+193.45	+0.87	+0.67	+3.31	+5.37	+0.72	+0.54	+4.08	+8.23
OURS VS. INFER NOISE	+24.68	+16.16	+77.40	+190.50	+0.92	+0.80	+3.42	+5.33	+0.83	+0.70	+4.14	+8.08
OURS VS. DER	+14.87	+11.41	+37.11	+144.71	+0.66	+0.62	+1.54	+3.16	+0.62	+0.55	+1.67	+2.95
OURS VS. LDS + FDS	+10.41	+10.42	+15.7	+40.35	+0.59	+0.64	+0.8	+1.23	+0.52	+0.49	+0.88	+0.38
OURS VS. RANKSIM	+6.36	+6.58	+3.31	+6.01	+0.31	+0.37	+0.28	+0.72	+0.34	+0.34	+0.14	+1.05
the preprocessing procedures in DIR [49]. Details for each datasets are to the Supplement, and details for label density distributions and levels of imbalance are discussed in DIR [49].