Making Reliable and Flexible Decisions in Long-tailed Classification
Authors: Bolian Li
Ruqi Zhang
Venue: Accepted by TMLR ()

Abstract: Long-tailed classification is challenging due to its heavy imbalance in class probabilities. While existing methods often focus on overall accuracy or accuracy for tail classes, they overlook a critical aspect: certain types of errors can carry greater risks than others in real-world long-tailed problems. For example, misclassifying patients (a tail class) as healthy individuals (a head class) entails far more serious consequences than the reverse scenario. To address this critical issue, we introduce Making Reliable and Flexible Decisions in Long-tailed Classification (RF-DLC), a novel framework aimed at reliable predictions in long-tailed problems. Leveraging Bayesian Decision Theory, we introduce an integrated gain to seamlessly combine long-tailed data distributions and the decision-making procedure. We further propose an efficient variational optimization strategy for the decision risk objective. Our method adapts readily to diverse utility matrices, which can be designed for specific tasks, ensuring its flexibility for different problem settings. In empirical evaluation, we design a new metric, False Head Rate, to quantify tail-sensitivity risk, along with comprehensive experiments on multiple real-world tasks, including large-scale image classification and uncertainty quantification, to demonstrate the reliability and flexibility of our method.

Published in Transactions on Machine Learning Research (01/2025)
Making Reliable and Flexible Decisions in Long-tailed Clas- sification
Bolian Li
k44688purdue.co edu rugizSpardue.co edu
Ruqi Zhang
Deportment of Computer Science, Purdue University West Lafayette, IN 47907, USA
Reviewed on OpenReview: https: // openreview. net/ formm? id= id=
Abstract
Long-tailed classification is challenging due to its heavy imbalance in class probabilities. While existing methods often focus on overall accuracy or accuracy for tail classes, they overlook a critical aspect: certain lypes of errors can carry grealer risks than others in real- world long-tailed problems. For example, misclassifying patients (a tail class) as healthy individuals (a head class) entails far more scrious consoquences than the reverse scenario. To address this critical issue, we introduce Making Reliable and Flexible Decisions in Long- tailed Classification (RF-DLC), novel framework aimed at reliable predictions in long- tailed problems. Leveraging Bayesian Decision Theory, introduce integrated gain to scamlessly combine long-tailed data distributions and the decision-makinn makinn procedure. We further propose an elficient variational optimization strategy for the decision risk objective. Our method adapts readily to diverse utility matrices, which can be designed for specific tasks, ensuring its ficxibility for different problem settings. In empirical ovaluation, we design a new metric, False Head Rate, quantify tail-sensitivity risk, along with compre- hensive experiments multiple real-world tasks, including large-scale image classification and uncertainty quantification, to demonstrate the reliability and flexibility of our method."
1 Introduction
Real-world categorical data is often long-tailed distributed, where the data distributions are biased towards few "head" classes and the "tail" classes have much fewer samples (Reed, 2001; Lin et al., 2014; Van Horn & Perona, 2017; Krishna et al., 2017; Liu et al., 2019b; Wang et al., 2020; Li et al., 2022). The long-tailed problem primarily stems from the inherent biases in the data collection process, which are challenging to avoid. Models trained on long-tailed data using conventional methods often exhibit notable performance drops compared to those trained on balanced datasets (Wang et al., 2022).
Existing approaches to long-tailed tailed classification, such as re-weighting loss (Lin et al., 2017; Cao et al., 2019; Cui et al., 2019; Wu et al., 2020), logit adjustment (Menon et al., 2020; Ren et al., 2020; Hong et al., 2021), and knowledge transfer (Liu et al., 2019b; Xiang et al., 2020), usually focus on improving overall evaluation metries or the metrics for tail classes. However, in real-world long-tailed data, the misprediction penalty between different classes often depends on their semantic meaning (He et al., 2024), and the ultimate objective is making optimal docisions rather than optimizing certain metrics. The probability of misclassifying tailed samples head samples is very high in existing methods. For example, in disease detection where healthy individuals belong to head classes and patients belong to tail classes, the risk of classifying patients healthy individuals is significantly larger than the reverse scenario (Yang et al., 2022). In autonomous driving, mispredicting tail classes like pedestrians or cyclists as head classes like vehicles will significantly increase the risk of accidents and injuries (Carranza-Garoia et al., 2021).
'httpe///github.ee wwt.a//ccccc/laaa//eeoooo

---

Published in Transactions on Machine Learning Research (01/2025)
To enable reliable long-tailed classification and simultancously integrate decision-makingg we propose Making Reliable and Flexible Decisions in Long-tailed Classification (RF-DI.C), general learning framework aimed at reliable predictions on diverse realistic long-tailed problems. Specifically, we introduce the integrated gain from Bayesian Decision Theory (Robert et al., 2007; Berger, 2013), an important branch of cost-sonsitive learning, which allows us to seamlessly incorporate decision risk and long-tailed data distributions in a single objective. While cost-sensitive learning has been mainly applied to standard classification (Elkan, 2001; He & Garcia, 2009; Thai-Nghe et al., 2010; Chung et al., 2016; Shu et al., 2019), its application to long- tailed classification is limited to the case of avoiding misclassifying tail samples head. We also propose variational optimization strategy to efficiently maximize the integrated gain w.r.t. model parameters. With "utility matrix" as a part of the objective, our method is flexible to many real-world tasks with different lypes of risks. In our empirical evaluations, we design a new metric, False Head Rate (FHR), lo quantity the mispredictions from tail classes to head classes a common source of high risk in long-tailed classification (Sengupta et al., 2016; Rahman ct al., 2021; Yang al., 2022).
The main contributions of this paper are summarized as follows:
RE-DLC is the first to consider decision-making in long-tailed classification. Built. upon Bayesian Decision Theory, RF-DLC enables optimal decision-making on long-tailed data.
RF-DLC introduces a new objective called integrated gain, an efficient variational optimization strategy, and several utility matrices tailored for different long-tailed scenarios. All of these are directly derived from Bayesian Decision Theory, ensuring that they are integral to the model's function.
RE-DLC is flexible and can be adapted to various lasks with diverse metrics. Users can adopt RF-DLC to many specific fields by re-designing utility matrices.
We conduct comprehensive experiments to demonstrate that RF-DLC significantly improves decision-making while maintaining or improving traditional metrics such as accuracy and calibration.
2 Related Works
Long-tailed Classification. Previous methods mainly tackle long-tailed classification from the following aspects: i) adjusting data distributions to obtain balanced datasets, including over-sampling (Han et al., 2005), under-sampling (Liu et al., 2008), and data augmentation (Chu et al., 2020; Kim et al., 2020; Liu et al., 2020)3; ii) re-balancing the importance of different classes in loss functions, including re-weighting loss (Lin et al., 2017; Mahajan et al., 2018; Cao et al., 2019; Cui et al., 2019; Menon et al., 2020; Wu et al., 2020) and logit adjustment (Menon et al., 2020; Ren et al., 2020; Hong et al., 2021); iii) applying heterogeneous model architectures to handle head and tail samples in different ways, including OLTR (Liu et al., 2019b), LFME (Xiang et al., 2020), RIDE (Wang et al., 2020), TLC (Li et al., 2022), and SRepr (Nam et al., 2023). Other attempts explore broader Long-tailed classifications, including non- non-uniten uniform testing distributions (Zhang et al., 2022), outlier samples (Wang et al., 2022; Bai et al., 2022), and partial-labeled datasets (Hong et al., 2022). Our method is siguificantly different from existing methods, targeting at decision maaiiionnnniingggg and asymmetric misprediction risks, which are important problems in realistic long-tailed data.
Bayesian Decision Theory and Cost-sensitive Learning. Robert al. (2007); Berger (2013) have comprchensively introduced Bayesian Decision Theory, which enables optimal decisions under diverse prob- lem settings (via the choices of utility functions). For its adaptation to deep neural networks, multiple ways have been explored: i) loss-calibrated variational inference, including Loss-calibrated EM (Lacoste-Julien et al., 2011), LCVB (Jaiswal el al., 2020), and variational inference on continuous utilities (Kušmierezyk et al., 2019); ii) loss-calibrated expectation propagation (Morais & Pillow, 2022). Bayesian Decision Theory is also one of the main methodologics to solve cost-scnsitive learning (Elkan, 2001; Ling Sheng, 2008; Chung al., 2016), which focuses on pocision-making under heterogeneous misprediction costs. Standard
2The words "prediction" " and "decision" " are used interchangeably throughout this paper. 3rhhese hods fall behind the SOTA of long-tailed classification significantly, and thus are not compared the experiments.
2

---

Published in Transactions on Machine Learning Research (01/2025)
cost-sensitive learning under expected loss minimization (Pires et al., 2013) overlooks the fact that long- tailed training data distribution may be distributed differently from the testing distribution. Our method is built upon Bayesian Decision Theory and focuses on the specific problem of long-tailed classification, which has not been explored in previous works. We include a detailed comparison with cost-sensitive learning in Appendix F.3.
3 Background
This section provides the necessary background knowledge on long-tailed classification and Bayesian Decision Theory for understanding our method.
Long-tailed Distributions. In Long-tailed categorical data, the training and testing sets follow different distributions (Moreno- (Moreno-Toree et al., 2012; Cui et al., 2019; Liu et al., 2019b). Specifically, the training data is distributed in a descending manner over categorics in terms of class probability:
ply kn) ply2 k2); if la S k2,
(1)
while most existing works (Cui et al., 2019; Cao et al., 2019; Wang et al., 2020; Li et al., 2022) assume that the testing data distribution is uniform over categories: ply k)) = ply = k2) for all pairs of (k1, The distributional shift. between training and testing data makes long-tailed problems extremely challenging.
Bayesian Decision Theory. To make optimal decisions in diverse problem scttings, Bayesian Deci- sion Theory provides principled framework (Robert et al., 2007; Berger, 2013). In the supervised set- ting, for dataset D {X,Y) Y {(x,, 9:) and 0-parameterized model, denote the likeli- I P(y x., 0) and p(0) respectively. The posterior distribution is then p(0)D) p(0) IT P(y: x;, 0) gained by choosing decision d for input æ when the model with parameters O controls the mapping from x to y. Subsequently, the posterior expected gain for an input æ is defined as
hood and prior to be IT,
N
p(y:)z;) We further assume a decision gain g(d to quantify the utility
G(d|æ, D) == Ea-p(0 0),
(2)
where average over all possible models weighted by their posterior probabilities. Previous works (Lacoste- Julien et al., 2011; Cobb et al., 2018) naively maximize Eq. 2 w.r.t. 0 to obtain a caalbrree posterior (0),, which does not consider non-uniform data distribtions.
4 Methodology
In this section, we introduce RF-DLC, a framework aimed at optimal dleiison-mmkknn for long-tailed data, grounded in Bayesian Decision Theory. The conventional posterior expected gain in Eq. implicitly assumes that both training and testing data share the same distribution, which is violated in long-tailed problems. To address this challenge we adopt the integrated gain from Bayesian Decision Theory, which incorporates data distributions into the posterior expected gain:
G(d) = FF (xx
G(d x, D) = (e s.). (x) Feee G(d
(3)
where G(d 0) II s(d x., 0) is the decision gain over the entire dataset and [d1, ..., dN is the decision vector. In the long-tailed setting, naturally want the model to fit the testing distribution. Therefore, the integrated gain used in method is defined as:4
G(d) = E wt).....x..... uw J (z. G(dd 0).
Eq. is our training objective, integrating the data distributions and the decision gain into a single function. To use this objective long-tailed data, there are four main challenges left to address: i) Section 4.1: how compute the decision gain g(d|e,, 0)? ii) Section 4.2: how to fit the long- tailed training distributions? iii) Section 4.3: how to learn the intractable posterior distribution D)? iv) Section 4.4: how to make optimal decisions during testing?
4The notion Prest (I. y) the testing distribution and will be discussed in Section 4.2.
3

---

Published in Transactions on Machine Learning Research (01/2025)
4.1 How to Design the Decision Gain?
To quantify the utility of making decision, we define the decision gain as:
gdd 0) II Ily' x o)
(5)
Here, d) refers to the utility function which scores the decision when the true label is y/ The utility values play a critical role in re-weighting the likelihoods of different decisions to prioritize particular mispredictions and allow us to encode human knowledge and expertise tailored to specific tasks.
The thcoretical foundation of utility function has been comprehensively studied in Robert et al. (2007): Berger (2013). For example, Chapter 2.2 of Robert al. (2007) guarantecs the existence of utility functions with rational decision-makers. Note that our design of the decision gain differs from previous work like Cobb et al. (2018), which uses g' d'dd) x, 0) == En Uyra PPy 0). Both definitions achieve the goal of averaging the pre- dictive probability pyy æ θ) weighted by the utility values. However, Eq. 5 has two advantagos: i) Stability: Eq. is more stable for training. After laking the logarithm, Eq. becomes Ey Uy ,dw)) logr(y) |ae, 0) which is a weighted average of the log probabilities, while Cobb et al. (2018) becomes log Ey (y)) which is a log of a weighted average of the probabilities and not commonly used in classification problems. fi) Flexibility: Eq. 5 allows for more general and flexible utility values whereas Cobb et al. (2018) requires the utility matrix U to be positive definite (otherwise we may not be able to compute the logarithm). Due to these reasons, we use Eq. 5.
((()) p(y'|a, 0).
Once we have defined the decision gain g(]],, 0), the next step is designing the utility function w(y'. .d. For classification tasks, we can employ a utility matrix U:
m(0,0)
(0 1)
u(0, pM ul,, DT)
w11,0)
w (1,1)
U
(6)
u(D') 0) 1())
" (DDD)
where [ is the number of classes and Uy u(y i, = j) is the utility score assigned to the case of predicting class as j. The utility matrix serves as the task-specific knowledge and is pre-defined before training. Users can assign negative values to discourage certain mispredictions and positive values to en- courage the desired ones, which reflects their knowledge about specific tasks. To illustrate how design utility matrices for different long-tailed problem settings, we provide practical examples in Fig. For an in-depth discussion utility designing, please refer to Chapter 2 of Robert et al. (2007).
Standard Classification. Standard classification on long-tailed data can be regarded as special case in our framework, where the overall accuracy is the most decisive metric in evaluation. In this case, the focus lies solely on determining whether the decision aligns with the ground truth (i.e., y = d). As shown in Fig. 1(a), a simple onc-hot utility can be defined by u(y',d) = 1gy d}, which corresponds to the standard accuracy metric.
Tail-sensitive Classification. Tail-class samples often have high importance due to their scarcity. Mis- predictions from tail classes to head classes usually induce severe consequences in real-workd tasks, such as activity recognition (Rahman et al., 2021) and medical images (Yang et al., 2022). In these domains, dan- gerous actions and illnesses are often scarce yet profoundly harmful neglected. Besides, the lack of training samples in tail classes has been empirically proved to be the bottleneck of classification performance (Li et al., 2022). Therefore, the ratio of false head samples in evaluation can often reflect model's real-world potential. To this end, tail-sensitive utility matrix can be defined by adding extra penalties on false head mispredictions, as shown in Fig. 1(b). The tail-sensitive utility matrix encourages the model to predict uncertain samples as tail rather than head, without affecting correct predictions made with confidence.
Class/ / -esta-class-onnsitivee Classification. In certain applications where semantic differences exist be- tween different categories, preventing mispredictions between specific (meta) classes becomes crucial, regard- less of their class probabilities within the long-tailed distribution. For example, object detection systems
4

---

Published in Transactions on Machine Learning Research (01/2025)
decision
docktion
dacistion 2
dacision ]
decision
decition 0
deciston un
decisiin
decision 3
daciaian
mamma!
matmmal
4
plane
Eap
bird
tricck
plane
Car
Bird
Imick
label 0-
1.0
0.0
0.0
0.0
0.0
label 0
1.0
0.0
0.0
0.0
0.0
planc
1.0
0.0
0.0
0.0
pkme
1.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
-0.5
0.0
0:0
100
0.0
0.0
1.0
0.0
45
9.5
label
0.0
0.0
1.0
0.0
0.0
label
-0.7
0.5
1.0
0.0
0.0
bird
11.0
0.0
1.0
0.0
0.0
60
bird
0.0
0.0
1.0
0.0
0.0
4.
label 3-
0.0
0.0
0.0
1.0
0.0
Jobel 3
0.99
00.7
00.5
1.0
0.0
mummal
0.0
0.0
0.0
1.0
0.0
-0.3
mammmnl
0.0
1.00
0.0
1.0
Jabel
label 4
truck
truck
-10
-10
-10
(a) Onc-hot utility
(b) Tail-sensitive utility
(c) Class-sensitive utility
(d) Mctaclass-scnsittvv utility
Figure 1: Examples of utility matrices, designed for (a) standard and (b) tail-sensitive classifications, along
with (c) class-sensitive classification for bird-plane misprediction and (d) meta-class-sensttive classification
for mammal-vehicle misprediction. The entries in the matrix reflect the risk levels (-1 is the most risky) and can be flexibly assigned based on task requirements.
airports must avoid misclassifying birds as plancs to ensure flight safcty (Shi et al., 2021), and autonomous driving systems must prevent misclassifying mamunals on the road as vehicles (both are meta-classes includ- ing various animal and vehicle types) to ensure driving safety (Yudin et al., 2019). In these contexts, we provide two examples in Fig. I(c) and (d) to illustrate how utility values can be assigned to prevent the specific mispredictions that are of primary concern.
These utility matrices enable us to customize decision-making based on the particular objectives and chal- lenges in different long-tailed classification scenarios, improving the reliability and flexibility in real-world applications.
4.2 How to handle the Distribution Shift?
In long-tailed classification, there exists a distribution shift between the training (long-tailod) and testing (uniform) sets. We denote the data distributions of the training and lesting sets as Ptrain (z, y) and Ptest (az, y) respectively. Since all models are aimed to perform well on the testing set, the data distribution in Eq. should be the testing distribution (x, y). To address the discrepancy between the training and testing distributions, importance sampling (Kloek & Van Dijk, 1978) is adopted:
/ Perain(10 (10,1) v) Piest (F. v (x, v) v)d(a, v)
(( (xx Prest (x. Va, v)d(ar v) =
Puraio (ar, y)
(7)
Prest (x, 1)
()( (2.,1) Purain (x. y)
V (x. y):
The ratio Prest (z, v)/Puaai(# (x. y) explicitly shows the discrepancy between training and testing data. One common assumption in long-tailed data is that distributional differences only cxist between classes (Hong et al., 2021). Formally, this assumption can be expressed as follows:
where V(x,y) denole any possible function, which will be specified in Section 4.3.
Assumption 1 (Intra-class Consistency). The distributional differences only exist between classes. Given fared class label, the dala distribulions are the same for training and lesting dala: Pevain (a:ly) Prest (ar (yly)
Based on Assumption 1, the discrepancy ratio can be further simplified:
Ptest (x, y) Piest Prest (x) y) Ptest (y) Perain (at. v) Perain (v) Perain (aly) Perain (v))
(8)
which only depends on the class probabilities of training and testing data. Given that the testing set is assumedd to be uniform, the probability Prest (y) would be a constant, and thus the ratio is equivalent to:
Prest (y)
Perain()) (y) x f(ny)'
(9)
Perain (y)
Trhis assumption widely adopted previous works (Hong al., 2021; Ren et al., 2022).
5

---

Published in Transactions on Machine Learning Research (01/2025)
Algorithm 1: RF-DLC
Inputs: Dataset D = (X,Y) (( w)) initial particles {0 utility matrix U, the step size f);
Results: Final particles (0) )
for each iteration do Xs, Y2 + A mini-batch sampled from D; Lz(g, d = Y=) f- Loss computed by Eq. 10; for j = 1,.., .M do
0, + 8 n Vo,LE
/* Update g(0)
end
end
where f()) is an increasing function and ny refers to the number of samples the class y. We introducc the notion of f(ny) because the class probability only depends on the number of samples in this class. Many existing re-weighting methods in long-tailed classification can be regarded as special instances of f(-). For example, (my nž is the most conventional choice with sensitivity factor y to control the importance of head classes (Huang et al., 2016; Wang et al., 2017; Pan al., 2021); f(ny) (1- B0v)/(1 - B) is the effective number which considers data overlap (Cui et al., 2019). A detailed analysis of the choice of f()) in RF-DLC is conducted in Section 5.6.
4.3
How to Learn the Posterior Distribution?
The posterior distribution p(0 in the integrated gain (Eq. 4) is generally intractable. To learn the posterior, we use a variational distribution z(0) to approximate the true posterior p(0)D). Specifically, we find a tractable lower bound for the logarithm the integrated gain:
log G(d=
Y) Y) log ((...)))...... MN Jrppe (2.y) BB Be- G(d= G(d=
Σ Uv log x 0) + log p(yi(oi 0) 0))
E (a ( Bang(o)
-L(((())l((O) C
(10)
N
~
Σ 1=1
Eg E (0) f(n
1
Σ Un log (r 0) log |x, 0)
(I(((0))b(0)) +C
= =lg,d=Y)
) is a constant. Here, the second equation obtained using Jensen's inequality and the third equation obtained using the importance sampling discussed Section 4.2 as well as one-sample Monte Carlo (MC) approximation of the expectation over data points (x),).). nt). The detailed derivation of this lower bound is similar to previous works (Lacoste-Julien al., 2011; Cobb et al., 2018) and is proved in Appendix C. In the lower bound, set d; Y4 since the true label is considered to be the optimal decision at the training stage. By maximizing L(g, d = Y) w.r.t. 9, we obtain good approximation for the postcrior (D)
where - log ( X)
Relationship with Standard Variational Inference. On the relationship between our method and standard variational inference (Jordan et al., 1999), clear similarity is the notion of variational distribution g(0), which is expected to approximate posterior distribution. However, due to the distributional shift in long-tailed data, we should also specify whether the posterior distribution is based on training or testing distribution, which are denoted as (arai and (a)s((( respectively. We show below that the lower bound L contains the objective for standard variational inference:
T(g, d = Y) E E ( (eN sepe (2. Bong(o) (0) log G(d Y 0) KL(g(0) |Prest (0)D)).
(11)

---

Published in Transactions on Machine Learning Research (01/2025)
where the approximation is due to the use of MC approximation. If further assuming the utility matrix to be one-hot (i.e., Ug = (y = d]), the equation can be further simplified as:
L(g, -u,d=Ya = Y) (KL( Prestt (t
(12)
The proof is provided in Appendix D.
Particle-based Variational Distribution. To pursue the efficiency of posterior inference, we construct the variational distribution using particles (Liu & Wang, 2016; D' D'Angelo & Fortuin, 2021):
(0))
Σ j=l
wy 8(0 -,),
(13)
where {w, } are normalized weights which hold Σj=1 wy = 1, and 8(-) is the Dirac delta function. The "particles" " {0 ensemble methods in the context. of long-tailed data (Wang et al., 2020; Li et al., 2022). Our framework provides theoretical foundation for these approaches long-tailed problems: due the scarcity of tailed data, there is not enough evidence to support single solution, leading to many equally good solutions (which give complementary predictions) in the loss landscape. Therefore, estimating the full posterior distribution is essential to get a comprehensive characterization of the solution space. Particle optimization reduces the cost of Bayesian inference and is more efficient than variational inference (Blundell et al., 2015) and Markov chain Monte Carlo (MCMC) (Brooks et al., 2011), especially on the high- high-dimensional and multimodal deep neural network posteriors.
JM are implemented through ensemble models. Empirical studies have previously explored
The integrated gain optimization in Eq. 10 includes regularization term KL(q(0)] p(0)), which keeps g be close to the prior p(0). If we assume the prior p(0) to be Gaussian distribution, the regularization can be extended to:
Repulsive Regularization.
(T)(())))())) = > A 10022-000000 + 1 (()) log a (0)d0
X
10,12 H()), Σ 3=1
M
(14)
M
where À a constant, θ is the parameter space and ()) is the entropy of O. The first term (I2- regularization) prevents the model from over-fitting and the second term (entropy) applies a repulsive force to individual models to promote their diversity, pushing the particles away from each other (Liu & Wang, 2016; introduce simple approximation:
D' D'Angelo & Fortuin, 2021). To make the entropy term computable, (0)) ~ log [Eol, where Σg is the covariance matrix estimated by particles. Other entropy approximations can also be used. By the technique of SWAG-diagonal covariance (Maddox et al., 2019), the covariance ma- trix can then be dircctly computed by: E. diag(0" Φ²). Overall, the regularization term a combination of L2 weight decay and the repulsive force and the final form of this regularization term is:
M
(L(((0)llp())
X
10,11
Σ
log ( -0²)
O²)x
(15)
M
2
which is different from existing diversity regularization used in long-tailed classification (Wang et al., 2020; Li et al., 2022). The repulsive regularization our method is naturally derived from the integrated gain optimization, with strong theoretical motivation.
4.4 How to make optimal decisions during testing?
Following the standard Bayesian Decision Theory, we maximize the logarithm of the posterior expected gain G(d" D) in (2) for each testing input xt to obtain optimal decisions:
d' == arg max log G(d (D) arg max
Σ
Σ
LVaa Ug logp(u') ",,)'
(16)
1=1
7

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1: False Head Rate evaluation for tail-sensitive long-tailed classification (%). Three datasets and three tail region settings are considered. Our method consistently outperforms all basclines across all settings.
CB Loss
LDAM
RIDE
TLC
Dataset
Tail Ratio
CE
RF-DLC
(Cui et al., 2019) (Cao et al., 2019) (Wang et al., 2020) (Li et. al., 2022)
25%
21.10 + 0.43
14.84 + 0.93
10.05 + 1.01
8,94 = 0.66
10.42 + 0.64
4.99 0.32
50%
37.87 + 0.57
27.98 + 1.44
19.64 + 1.66
17.80 + 1.39
20.27 = 0.77
11.76 + 0.29
CIFARIO-LT
75%
48.75 1 1.39
33.93 1 1.60
21.37 1 2.10
19.77 1 3.20
22.24 1 1.53
11.01 1.28
avcrage
35.91 1 0.54
25.58 1 1.27
17.02 1 1.56
15.50 1 1.68
17.64 1 0.93
9.25 1: 0.49
25%
45.53 + 1.54
24.88 + 0.34
21.22 + 0.99
18.83 + 0.70
21.18
15.39
50%
73.03 1.59
48,41 1.24
43.04 1.18
39.50 1.53
31.34 0.55
CIFARI00-L1'
75%
91.30
1.24
74.38 + 1.47
65.62 + 1.31
62.01
1.03
49.51 1:45
average
69.95
1.40
49.22
43.29 + 1.04
1.61
0.55
25%
3.99 + 0.08
3.66 + 0.17
0.12
0.13
2.70
50%
12.77 0.29
11.80 0.12
12.73 0.28
11.42
1.14
9.68
0.25
ImageNet-LT
75%
30.99
0.40
29.39 + 0.28
29.90 0.41
26.92 - 0.33
27.12
24.42 0.19
avcrage
15.92 1 0.17
14.95 1 0.15
15.60 0.21
13.99 0.24
14.03
12.27 0.08
Table 2: Class-sensitive long-tailed classification on CIFAR10-LT, comparing the baseline one-hot utility with class/mo http://aaaaaaasssssstite utility. The specifically designed utilitics effectively improve tailored metrics with negligible drops in standard overall accuracy.
Task
Metric
Evaluation Score (%)
ACC (%) t
plaae Duiied
bird-plane detection
Rplane
5.10 + 3.80 (-25.5%)
84.11 - 83.84 ( (-0.3%)
Wirrd
vehicle-mammal detection
Rmammall
P Punhicle mammmall
1.40 - 0.35 (-75.5%)
84.11 - 83.83 (-0.3%)
vechiele
where the approximation is due to q()) R p(O)D). Importantly, for symmetric utility matrices (e.g,, one-hot utility in Fig. 1(a)), Eq. 16 can be further simplified to d' x arg max, E5= podp(d 0,), which aligns with the standard ensemble models.
In summary, the integrated gain enables method to simultaneously consider the posterior distribution, preision-making (utility matrix), and data distribution. It provides principled way to address more realistic problems on the long-tailed data. The proposed RF-DLC is summarized in Algorithm 1.
5 Experiments
In this section, we show the experimental results to demonstrate the effectiveness of our method. We use CIFAR10/100-LT (Cui et al., 2019), ImageNet-IT (Liu al., 2019b), and iNaturalist (Van Horn et al., 2018) the long-tailed datasets, and compare our method with multiple long-tailed baselines. Detailed implementation is summarized in Appendix E.
5.1 Tail-sensitive
Classification with False Head Rate
As discussed in Section 4.1, mispredictions from tail classes to head classes generally pose higher risks and quantifying the likelihood of such occurrences is crucial. Inspired by the false positive rate, we define the False Head Rate (FHR) follows:
Phead O Grail Grall
FHR
(17)
where Grail is the set of samples that are labeled as lail classes and Phoad is the set of samples that are predicted as head classes. To consider different tail regions, we select the last 25%, 50%, and 75% classes as tail classes, respectively. We apply the tail-sensitive utility in Fig. 1(b) our method. From Table 1, we observe substantial improvements over all baselines across all settings, especially on the relatively small CIFAR R datasets, which means that the False Head Rate is more challenging on small datasets.
8

---

Published in Transactions on Machine Learning Research (01/2025)
Table 3: Top-1 overall accuracy and tail-class accuracy on standard long-tailed classification (%). Our
method outperforms all baselines on both metrics.
CIFARIO-LT
CIFARI0O-LT
ImageNet-LT
Method
Tail
All
Tail
All
Tail
LAt (Menon et al., 2020)
77.67
43.89
55.
ACE| (Cai et al., 2021)
81.2
49.4
23.5
54.7
SRepri (Nam et al., 2023)
82.06 1 0.01
47.81
1: 0.02
23.31 1: 0.11
52.12
0.06
32.14 1: 0.41*
Cb
+ 0.39
38.51
38
0.52
47.80 0.15
CB Loas (Cui et sl., 2019)
77.62 + 0.69
68.73 1.52
42.24 0.41
20.50 + 0.51
51.70 0.20
18.29 = 0.41
LDAM (Cao et al., 2019)
80.63 + 0.69
77.14 1.61
43 = U.67
23.50 + 1.28
31.01
0.20 0.21
17:21
0.22
RIDE (Wang et al., 2020)
83.11 1. 0.52
79.62
1.56
48.99
0.44
28.78 1: 1.52
54.32 0.54
50.74
0.62
TTC (1. et al., 2022)
79.70 + 0.65
76.39
0.98
48.75
- 0.16
28.40
+ 0.72
55.03 + 0.34
51.56
0.35
RF-DLC (ours)
83.75 + 0.17
82.33
1.16
50.24
+ 0.70
30.34 1.49
55.73
0.17
51.98 + 0.40
Table 4: Calibration evaluation of different uncertainty algorithms. The Bayesian predictive uncertainty adoptod in our method outperforms other algorithms.
CIFARIO-LT
CIFARI00-LT
ImageNet-LT
Uncertainty Algorithm
AUC (%) f
ECE (%)
AUC (%) f
ECE (%)
AUC (%) f
ECE (%)
MCP (Hendrycks & Gimpel, 2017)
79.98 + 0.10
14:33 = 0.37
80.48
23.75=
84.02 0.22
18.35 = 0.12
Evideutial (Li et al., 2022)
83.20 + 0.59
13.24 = 0.55
0.55
77.37 + 0.33
21.64
81.45 + 0.13
15.29 = 0.12
Bayesian (RF-DLC)
86.83 = 0.68
9.84 + 0.17
81.24 + 0.25
10.35 + 0.28
84.45 = 0.09
8.72 + 0.13
5.2 Class-sensitive Long-tailed Classification
We further try the (meta) class sensitive cases on CIFAR1O-LT, including bird-plane detection (Shi et al., 2021) and vehiele-mammal detection (Yudin et al., 2019), as discussed in Section 4.1. The utility matrices used for these two tasks are Fig. 1 (c) and (d) respectively, and their full formats are shown in Appendix B. We also design two correspouding metrics for accurate evaluation, and list the results in Table 2. We again observe that our method improves significantly on the metrics with negligible drops in standard accuracy. These real-world tasks show the importance of taking the decision loss into account and also demonstrate the flexibility of our method which is compatible with different utilities, leading to better performance for different lypes of lasks.
5.3 Standard Long-tailed Classification
We evaluate the overall accuracy and tail-class accuracy. The results are shown in Table 3, where means the results are directly copied from the original papers and * is due to different setting of tail classes We apply the one- hot utility in our method. In particular, Our method significantly outperforms other baselines on the crucial tailed data. The results on iNauuaaist are summarized in appendix F. These results demonstrate the reliability of our mothod in standard long-tailed classification.
5.4 Uncertainty Quantification
In our method, the predictive uncertainty can be naturally obtained by the entropy of predictive distri- bution (Malinin & Gales, 2018). For the compared uncertainty estimation algorithms, MCP is a trivial baseline that obtains uncertainty scores from the maximum value of softmax distribution (Hendrycks & Gimpel, 2017); evidential uncertainty rooted the subjective logic (Jsang, 2018), and introduced to long-tailed classification by Li al. (2022). We evaluate the three uncertainty algorithms in Table Our Bayesian predictive uncertainty outperforms the other two and has remarkable advantage the ECE metric, demonstrating the superiority of using principled Bayesian uncertainty quantification. The uncer- tainty results separated by class regions are in Appendix F.2, where the superiority of Bayesian predictive uncertainty persist on all class regions.
6gRepr (Nam al., 2023) has different ImageNet settings from other baselines, with fewer samples the tailed classes. Therefore, their tail-class accuracy is much lower.
9

---

Published in Transactions on Machine Learning Research (01/2025)
Table 5: Experimental results on long-tailed medical image classification, evaluated on DermaMNIST (Yang
et al., 2023). RF-DIC outperforms other basclines on this real-world application.
ACC
(%) +
FHR.
(%) 4
Method
AUC (%) t ECE (%) +
All
Head
Med
Tail
avg
25%
50%
75%
CE
69.30
85.51
50.17
38.51
82.30
24.31
47.00
30.68
52.71
57 61.61
Focal Loss (Lin et al., 2017)
71.75
86.08
51.60
40.57
79.14
16.53
40.08
25.68
46.14
48.41
CB Loss (Cui et al., 2019)
73.89
84.41
53.27
43.99
75.15
27.81
36.73
26.73
38. 14
45.32
LDAM (Cao et al., 2019)
72.82
82.62
53.71
43.15
81.66
17.61
39.35
24.48
47.33
46.24
RF-DLC (our)
77.67
84.65
56.17
44.07
82.90
13.80
28.46
19.00
35.59
30.77
Table 6: Comparison of different utilities in terms of False Head Rate on CIFAR100-LT, The tail-sensitive utility can effectively improve FHR with negligible drop in standard overall accuracy.
FHR (%) @tail ratio
Utility 25%
Better (%)
ACC (%)
Worse (%)
50%
75%
average
one-hol
18.55 1 0.38
38.62 1 0.62
60.17 1 1.48
39.12 1 0.72
49.91 + 0.33
18.00
0.04
tail-sensitive
15.39 + 0.57
31.34 + 0.55
49.51 + 1.45
32.08 + 0.78
49.89 0.19
5.5 Medical Image Classification
Our method targets at real-world applications where long-tail problems exists. We also conduct long- tailed medical image classification experiments in Table 5, where our RF-DLC successfully recognize different dis- case types and outperforms compared baselines. The experiments are based on ResNet32, and the number of particles is set to 3. The DermaMNIST dataset (Yang al., 2023) is originally an imbalanced classi- fication dataset with the imbalance ratio to be around 60. The images are resized to 32x32, and other hyperparameters are the same as our CIFAR experiments.
5.6 Ablation Studies
Effect of utility. The effect tail-sensitive utility is shown in Table 6. We compare the one-hot and tail- sensitive utilities terms of False Head Rate and standard overall accuracy. By applying the tail-sensitive utility, the performances on FHR can be significantly improved (18.00%) with negligible drop in standard accuracy (0.04%). Besides, we also observe that the performance is relatively robust to the specific values of utility as long the sign is correct, which is detailed in Appendix F.4.
Forms of class probability. We compare five different forms of f fmyy in terms of standard overall accuracy in Table 7 and Fig. 2a. We also analyze the weight values (i.e., 1/ 1/f(")) and their growth rates bet ween the first and the last class across different forms. We find that as the growth rate becomes larger, ACC will be better accordingly, which suggests a high level of class imbalance in the dataset.
Fig. 2a shows similar results on the relationship between growth rate and the tail-class accuracy. As the growth rate bocomes larger, the tail and med accuracy will both become significantly better despite the slight drop in head accuracy, which is consistent with the improvement in overall accuracy. Based on these results, suggest using f(ny) ny in general.
Number of particles. Generally, using more individual models will induce better performances. However, we also need to balance the performance with the computational cost. We visualize accuracy under different numbers of particles in Fig. 2b. The error bars are scaled to be within two standard deviations. The accuracy curves are all logarithm-like and the accuracy improvement is hardly noticeable for more than 6 particles. However, the computational cost is increasing at a linear speed. Therefore, we recommend using no more than six particles practice for a desirable performanco-cost trade-off.
Repulsive force. We also evaluate the effectiveness of the repulsive force. The repulsive force effectively pushes the particle-based variational distribution to the target posterior and avoids collapsing into the same solution. Therefore, with the repulsive force, better predictive distributions can be learned, and thus better
10

---

Published in Transactions on Machine Learning Research (01/2025)
Table Comparison of different forms of class probabilities. CIFAR100-LT, The linear f(ny) is the most suitable form in terms of standard accuracy.
Top-1 standard accuracy evaluated on
Weight Value
Form of f(ny)
ACC (%) t
first class
last class
growth (%)
linear (Wang et al., 2017) effective number (Cui et al., 2019)
fhy (1- (1-8"x)/(1) B)
0.0020
0.1667
8250
50.17 + 0.25
0.0023
0.1669
7297
49.90 = 0.36
sqrt (Pan et al., 2021)
0.0447
0.4082
814
47.03 = 0.30
log constant
log ny
0. 0.1609
0.5581
247
45.26 = 0.51
C
1.0000
1.0000
0
43.27 0.30
linear
72-
effective number
sqrt
log
68-
constunt
ACC (%)
ACC (%)
44
All
Head
64
40
55.0
30
52.5
50.0
27.5
15.0
Mod
Tail
0
22.5
Head
Med
Tail
Class region
Number particles
(b) Different numbers of particles Figure 2: Ablation studies on CIFAR100-LT, Comparing (a) different forms of class probabilities and (b)
(a) Different forms of f(ny)
varying numbers of particles in terms of overall accuracy and the accuracy of class regions.
predictive uncertainty can be obtained. Besides, the repulsive force can also improve the overall accuracy by promoting the diversity of particles. We show the ablation study on repulsive force in Table 8. Increasing the repulsive force weight is generally beneficial to the uncertainty quantification, but may hurt the ACC when À is too large.
Comparison with ensembles of single-model methods. For fair comparison, we include the results on ensembles of single-model long-tailed classification methods, shown in Table 9. Ensembling single-model methods can reduce the performance gap between these methods and our RF-DLC, but will not outperforms RF-DLC.
6 Conclusion and Limitations
This paper proposes Making Reliable and Flexible Decisions in Long-tailed Classification (RF-DLC) address docision-making in realistic long-tailed problems. We focus on the problem of asymmotric mis- prediction cost in real-world long-tailed classification, and derive novel decision-making framework from Bayesian Decision Theory. Specifically, introduce the integrated gain to naturally incorporate the dis- tributional shift in long-tailed data, and leverage the utility matrix to make flexible decisions for various task settings. In empirical evaluations, we propose a new False Head Rate metric to quantify the particular type of misprediction from tail classes to head classes, along with large-scale image classification and un- certainty quantification experiments. The evaluation results demonstrate the superiority of our method on both standard and decision-critical long-tailed classifications.
However, believe there still some limitations for future developments. We list few limitations below:
11

---

Published in Transactions on Machine Learning Research (01/2025)
The effect repulsive force, compared by calibration on CIFARIOO-LTT. Applying an appropriate level of repulsive force will induce better predictive uncertainty.
Table 8:
Metric
A
À
x 10
== 10-5
10
À
10
5
10
ACC (%)
50.15
50.19
50.20
50.24
49.96
48.18
AUC (%) t
75.94
78.96
79.25
81.24
81.80
77.51
ECE (%) +
13.40
13.31
12.45
10.35
10.17
10.79
Table 9: Comparison with ensembles of single-model methods. Ensembling can only reduce the performance gap between single-model basclines and RF-DLC.
ACC
(%) f Med
FHR.
(%)
Method
AUC (%)
ECE (%) 4
All
Head
Tail
avg
25%
50%
75%
3xCB Loss (Cui al., 2019)
47.37
67.72
49.16
25.28
80.67
17.39
45.56
21.88
45.41
69.38
3xLDAM (Cao al., 2019)
48.84
68.59
47.87
28.47
80.28
19.27
39.96
19.22
40.04
60.62
RF-DLC (3 particles)
50.24
69.92
51.07
30.34
81.24
10.35
32.08
15.39
31.34
49.51
Long-tailed Regression. We have not explored long-tailed problems in regression, where the distribution of targets can also be highly imbalanced. However, with adjustments to the decision gain, we believe our framework can be adapted for regression tasks as well.
Dataset Shift. We have not accounted for general dataset shift scenarios, such as out-of-distribution data, where the assumption of semantically identical training and testing sets becomes invalid. Another example the distribution testing data. is no longer assumed to be uniform, the discrepancy ratio Ptess (x, y) Preain (2, y) will not be expressed as 1/ 1/f(ny), but rather in a more general form.
Broader Impact Statement
As general framework for long-tailed classification, our method is cligible for handling imbalanced data in many real-world applications. For example, the strategy adopted in our method can promote fairness and improve performance, especially for the underrepresented demographic groups. Besides, the use of utility functions allows for considering specific groups and raising their importance. We believe that adapting our method to realistic social problems is an interesting and important direction. We will investigate this direction in future works.
References
Jianhong Bai, Zuozhu Liu, Hualiang Wang, Jin Hao, Yang Feng, Huanpeng Chu, and Haoji Hu. On the effec- Liveness oul-of-distribution data in self-superiieed long-tail learning. In International Conference on Learning Representations, 2022.
James O Berger. Statistical decision theory and Bayesian analysis. Springer Science Business Media, 2013.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra, Weight uncertainty in neural net wetwork. International Conference on Machine Learning, pp. 1612-1622. PMLR, 2015.
Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng. Handbook of markou chain monte carlo. CRC press, 2011.
Jiarui Cai, Yizhou Wang, and Nenq Hwang. Ace: Ally complementary experts for solving long-tailed recognition in one-shot. In Proceedings of the /CV/ International Conference on Computer Vision, pp. 1211, 2021.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasels with label- distribution-aware margin loss. Advances in neural information processing systems, 32, 2019.
Manuel Carranza- Carranza-Garcia, Pedro Lara-Benítez, Jorge Garefa-Cuttérrez, and José C Riquelme. Eubancing object detection for autonomous driving by optimizing anchor generation and addressing class imbalance. Neurocompuling. 449:229 244, 2021.
12

---

Published in Transactions on Machine Learning Research (01/2025)
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long-tailed data, In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 28, 2020, Proceedings, Part XXIX 16, pp. 394-710. Springer, 2020.
Yu-An Chung, Hsuan-Tien Lin, and Shao-Wen Wen Yang. Cost-aware pre-training for multiclass cost-sensitive deep learning. In Proceedings of the Twenty-Fiith International Joint Conference on Artificial Intelligence, pp- 1411- 1417, 2016.
Adam Cobb, Stephen J Roberts, and Yarin Gal. cass-aalbbatee approximate inference in bayesian neural networks. arXiv preprint arXix:1805. 099009 2018.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9268-9277, 2019.
Francesco D' D'Angelo and Vincent Fortuin. Annealed stein variational gradient descent. In Third Symposium Aduances Approzimate Bayesion Fnference, 2020.
Francesco D' D'Angelo and Vincent Fortuin. Repulsive deep ensembles are bayesian. Advances Neural Information Processing Systems, 34:3451-3465, 2021.
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pp. 1-15. Springer, 2000.
Charles Flkan. The foundations cost-sensitive learning. International joint conference on artificial intelligence, volume 17, pp. 973 978-978. Lawrence Erlbaum Associates Ltd, 2001.
Chuanxing Geng, Sheng-jun Huang, and Songcan Chen. Recent advances in open set recognition: A survey, IEEE transactions on pallern analysis and machine intelligence, 43(10):3614 3631, 2020.
Hui Han, Wen- Yuan Wang, and Bing- Huan Mao. Borderline-smote: a new over-sampling method in imbalanced data sets learning. In International conference intelligend compuling, pp. 878 887. Springer, 2005.
Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, (9)))1266 1284, 2009.
Yu-Cheng He, YianXiang Ding, Han-Jia Ye, and Zhi-Hua Zhou. Learning only when it matters: Cost-aware long- tailed classification. In Proceedings of the A AAT Conference on Artificial Intelligence, volume 38, pp- 12411-12420, 2024.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribation examples in neural networks. In International Conference on Leurning Representations, 2017.
Feng Hong, Jiangchao Yao, Zhihan Zhou, Ya Zhang, and Yanfeng Wang. Long-tailed partial label learning via dynamic rebalancing. In International Conference on Learning Representations, 2022.
Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disentangling label distribution for long-tailed visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6626 6636, 2021.
Minghui Hu, AK Malik, M Tanveer, and PN Suganthan. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 115:105151, 2022.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoon Tang. Learning deep representation for imbalanced classifica- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5375-5384, 2016.
Prateek Jaiswal, Harsha Honnappa, and Vinayak A Rao. Asymptotic consistency of loss coss-calibrated variational bayes. Stat, 9(1):e258, 2020.
Johan Ludwig William Valdemar Jensen. Sur les fonctions convexes les inégalités entre les valeurs moyennes. Acta mathematica, 30(1):175 193, 1906.
Michael Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine learning, 37:183 133,3333,, 233, 1999.
13

---

Published in Transactions on Machine Learning Research (01/2025)
Audun Jsang. Subjective Logie: A formalism for reasoning under uncerlainty. Springer Publishing Company, Incor- porated, 2018.
Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major-to-minor translation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognitton, pp. 13896 13395,, 2020.
Tuen Kloek and Herman K Van Dijk. Bayesian estimates of equation system parameters: an application of integration by monte carlo. Econometrica: Journal of the Econometric Society, pp. 1 1-19, 1978.
Anna Korba, Adil Salim, Michael Arbel, Giulia Luise, and Arthur Gretton. A non-asymptotic analysis for stein variational gradient descent. Advances in Neural Information Processing Systems, 33:4672-488 -6822 2020.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32 73, 2017.
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig, Being bayesian, even just a bit, fixes overconfidence in relu networks. In International Conference Machine Learning, pp. 5436 5146. PMLR, 2020.
Tomasz Kušmierczyk,, Joseph Sakaya, and Arto Klami. Variational bayesian decision-making for continuous utilities. Advances Neural Information Processing Systems, 32, 2019.
Simon Lacoste-Julien, Ferenc Huszár, and Zoubin Ghahramani. Approximate inference for the loss-calibrated bayesian. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 416-424. JMLR Workshop and Conference Proceedings, 2011.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Aduances in Neural Information Processing Systems, 30, 2017.
Bolian Li, Lougbo Han, Haining Li, Huazhu Fu, and Changqing Zhang. Trustworthy longetailed classification. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pp. 6970-6979, 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740 755. Springer, 2014.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp- 1980-2988, 2017.
Charles X Ling and Victor S Sheng. Cost-sensitive learning and the class imbalance problem. Encyclopedia of machine learning, 2011 2011:231-235, 2008.
Chang Liu, Jingwei Zhuo, Pengyu Cheng, Ruiyi Zhang, and Jun Zhu, Understanding and accelerating particle- based variational inference. In International Conference on Machine Learning, pp. 4082-4092, PMLR, 2019a.
Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. Deep representation learning on long-tailed data: A learnable embedding augmentation perspective. In Proceedings of the JEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 2970 2979, 2020.
Qiang Liu and Dilin Wang. Stein variational gradient desoent: A general purpose bayesian inference algorithm. Advances in neural information processing systems, 29, 2016.
Xu-Ying Liu, Jianxin Wi, and Zhi-Hua Zhou. Exploratory undersampling for class-imbalance learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 39(2):539-55) 550, 2008
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the BEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2537-2546, 2019b.
Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining, In Proceedings of the European conference on computer vision (ECCV), pp. 181-196, 2018.
14

---

Published in Transactions on Machine Learning Research (01/2025)
Andrey Malinin and Mark Gales. Predictive uncertainly estimation via prior networks. Advances in Neural Infor- mation Processing Systems, 31, 2018
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference Learning Representations, 2020.
Michael Morais and Jonathan W Pillow. Loss-calibrated expeetatioottntntttttiiioon propagation for approximate bayesian decision- making. Xix preprint Nr:::2201 003128, 2022.
Jose G Torres, Troy Raeder, Rocío Alaiz- Rodriguee, Nitesh V Chawla, and Francisco Herrera. A unifying view on dataset shift in classification. Pattern recognition, 45(1):5221-55 2012.
Giung Nam, Sunguk Jang, and Juho Lee. Decoupled training for long-tailed classification with stochastic represen- tations. In International Conference on Learning Representations, 2023.
Tai-Yu Pan, Cheng Zhang, Yandong Li, Hexiang Hu, Dong Xuan, Soravit Changpinyo, Boqing Gong, and Wei- Lun Chao. On model calibration for long-tailed object detection and instance segmentation. Advonces in Newral Information Processing Systems, 34:2529 2542, 2021.
Bernardo Avila Pires, Csaba Szepesvari, and Mohammad Ghavamzadeh, Cost-sensitive multiclass classification risk bounds. In International Conference on Machine Learning, pp. 1391 1391-1399. PMLR, 2013.
Arafat Rahmann Iqbal Hassan, and Md Atiqur Rahman Ahad. Nurse care activity recognition: cost-sensilive ensemble approach to handle imbalanced class problem in the wild. In Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitons Computing and Proceedings of the 2021 ACM Inter- national Symposium on Wearable Computers, pp. 440-445 2021.
William J Reed. The pareto, zipf and other power laws. Economics letters, 74(1):15-19, 2001.
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long-tailed visual recog- nition. Advances in Neural Information Processing Systems, 33:4175-4186, 2020.
Jiawei Ren, Mingyuan Zhang, Cunjun Yu, and Ziwei Liu. Balanced mse for imbalanced visual regression. In Proceedings of the IEEE/CVE Conference Computer Vision and Pottern Recognátion, pp. 7926 7935, 2022.
Christian P Robert al. The Bayesion choice: from decission-thoorele foundalions to compulational implementatiin,
volume 2. Springer, 2007.
Soumyadip Sengupta, Jun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa, and David W Jacobs. Frontal to profile face verification in the wild. In 2016 IEEE winter conference on opplications of computer vision (WACV), pp. 1-9. IEEE, 2016.
Xiaohang Shi, Jun Hu, Xueyue Lei, and Shiyou Xu, Detection of flying birds in airport monitoring based on improved yolov5. In 2021 6th International Conference on Intelligent Computing and Signal Processing (ICSP), pp. 1446- 1451. IEEE, 2021.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight-net Learning an explicit mapping for sample weighting. Advances neural information processing systems, 32, 2019.
Nguyen Thai-Nghe, Zeno Gantner, and Lars Schmidd- Thiemme. Cost-sensitive learning methods for imbalanced data. In The 2010 International joint conference on neural networks (IJCNN), pp. 1-8. TERE, 2010.
Grant Van Horn and Pietro Perona. The devil is in the tails: Fine-grained classification in the wild. arXiv preprint arXiv: 709.0090000 709.00199000.. 01450, 2017.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognstion, pp. 8769-8778, 2018
Haotao Wang, Aston Zhang, Yi Zhu, Shuai Zheng, Mu Li, Alex J Smola, and Zhangyang Wang. Partial and asym- metric contrastive learning for out-of-distributio detection in long-tailed recognition. In International Conference on Machine Learning, pp. 23/16 23158. PMLR, 2022.
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu. Louged recognition by routing diverse distribation-awaree experts. In International Conference on Learning Representations, 2020.
15

---

Published in Transactions on Machine Learning Research (01/2025)
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. Adcances in Neural Information Processing Systems, 30, 2017.
Tong Wu, Qingqiu Huang, Ziwei Liu, Yu Wang, and Dahua Lin. Distribution-bllacee loss for multi- classification in long-tailed datasets. In European Conference on Computer Vision, pp. 162-178. Springer, 2020.
Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification. In European Conference on Computer Vision, pp. 2/7 263. Springer, 2020.
Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):11, 2023.
Zhixiong Yang, Junwen Pan, Yanzhan Yang. Xiaozhou Shi, Hong-Yu Yu Zhou, Zhicheng Zhang, and Cheng Bian. Proco: Prototype-aware contrastive learning for loilet medical image classification. In International Conference Medical Image Computing and Computer-Assisteee Intervention, pp. 173-182. Springer, 2022.
Dmitry Yudin, Anton Sotnikov, and Andrey Krishtopik. Detection big animals on images with road scenes using deep learning. In 2019 International Conference on Artificial Intelligence: Applications and Innovations (IC-ATAT), pp. 100-1003. IEFE, 2019.
Yifan Zhang, Bryan Hooi, HONG Lanqing, and Jiashi Feng. Selfssppeviised aggregation of diverse experts for Lest-aguostic long-tailed recognilion. In Advances in Neural Information Processing Systems, 2022.
16

---

Published in Transactions on Machine Learning Research (01/2025)
A Related Model Architectures
The model architecture of our method an ensemble of multiple individual models. This architecture belongs to general type of Bayesian neural networks, called particle-based BNN or particle optimization (Liu & Wang, 2016; D'Angelo Fortuin, 2021). Generally, the particle-based based variational distribution is the form q(0) Σ5=1 Wy 5(0 B,), where {w, 1 are normalized weights which hold Σ w, 1, and 6(-) the Dirac delta function. Each "particle" f is deterministic model and provides the predictive probability p(y) xr 0y). The particle- based BNN is first studied in Stein variational gradient descent (SVGD) (Tiu & Wang, 2016) and then explored by Liu al. (2019a); Korba et al. (2020); D'Angelo & Fortuin (2020). Instead of directly modeling the gradient flow, our framework optimizes the particles through stochastic gradient descent (SGD), with repulsive force induced by the integrated gain objective. Compared existing particle optimization, our method is easy and cheap to implement, which is especially beneficial for large-scale deep learning.
Similar architectures like "multi-expert" models have been explored by previous long-tailed classification meth- ods (Xiang et al., 2020; Wang et al., 2020; Li et al., 2022). They usually combine several individual classifiers with shared encoder to obtain better generalization performances (Talshminarayanan et al., 2017; Hu et al., 2022), which is inspired by the observation that multiple i.i.d. initializations are less likely to generate averagely "bad" models (Dietterich, 2000). The particle-bas braiille-baedd models reduce the cost Bayesian inference and are more efficient than variational inference and Markov chain Monte Carlo (MCMC), especially high-dimensional and multimodal distributions. Besides, the computational cost of our method can be further reduced by leveraging recent techniques, such partially being Bayesian in model archilectures (Kristiadi et al., 2020).
B Full Utility Matrices for
Utility
We show the full utility matrices of class-sensitive and metaclass-sensitive utilities in Fig. 3, which is based the categories of CIFAR10-LT (Cui et al., 2019).
Plane
1ar
bird
ca1
dder
dog
frog
horee
Ahip
trick
plane
car
bird
Ca1
deer
dog
frog
borsc
ship
truck
1.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
plane
1.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
car-
0.0
1.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
C82
0.0
1.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
bird
1.0
0.0
0.0
0.0
bird
0.0
0.0
0.0
0.0
3.0
0.00
0.0
1.0
0.0
0.0
0.0
0.0
0.0
-1.0
0.0
0.0
0.0
0.0
0.0
-1.0
424
031
deur
0.0
0.0
0.0
0.0
1.0
0.0
0.0
0.00
0.0
0.0
deer
0.0
-1.0
0.0
0.0
1.0
0.0
0.0
0.0
0.0
-1.0
6.00
8000
dog-
0.0
0.0
0.0
0.0
1.0
0.0
0.0
0.0
dog
0.0
-1.0
0.0
0.0
0.0
1.0
0.0
0.0
-1.0
frog-
0.0
0.0
1.0
0.0
0.0
frog
-1.0
0.0
0.0
0.0
-1.0
horse-
0.0
0.0
0.0
0.0
0.0
0.0
0.0
1.0
0.0
0.0
horse
0.0
-1.0
0.0
0.0
0.0
0.0
0.0
1.0
0.0
-1.0
ship
0.0
0.0
0.0
0.00
0.0
0.0
0.0
0.00
1.0
0.0
ship
0.0
-1.0
0.0
0.0
0.0
0.0
0.0
1.0
-1.0
truck-
truck
0.0
0.0
(a) Full Class-sensitive Utility Matrix
(b) Full Metaclass-semsitive
Utility Matrix
Figure 3: Full utility matrix configurations on CIFAR10-LT. The two matrices are designed to particularly
avoid certain misprediction lypes (bird-place and mammal-vehicle mispredictions respectively).
C Derivation of Eq. 10
The following proof based on the assumption that both training and testing data are semantically identical", and the only difference lies in class probabilities (Perain (y) and Prest (v))-
In contrast the open-set scenario (Geng et al., 2020), where additional classes may cause testing data be semantically irrelevant training data.
17

---

Published in Transactions on Machine Learning Research (01/2025)
Proof. We denote the datasel D (x, Y) m.)] where each pair is drawn from the same data distri- bution. Then the logarithm of integrated gain would be:
log G(d) = log B (tw.: Ba G(d]X,6)
(ee...n)......x.ww)--re (w.w) log B. p C(d
p(0 ()
(oa (m log
((0)
d0
(18)
(b)
G(d 0) P) (0)
(e (2.1)
(()) log
d0
(e (2.)) (D).
Here, (a) and (b) are by Jensen's 's inequality (Jensen, 1906). The (D) inside expectation can be further extended to be:
(() log G(d () D)
(D)
d0
(0)
p(0)
P(Y |X, 0)
(()) log
G(d 0)
d0
(())
p
(19)
o(d. x. log 9())
(())
log
+ log
p(u. x. 0) logppY (X)
d0
p())
E od, 0) log m. 0) KL ((())p)p)( -oogp(Y(X).
Therefore, the ultimate expression of integrated gain is:
log G(d) 2 Eig an ).... (ow (DD
B( st (z.w) Bg g(d xr, 0) logp(v;(o.,00 0)] (L({(0)lPP(O)) - -oggp( ( X)
Preet (æ;; Vs)
Ei se) (w.w) Ea-o(o)
[log (dd. (zs,, 0) + log 0)] KL(g(0)) p(O)) logp(Y(
Perain (2t, 91)
Peest (mi, Vi) B
Pirain (Ii, [log g(d, (x,, 0) + logp(u 0)] KL({(())p(())) log
iml
N
1
Uuya, Uyia, logp(y' w., 0) (oge(,ee, 0) KL({(0)[I((O) - log (
Ea- f(nys)
1=1
x
L(g, d).
(20)
Here, (c) is by the one-sample MC estimates of the expectation over the data point (.).).)
n
18

---

Published in Transactions on Machine Learning Research (01/2025)
D Proof of Eq. 11 and Eq. 12
Proff. The objective in Eq. can be written as:
N
L(g, d) ~
-p (2) Banato) log g(di æi, 0) logp(us(2;, 0)] KL(g(0)lp(()) - logp(Y|
[a (1).... (xx...))) (x..g)
Σ
B [logg(d;] 0) + logg(vi(z; 0)] (L(((())lp()) - logp(Y X
1=1
E "i.......... (oN.:...oN...N//eeeeeeee Ea [log Gid] 0) + log (opp((Y1 XY| 0) KL(g(0) (p(0) log p
(x)
log (()) log Peest () log + T5 (w..).....................
Ja ((0)
Ea-g( log G X,
p(0)
(T) |Prest (D) (xen..))....(.xxxxx)))))))) (0) log G(
(21)
Here, (a) is by N independent sam-le MC approximation over all of the data points. When we choose the oot-hot utility matrix U = I, the decision gain G(d Y|X, 0) will equal the predictive probability p(Y (X, and thus it can be ignored. Therefore, the objective with one-hot utility matrix is:
L(g.) KK(g(0))]Pee)) (0
(22)
E Implementation Details
We summarize the necessary implementation details in this section for the reproducibility our method. 'The hyper- parameter choices concluded in table 10. The optimal values of those hyper-paramenn parameters are determined by grid search. We will release the code after acceptance.
Table 10: Hyper-paranene configurations.
Base Model
Batch Size
Learning Rate
Training Epochs
Discrepancy Ratio
Dataset
Optimizer
X
α
CIFAR10-LT
ResNet32
SGD
128
0.1
200
linear
5e-4
0.002
CIFAR100-LT
ResNet32
SGD
128
0.1
200
linear
5e-4
40
0.3
ImageNet-Lir
ResNet50
SGD
256
0.1
100
linear
2e-4
20
50
iNaturalist
Slim ResNet50
SGD
512
0.2
100
linear
2c-4
20
100
E.1 Evaluation Protocol
The evaluation protocol consists of standard classification accuracy, three newly designed experiments the False Head Rate (FHR) along with other two metrics, and calibration experiments on AUC and ECF metrics. Besides, we conduct several ablation studies to evaluate different choices of implementation and the effectiveness of components in our method. For all quantitative and visual results, we repeatedly run the experiments five times with random initialization to obtain the averaged results and standard deviations to eliminate random error. We use f(n,) ny unless otherwise specified.
E.2 Training Objective
We slightly modify the training objective in the code-level implementation to apply two practical tricks for better performance.
Repulsive force. We find in experiments that although applying repulsive force promote the diversity of particles, will certainly disturb the fine-tuning stage in training, which consequently results in sub-optimal perfor- mances by the end of training. To address this issue, we apply annealing weight the repulsive force to reduce
19

---

Published in Transactions on Machine Learning Research (01/2025)
its effect as the training proceeds:
exp(-t/r) ; log (0 -T²)a
(23)
where t refers to the epoch and is stride factor that controls the decay of annealing weight. With the annealing weight, the repulsive force will push partieles away at the beginning of training, and gradually become negligible at the end of training.
Utility matrix. Although the utility matrices in Fig. 1 are designed to address the many realistic problem settings, they will also affect the accuracy of the classification task. Therefore, the utility term in Eq. 10 needs re-scaling so that its negative effect on the accuracy controllable:
logply 0) a Euyra logpy'
(24)
where a is the scaling factor. We can adjust the value a to carefully control the effect of the utility term, which will bring significant improvement on the False Head Rate (along with other metrics) with an acceptable accuracy drop.
E.3 Discussion on Computational Cost
We run all experiments on NVIDIA RTX A6000 GPU (49 GB) and do not need multiple GPUs for one model. The model architecture follows RIDE (Wang et al., 2020) and TLC (Li et al., 2022), in which the first few layers in neural networks are shared among all particles. Therefore, the computational cost of our method is comparable to existing ensemble models. Besides, compared with gradient-flow-base BNN like D'Angelo & Fortuin (2021), which typically uses 20 particles, our model is far more efficient with
than 5 particles.
We also provide a computational cost comparison with other multi-model methods in Table 11. We use 3 particles of ResNet32 the backbone of each method, and train CIFAR100-LT for 200 epochs at NVIDIA RTX A6000. The results show that the efficiency of our method is comparable with baselines.
Table 11: Comparison of computational cost
Method
Training Time per 200 Epoch (min)
# Parameters
ResNet32 (single model)
31.85
469,904
RIDE (Wang et al., 2020) TLC (Li et al., 2022)
43.
1,408,788 784
45.27
1,408,784
RF-DLC (ours)
44.68
1,408,784
F Additional Experimental Results
F.1 Full Experimental Results Classification
We list the full experimental results of top-1 accuracy in Table 12, including the results on iNaturalist (Van Horn et al., 2018). Classes are equally split into three class regions (head, med and tail). For example, there are 33, 33 and 34 classes respectively in the head, med and tail regions of CIFAR100-LT. The results on iNaturalist are obtained by single run due to the large size of the dataset. We use slim version of ResNet50 for all baselines due to CPU memory limitation. The reported results iNaturalist are different from other papers but are still fair comparisons. Our method successfully outperforms other baselines especially on the tailed classes.
F.2 Full Experimental Results on Uncertainty Quantification
Table 13 shows the full experimental results on uncertainty quantification. The results are broken down to 3 class regions, where Bayesian predictive uncertainty outperforms other uncertainty algorithms on all of them.
F.3
Comparison with Cost-sensitive Learning Approaches
The canonical approaches in cost-sensitive learning (Elkan, 2001) could not effectively solve the decision-making problem in long-tailed classification due to their lack of a holistic approach that integrates distributional shift and
20

---

Published in Transactions on Machine Learning Research (01/2025)
Table 12: Full top-1 accuracy results (%)). means the results are directly copied from the original paper. means the results are obtained from slim version of ResNet50 due GPU memory limits.
Dataset
Method
All
Head
Med
Tail
LA
77.67
ACEA
81.2
SRepri
82.06 - 0.01
CE
73.65 0.39
93.22 + 0.26
74.27 0.42
58.51 + 0.62
CIFARIO-LT
CB Loss
77.62 = 0.69
91.70 + 0.57
75.41 + 0.76
68.73 + 1.52
LDAM
80.63 0.69
90.03 0.47
75.88 0.81
7
RIDE
83.11 = 0.52
91.49 + 0.40
79.39 + 0.61
79.62 + 1.56
TLC
79.70 0.65
89.17 0.33
74.33 0.96
76.39 0.98
RE-DLC
83.75 + 0.17
90.49 1 0.60
78.89 1 0.87
82.33 + 1.16
LAt
43.89
ACEi
49.4
66.1
55.7
23.5
SRepri
47.81 = 0.02
66.69 1 0.01
49.91 +: 0.01
23.31 1: 0.11
CE
38.82 = 0.52
68.30 0.61
38.39 0.49
10.62 + 1.23
CIFAR100-LT
CB Loss
42.24 = 0.41
62.53 0.44
44.36 0.96
20.50 + 0.51
LDAM
43.13 = 0.67
63.58 0.93
42.90 = 1.03
23.50 + 1.28
RIDE
48.99 0.44
0.54
49.70 = 0.59
28.78 1.52
TLC
48.75 = 0.16
69.43 1 0.36
19.02 + 0.94
28.40 1 0.72
RE-DLC
50.24 0.70
69.92 + 0.77
51.07 + 0.82
30.34 1.49
LAt
51.11
ACEA
54.7
SRepri
52.12 = 0.06
62.52 1 0.26
49.44 0.18
32.14 1: 0.41
CE
17.80=0.15
53.46 + 0.36
45.92 0.19
44.03 + 0.24
ImageNet-LT
CB Loss
51.70 1 0.25
$7.62
4..19
48.29 0.41
LDAM
51.04 0.21
57.66 0.440
4..16
47.21 0.22
RIDE
54.32 = 0.54
60.88 0.71
3.43
TLC
55.03 0.34
61.19 0.53
52.35 0.31
51.56 + 0.35
RF-DLC
55.73 + 0.17
62.18 + 0.28
53.06 + 0.22
51.98 + 0.40
LAt
66.36
ACEA
72.9
SReprt
70.79 = 0.17
70.70 1 0.31
70.83 + 0.20
70.79 + 0.17
iNaturalist
CE
65.17
75.28
64.00
54.22
IDAM*
67.20
74.58
65.17
60.84
RIDE
72.87
76.71
69.73
69.28
RF-DLC*
73.80
76.66
70.79
70.55
Table 13: Full results of uncertainty quantification. uncertainty algorithms on all class regions.
Bayesian predictive uncertainty outperforms other
AUC (%) t
ECE (%) 4
Uncertainty Algorithm
All
Head
Med
Tail
All
Head
Med
Tail
MCP (Hendrycks Cimpel, 2017)
80.15
87.56
80.78
62.15
19.39
11.52
18.30
36.84
Evidential (Li et al., 2022)
78.16
84.51
79.62
67.63
21.88
10.27
22.01
34.53
Bayesian (RF-DLC)
80.62
88.47
81.82
63.05
10.87
6.37
11.09
16.71
decision- maaiiinn-makiggggg processes. For example, the method in Section 2 of Elkan (2001) does not consider specific error types during the training phase, and fails to incorporate the utility matrix during testing. The Bayesian method in Section 4 of Elkan (2001) learns standard posterior without accounting for data distribution the utility matrix, applying the latter only during the lesting phase. Additionally, Elkan (2001) employs a different decision strategy as flexible and has been discussed in Section 4.1.
(Eq. 1), which
21

---

Published in Transactions on Machine Learning Research (01/2025)
To further illustrate the advantages of method, empirically compare method with two baselines: i) the Bayesian method in Elkan (2001) and ii) the naive combination of re-weighling loss and the Bayesian method in Elkan (2001). Table 14 shows that method significantly outperforus both baselines in terms of ACC and FHR. This demonstrates the advantages of our method which concurrently addresses long-tailed distributions and decision-making during both training and testing phases unified way.
Table 14: Comparison with cost-scnsitive learning approaches on CIFARIO0-LT. Tail-scnsitive utility is applied.
ACC
(%)
FHR (%)
Method
All
Tail
25%
50%
75%
Avg
RF-DLC
49.92
33.74
14.92
30.22
51.80
32.31
Elkan (2001)
43.36
24.68
25.97
41.98
52.64
40.20
Reweighting Loss + Elkan (2001)
45.40
26.35
20.69
41.28
65.36
42.44
F.4 Ablation Study on the Robustness of Utility Values
We conducted the experiment on the robustness of utility values on CIFARI00-LT. The utility matrix used in this experiment is:
0
u
0
U
(25)
The results, as shown Table 15, demonstrate that all values of u will lead to improvement in FHR and comparable or better ACC compared with the baseline one-hot matrix (i.e. 0). These findings indicate that our method's performance is relatively insensitive variations in utility values (we simply use -1 in the paper) and tuming these values can further improve performance.
Table 15: Ablation study on the robustness of utility values on CIFAR100-LT.
ACC
(%)
FHR (%) 1
Utility Value
All
Tail
25%
50%
75%
Avg
49.76
30.00
19.12
38.04
60.44
39.20
-0.1
49.70
29.88
18. 18.12
36.56
59.32
38.00
-0.2
49.09
29.38
18. 16
36.34
60.28
38.26
-0.3
49.41
29.82
17.81
35.28
58.12
37.07
-0.4
49.49
30.41
17.20
34.64
56.80
36.21
49.90
31.47
16.71
33.52
55.72
35.32
-0.6
49.84
31.82
16.92
33.82
54.72
35.15
-0.7
49.32
30.85
16.33
32.04
55.08
34.48
-0.8
49.66
33.00
15.91
31.32
51.60
32.94
-0.9
49.27
32.12
15.53
30.68
51.24
32.48
-1
49.92
33.74
14.92
30.22
51.80
32.31
F.5 Ablation Study on the Task-specific Utility Functions
The task-specific utility function (used in Eq. 10 and Eq. 16) is one of the key contributions of this paper. It specifically avoids mispredicting some classes as other classes. To understand the effect of utility functions, we add ablation study in Table 16, where Eq. 16 with tail-sensitive utility is added to haselines at inference time. Adding Eq. improves FHR performance and maintains the ACC, which demonstrates effectiveness our inference strategy. Importantly, method is still better than baselines with utility functions inference time.
22

---

Published in Transactions on Machine Learning Research (01/2025)
Table 16: Ablation study on the effectiveness of task-specific utility functions (Eq. 16) at inference time. Naively applying Eq. 16 to other basclines can improve FHR, but will not outperforms RF-DLC.
ACC
(%) Med
FHR (%) 1
Method
All
Head
Tail
avg
25%
50%
75%
CB Loss
42.24
62.53
44.36
20.50
49.22
24.88
48.41
74.38
CB Loss
Eq. 16
42.07
60.35
43.69
23.10
47.89
21.86
49.66
72.16
LDAM
43.13
63.58
42.90
23.50
43.29
21.22
43.04
65.62
LDAM + Eq. 16
43.16
62.11
43.07
24.59
40.90
20.17
40.85
61.68
RF-DLC
50.24
69.92
51.07
30.34
32.08
15.39
31.34
49.51
23


Reviews:
[{"review_id": "kucQmqAqOp", "reviewer": "Reviewer_1T7o", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n\n- The paper tackles a more generalized and realistic long-tailed classification problem.\n- The proposed method is theoretically driven and includes detailed and valid derivations.\n- The paper is easy to follow.\n\n## Weaknesses:\n\n- For long-tailed distributions, the proposed method simply reuses resampling to address class imbalance. Resampling is known to cause optimization challenges and instability[1, 2], and numerous works[3, 4, 5] have addressed class imbalance issues built upon this. I am curious about how the proposed method addresses or mitigates these issues and whether other widely used methods addressing class imbalance, like logit adjustment[6], could be integrated into the proposed framework.\n- The proposed method relies on the assumption $P_{train}(x|y) = P_{test}(x|y)$ (Assumption 1) to estimate decision gain on the balanced distribution (as in (7)(10)). However, it was shown that due to the scarcity of examples, estimating the risk on $P_{train}(x|y)$ with the empirical distribution for tail classes can cause substantial discrepancy due to scare instances[7, 8, 9, 10]. When the number of tail examples is extremely limited, could the proposed method still be essential? I encourage the authors to include a discussion of these works or add more empirical results.\n- Potential unfair comparison. 1) In tail-sensitive and class-sensitive tasks(Table 1 and Table 2), the proposed method adopts task-specific utility functions, while the baselines are directly adopted without task-specific prior information (utility preferences). It is more fair to use parameters learned by baselines as point estimates of $P(\\theta|D)$, making decisions with a scoring function similar to (16). 2) In tail-sensitive and class-sensitive tasks, most metrics (Table 1 and bird-plane detection in Table 2) measure the risk of predicting tail samples as head samples, a tendency of errors in long-tailed classification[11]. Therefore, it’s hard to tell if the improvements result from enhanced long-tailed classification performance(Table 3) or better alignment with the predefined utility. I encourage the authors to add results on randomly selected classes for group-sensitive long-tailed recognition.\n\n[1] Class-balanced loss based on effective number of samples. CVPR 2019.   \n[2] Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss. NeurIPS 2019.  \n[3] Disentangling Label Distribution for Long-tailed Visual Recognition. CVPR 2021.  \n[4] Decoupling representation and classifier for long-tailed recognition. ICLR 2019.    \n[5] Towards calibrated model for long-tailed visual recognition from prior perspective. NeurIPS 2021.  \n[6] Long-tail learning via logit adjustment. ICLR 2021.   \n[7] The Majority Can Help The Minority: Context-rich Minority Oversampling for Long-tailed Classification. CVPR 2022.   \n[8] Distributional robustness loss for long-tail learning. ICCV 2021.   \n[9] A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning. NeurIPS 2023.  \n[10] Enhancing Robustness to Class-Conditional Distribution Shift in Long-Tailed Recognition.  TMLR 2024.   \n[11] Overcoming Classifier Imbalance for Long-tail Object Detection with Balanced Group Softmax.  CVPR 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to tackle a more generalized long-tailed classification problem of training classifiers on an imbalanced class distribution that can adapt to decision-making utilities (i.e., predefined bi-class classification error risks), rather than merely minimizing the average classification error across classes. To achieve this, a Bayesian decision-theoretic framework is proposed, including a novel optimization objective to approximate utility gains from decisions, along with a variational inference method to optimize this objective, which transforms the task into training an ensemble classifier with a KL-divergence regularization term. On standard long-tailed classification tasks and class-sensitive long-tailed distribution tasks, the proposed method achieves superior performance in both accuracy and uncertainty estimation compared to baseline methods tailored for standard long-tailed classification.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the questions and requested modifications in the Weaknesses. \n\nI also recommend improving implementation details: 1) The imbalance ratio of the CIFAR-LT Dataset. 2) The specific model architecture (instead of the rough description in E.3). 3) A quantitative comparison of computational costs with previous methods. \n\nMore discussion of closely related works in the long-tailed field should be added, including: 1. Distributionally robust optimization methods[8, 10, 12] that also consider the costs of misclassification between labels. 2. Cost-sensitive loss functions based on scaling prediction logits[2, 9, 13].\n\n\n[12] Coping with Label Shift via Distributionally Robust Optimisation. ICLR 2021.   \n[13] Label- imbalanced and group-sensitive classification under overparameterization. NeurIPS 2021.", "broader_impact_concerns": "This paper includes a statement for Broader Impact, and I do not observe any ethical concerns that are insufficiently addressed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BhZjEZYbZU", "reviewer": "Reviewer_C3P7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n1. The authors highlight the importance of incorporating decision-making in long-tailed classification and identify specific challenges within this context.\n\n2. They propose targeted solutions, including a novel design for decision gain and particle-based repulsive regularization to address these challenges.\n\n3. Comprehensive experiments validate the significance of this setting and support the rationale behind the proposed method.\n\n**Weaknesses**:\n\n1. The proposed method appears relatively incremental, with the main contributions being the new decision gain design and the False Head Rate metric, while other critical components are adapted from prior work.\n\n2. The authors claim improved stability for the new decision gain design over previous methods but do not provide experimental evidence to substantiate this.\n\n3. Although medical images are cited to emphasize the importance of decision-making in long-tailed classification, no experiments on medical image datasets are conducted.\n\n4. The authors perform an ablation study on the repulsive force; however, examining the impact of the parameter $\\lambda$ on final performance would provide added insight.\n\n5. The algorithm requires optimization of six particles, which adds computational cost.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a framework for reliable and flexible decision-making in long-tailed classification, leveraging Bayesian Decision Theory to integrate decision making into long-tailed classification. They address essential challenges for optimal decision-making in long-tailed data, including designing decision gains, managing train-test distribution shifts, learning posterior distributions, and making optimal test-time decisions. They analyze connections between the framework’s design elements and standard classification methods and demonstrate its effectiveness on CIFAR10/100-LT, ImageNet-LT, and iNaturalist, outperforming previous methods. Additionally, they investigate the effects of utility, class probability forms, particle count, and repulsive forces. The authors introduce a new metric, False Head Rate, to measure tail-sensitivity risk, further strengthening decision-making in long-tailed classification.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Provide experiments to support the claim that the proposed decision gain design is more stable than previous methods.\n\n2. Conduct experiments on medical imaging datasets to reinforce the importance of this setting.\n\n3. Include experiments demonstrating performance under different regularization strengths ($\\lambda$).\n\n4. Clarify the parameters $k$, $\\bar{\\theta^2}$, and ${\\bar{\\theta}}\\ ^2$ in Equation 15.\n\n5. It would be beneficial to include experiments on the Places-LT dataset.\n\n6. The use of six particles increases computational cost. Could the authors provide a detailed computational cost analysis, including training time and number of parameters?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1aiqfU07oD", "reviewer": "Reviewer_2YWu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n - This paper proposes a novel solution to an important problem in machine learning, that of learning from extremely balanced data in high-risk prediction scenarios in sensitive domains.  It has applications in safety critical domains like healthcare and autonomous transportation.  By confronting the long-tailed data challenge with a focus on decision-making rather than pure accuracy, the paper addresses a practical gap often overlooked in previous research. RF-DLC’s versatility with customizable utility matrices is a strong contribution for diverse use cases.\n - The use of Bayesian Decision Theory to produce a framework for handling both data imbalance and asymmetric decision-making risk is a thoughtful and rigorous approach, and the proposed variational optimization strategy aligns well with the model’s objectives.\n - The paper thoroughly evaluates RF-DLC across various benchmarks (e.g., CIFAR10-LT, ImageNet-LT), showcasing its effectiveness in reducing high-risk errors, which is well-supported by comprehensive metrics.  I also appreciate the section with results on uncertainty quantification.\n\n**Weaknesses**\n - It seems unrealistic to assume knowledge of class frequencies in the test set (whether uniform or not).  More discussion of this assumption, and experiments where it is violated, would significantly strengthen the case for adoption of this method.\n - While baseline methods are not spelled out in detail, it seems likely that RF-DLC is the most computationally expensive method included in your experiments, given the particle based approach.  It would be useful to clarify the relative computational costs of the different methods (both in training and inference), so the reader can understand the relative performance differences in these terms.  Also, it would be time-consuming but ideal to not only enumerate the different computational costs of the methods, but also to make some experimental comparison on an even footing.  For example, take some strong method that uses a single model, and extend it to an ensemble of equal size as RF-DLC.  (Note that this comment could be wrong/useless if all baselines are of equal computational cost to RF-DLC, although I suspect this is not the case).\n - Having worked on medical problems where false positives are similarly costly to false negatives (e.g., because unnecessary treatment is invasive, harmful, or otherwise very costly), I think it would be useful to sharpen the motivating examples where given.  For example, the statement “in disease detection where healthy individuals belong to head classes and patients belong to tail classes, the risk of classifying patients as healthy individuals is significantly larger than the reverse scenario” should be hedged to say that this is often or usually the case, and ideally give some more specific examples (and even counter-examples).\n - The paper assumes that misclassifications are mainly between head and tail classes, which might limit its flexibility in broader use cases where misclassification costs vary widely even within head or tail classes.  Additional discussion on practical considerations for designing the utility matrix could strengthen uptake of this method in more domains.\n - With respect to the UQ results, it would be interesting (and in keeping with the rest of the results section) to see these broken down across different class types.\n - Small nitpick, but I don’t find the method name to be particularly informative or useful.  If it’s not too late, I would reconsider and attempt to sharpen this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a new framework named RF-DLC, designed to enable improved decision-making in long-tailed classification by building decision-risk awareness directly into the model. Built on insights from Bayesian Decision Theory, RF-DLC is trained to make predictions based on concerns beyond top-level accuracy, assigning higher significance to errors with greater real-world risks.  The proposed method leverages an integrated gain function from BDT that combines both decision-making risk and information about the long-tailed data distribution into a single objective. To make this objective feasible for real-world datasets, the authors propose a variational optimization strategy that approximates the posterior distribution, enabling efficient model training while accounting for asymmetric error type costs.  Additionally, the authors present \"False Head Rate\" as a novel metric that quantifies the specific risks of misclassifying tail classes as head classes, an issue that can be particularly crucial in fields like medical diagnostics or autonomous driving, where certain errors are especially costly. RF-DLC is adaptable to various task settings and decision dynamics via the customization of utility matrices to address different risk profiles, providing flexibility in decision-making frameworks. Experimental results show RF-DLC effectively minimizes high-risk misclassifications while maintaining solid performance on traditional accuracy metrics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- More discussion of the relative computational costs of the methods compared\n - Since RF-DLC assumes knowledge of class frequencies in the test set, it seems important to discuss the limitations of this assumption and its impact if violated. An additional experiment where the assumption is intentionally violated would help clarify the robustness of RF-DLC in real-world scenarios.\n - It would be helpful to have more discussion on designing utility matrices for scenarios where misclassification costs vary within head or tail classes. This could enhance the method's flexibility and applicability across diverse domains.\n - Some UQ results should be segmented by head and tail classes. This breakdown would align with other results, and give a more nuanced view of RF-DLC’s UQ performance.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There were a number of responses to reviewers that should be incorporated in a final version of the paper. From the AE's reading, there are also some claims about the positioning of the paper to prior work that ought to be clarified.

**Requested changes**

- _"Existing approaches to long-tailed classification usually focus on improving overall accuracy... Existing methods on long-tailed classification assume that mispredictions between any pair of classes are equally risky"_. These claims must be clarified. The dominant focus in prior work on class-imbalance (classically) and long-tail learning (more recently) has been optimising performance measures that go beyond the standard zero-one misclassification accuracy. This includes the balanced accuracy, F1-score, ... Please see, e.g., Table 1 of Menon et al., "On the statistical consistency of algorithms for binary classification under class imbalance.
", ICML 2013.

- _"While cost-sensitive learning has been applied to standard classification (Elkan, 2001; Chung et al., 2016; Shu et al., 2019), its application to long-tailed classification remains unexplored."_ This claim must be clarified. See, e.g., the Ling & Shen paper cited by the authors; Sec 2.3 of He and Garcia, "Learning from Imbalanced Data", TKDE 2009; Nguyen et al., "Cost-Sensitive Learning Methods for Imbalanced Data", ICDNN 2010. Indeed, the application to class imbalance is often used to motivate cost-sensitive learning.

- _"In long-tailed categorical data, the training and testing sets follow different distributions... the testing data distribution is uniform over categories"_. This statement could be clarified. Indeed, it is common to evaluate long-tail models on uniform test sets; this way, the accuracy on the test set roughly corresponds to a balanced accuracy on a distribution with imbalanced class frequencies. However, in some real-world applications of long-tail learning, it is unclear whether the test frequencies will be uniform. e.g., in an animal-image classification application, we are unlikely to expect that at deployment time, frequencies of rarely sighted animals during training will become uniform. (There are _other_ settings where the assumption is appropriate, owing to the training data being collected in a biased manner.)

- _"The standard long-tailed classification can be regarded as a special case in our
framework, where the overall accuracy is the most decisive metric in evaluation"_. Do you mean _"The standard classification"_, since here you are treating all classes equally? Or is the point that since you assumed that the test frequencies are uniform, this captures long-tail learning? If the latter, it may be worth somehow clarifying that the genuine standard (non long-tail) classification is a different setting still, one outside the purview of the present cost-sensitive discussion.

- _Comparison with Cost-sensitive Learning Approaches_. In Appendix F.2, some more mathematical detail would be useful. Indeed the method of (Elkan, 2001) would essentially estimate $p_{\rm train}( y \mid x )$ during training, and then apply a correction based on the costs during testing. In the present setup, if you assume the "standard" utility matrix, then is your Equation 5 not equivalent to the first stage of (Elkan, 2001)? Is the claimed distinction the use of Bayesian inference of the parameters?

- _Comparison with expected risk minimisation framework_. The paper presents the Bayesian decision theory framework as one of the key contributions. In statistical learning, it is common to propose and analyse the expected (surrogate) loss. e.g., in a cost-sensitive learning context, see Equation 1 of Avila Pires et al., "Cost-sensitive Multiclass Classification Risk Bounds", ICML 2013. Can you clarify the conceptual distinctions to such a framing of long-tail learning?

- Equation 10, notation $U_{y', d_i = y_i}$ ought to be simply $U_{y', y_i}$.

- Equation 10 and elsewhere, use $\mathrm{KL}$.

- add results comparing to ensemble methods.

- add breakdown of UQ results by class.

- add results on DermaMNIST.
Decision: Accept (poster)