Evidential Mixture Machines: Deciphering Multi-Label Correlations for Active Learning Sensitivity
Authors: Dayou Yu
Minghao Li
Weishi Shi
Qi Yu
Venue: NeurIPS 2024 poster (2024)
Keywords: Active learning
multi-label classification
Primary area: active_learning

Abstract: Multi-label active learning is a crucial yet challenging area in contemporary machine learning, often complicated by a large and sparse label space. This challenge is further exacerbated in active learning scenarios where labeling resources are constrained. Drawing inspiration from existing mixture of Bernoulli models, which efficiently compress the label space into a more manageable weight coefficient space by learning correlated Bernoulli components, we propose a novel model called Evidential Mixture Machines (EMM). Our model leverages mixture components derived from unsupervised learning in the label space and improves prediction accuracy by predicting weight coefficients following the evidential learning paradigm. These coefficients are aggregated as proxy pseudo counts to enhance component offset predictions. The evidential learning approach provides an uncertainty-aware connection between input features and the predicted coefficients and components. Additionally, our method combines evidential uncertainty with predicted label embedding covariances for active sample selection, creating a richer, multi-source uncertainty metric beyond traditional uncertainty scores. Experiments on synthetic datasets show the effectiveness of evidential uncertainty prediction and EMM's capability to capture label correlations through predicted components. Further testing on real-world datasets demonstrates improved performance compared to existing multi-label active learning methods.

Evidential Mixture Machines: Deciphering Multi-Label Correlations for Active Learning Sensitivity
Dayou Yu'
Minghao Li²
Weishi Shi
Qi Yu'
Rochester Institute of Technology, Rochester, NY 146231 University of North Texas, Denton, TX 76203
{dy2507, grit.
minghaoli@my, weisha
Abstract
Multi-label active learning is crucial yet challenging area in contemporary ma- chine learning, often complicated by a large and sparse label space. This challenge is further exacerbated in active learning scenarios where labeling resources are con- strained. Drawing inspiration from existing mixture of Bernouli models, which ef- ficiently compress the label space into more manageable weight coefficient space by learning correlated Bernoulli components, we propose a novel model called Evidential Mixture Machines (EMM). Our model leverages mixture components derived from unsupervised learning in the label space and improves prediction ac- curacy by predicting weight coefficients following the evidential learning paradigm. These coefficients are aggregated as proxy pseudo counts to enhance component offset predictions. The evidential learning approach provides uncertainty aware connection between input features and the predicted coefficients and components. Additionally, method combines evidential uncertainty with predicted label embedding covariances for active sample selection, creating a richer, multi-source uncertainty metric beyond traditional uncertainty scores. Experiments synthetic datasets show the effectiveness of evidential uncertainty prediction and EMM's capability capture label correlations through predicted components. Further testing on real-world datasets demonstrates improved performance compared to existing multi-label active learning methods.
Introduction
Active learning (AL) is paradigm where we have access to abundant unlabeled data instances with limited labeling budget [35) [.5]. Most AL methods focus on the selection strategy that helps the machine learner achieve better performances with informed selection of labeled data instances. However, while AL for standard classification tasks has been studied extensively, an important task that is multi-label classification has been largely overlooked. In real-world problems, each data instance may be associated with more than one labels [1] 39 27). For machine learning models, the difference between the multi-class problem where only ground truth label is associated with data instance and the multi-label classification (MLC) problem fundamental. Classic solutions either transforming the MLC problem into multiple binary problems directly build joint learning problem for all labels. With the binary approach, we lose the common underlying correlations between input features and labels. The number of classifiers may be large due the large label space, thus the separated training process can be costly. Also, many labels are relatively rare and may depend on other labels, and we can not learn such dependencies when isolate these labels. With the joint learning approach, the biggest challenge is to combine common labels with rare labels in balanced
*Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(x
Active Sampling Criterion
(MSU Score)
(x,
Axy(x)
P
Mixture Coefficients
Input Feature
9x()
Pk
e()
7/
pd
True Label
bi
Multi-Label Predictive Distribution
Mixture Components Mixture Modeling Layer
Encoder
Decoder
Evidential Layers
Evidential Loss eq(2)
SoftMargin Loss eq(3)
Figure 1: Overview of evidential mixture machines
way. For the rare labels, there might be few positive data instances. A typical model is likely to optimize the feature embedding according popular labels adopt the so-called "shortcut" learning, making direct connection from the input features to the rare labels more difficult. Although we now have access to common and rare labels at the same time, it might not suffice to promote the learning of their correlations we directly model the labels from the input features. These unique challenges of the MLC problem become more pronounced in AL settings, where the rare labels become more scarce. They also increase the difficulty of obtaining high-quality uncertainty estimation, which often crucial in many AL selection strategies it allows us to know when the model "does not know" in order to make an educated decision on which data instances to label.
To address these challenges, we draw inspiration from the mixture of Bernoulli model, which can model a large label space with a small number of mixture components. There are existing methods that try to capture the label correlation using such model [17 (30. However, to connect with the input features, they either resort to a purely conditional case, namely conditional Bernoulli mixtures conjugate classification head, which is a Gaussian Process model (GP-B2M) B0).
(CBM) [17 For the former, a distinct set of label clusters is predicted for each data instance, meaning that the label correlations are completely separated from the learning tasks. This shorteoming makes the CBM model unsuitable for AL. The latter relies on the Gaussian Process (GP) which outputs the weights of the mixture components. For the intended task, complete GP is too expensive 20 [6]. while sparse GP has limited predictive ability. Unlike the CBM model, GP-B²M M creates set of global label clusters. However, the label prediction for rare labels remains challenging because they can only ever be as good as the best label cluster available. Furthermore, the uncertainty quantification superficial because only captures the approximate covariance of the label prediction and the variance of the GP predictions. This point estimate does not fully capture the unknown and is not sufficiently effective in AL.
In this work, we propose to combine the mixture of Bernoulli with a deep evidential model 28 both incorporating deep learning and enabling more fine-grained uncertainty analysis. The deep learning model needs only a forward pass during prediction time, which is much more efficient than estimating the predictive distribution in random process model. The uncertainty analysis can now include each evidential prediction. This flexibility is crucial in improving the informativeness evaluation of multi-label data samples as mentioned above. Structurally, the proposed model is composed of a shared encoder that provides embeddings and two decoders that predict the weight coefficients and the proxy pseudo counts correction for the Posterior Beta which models the global label cluster, respectively, as shown in Figure] [] The weight coefficient predictor is trained as a deep evidential regression model, which makes uncertainty-ave aware predictions of optimal assignments to the mixture of label clusters for each data instance. By using deep evidential model as the classification head, we achieve the aforementioned improvement of efficiency during inference time. By sharing the encoder, we maintain a close connection beginning from the input features, through the weight coefficients, and to the label clusters. Compared to the CBM model, the proposed Evidential Mixture Machines (EMM) maintains global label cluster. Compared to the GP-B " model, the EMM model allows

---

adjustment to the label clusters based on each data instance. Additionally, the uncertainty information can be captured by both the dine-grained uncertainty decomposition from the evidential posterior parameters and the predicted variability in final labels. The evidential uncertainty decomposition is implemented through a conjugate interpretation of the Normal Inverse Gamma parameters, which gives these parameters a more impactful pseudo count meaning. The label variability is assessed using the covariance approximation in predictions and the discrepancy between the global label cluster and the proxy pseudo counts. Apart from the novel sampling strategy, the model is trained using alternating loops of the evidential training step and a joint label-based training step, which strengthens the ability to accommodate individual samples.
With both the demonstration on synthetic datasets and the AL evaluation on real-worle datasets, we show the effectiveness of the EMM model. Our main contribution is threefold:
integration of evidential learning with multi-label classification through EMM,
principled uncertainty quantification for active sampling using EMM,
intensive evaluation showing performance gain by EMM for large label spaces, especially on sparse and rare labels.
2 Related Works
Multi-label classification. In the realm of multi-label classification (MLC) 41 Binary Relevance Models (BRMs) have gained widespread use, leading to the development of various active learning (AL) models based BRMs. Notable examples include employing the estimated reduction of a BRM loss function as an uncertainty criterion for data sampling, as demonstrated by (38). A large portion of the multi-label active learning work do not require annotators to label all possible labels for a given data instance [25] 37 77. The main consideration behind these approaches is the significant reduction in annotator labeling costs. However, this strategy inevitably breaks the inherent connections between labels, making it impossible to comprehensively measure the informativenes, of data instance using label correlations. Additionally, models designed to handle partially labeled data are required, limiting the applicability of such methods. Therefore, in this paper, we will not compare our approach to these methods. Other approaches integrate the properties
the support vectors of individual support vector machines (SVM) within BRMs, using label correlation more as a means to simplify the querying process than to enhance active sampling (29 32
9 [Q] While these methods incorporate label correlation to some extent, such through label inconsistency (18), label ranking [26). or learning reularization (40), they do not systematically capture label correlations, potentially leading imprecise uncertainty measures in ML-AL contexts.
Label correlations and multi-label AL. Some existing models attempt to explicitly capture label correlations or use latent embeddings to facilitate active multi-label sampling. For instance, the CBM model uses the approximate entropy of predicted labels for data sampling [5], but its dependency on an external multi-class classifier for predicting component coefficients complicates AL due to the challenges in model selection and parameter tuning. Furthermore, CBM, designed primarily for MLC rather than AL. predicts distinct label clusters for each data sample without discovering global label clusters, thus limiting its effectiveness in multi-label AL [17]. Other approaches like correlation. aware method for transfer learning 7 struggle with large and sparse label spaces due to their reliance on kernel functions for measuring label similarity. Compressed sensing (CS) techniques [34] 31) innovative in learning latent embeddings to capture label correlations but assume labels are drawn from a Gaussian distribution and are not efficient in AL, especially in early stages with limited training data. In [30), a Bayesian mixture of Bernoulli model is proposed. A set of global label clusters captured in a Bayesian manner. However, the inference process of the model is complicated and the fixed label clusters limit the predictive ability in the final label space.
Evidential learning. Evidential models have been developed to enable fine-grained uncertainty quantification in deep learning (DL.) models (28) 11. These models introduce a higher order conjugate prior distribution over the likelihood distribution, and train the DL model to output the parameters of the higher order distribution [4] (3) The higher order distribution enables the model express the fine-grained uncertainty information. Evidential models have been successfully extended to classification 212 19]. regression [I 21), action recognition [2]. OOD detection [13], and meta-
learning problems [22]. We extend the evidential deep learning framework to our setting that leads to novel fine- -gaiine uncertainty guided active-learning for multi-label classification.
3

---

3 Methodology
Problem setting. In our multi-label active learning problem. the essential task is to predict multi- variate 0-1 label vector (y1,... ...... yr) E {0, 1 from the input features x E E For AL, we start with a small initial labeled set St. and a large unlabeled pool Sy (N₂, = | < No = [So() The AL strategy A(x)= I + R is function that grades the instance x" E Sy according to its informativeness. The top graded samples will be labeled as a batch b, and added to Sz. The building blocks of our proposed method include the Mixture of Bernuuli label clusters and the coefficient predictor which connects the input features to the label clusters.
3.1 Preliminaries
Mixture of Bernoulli. The set of label clusters contains K mixture components. Each mixture component is a L-variate Bernoulli distribution 2μ: = II Bernoulli (ui Pks) that captures local 'stereotype' of the complete label distribution, where Bernoulliy:: p&1) = n (1 p==) and the total number of labels. The Bernoulli parameter Pug has conjugate prior Beta( agt, bu) Initially, the mixture of Bernoulli can be found through label-only learning. Using an EM algorithm,
(0) can learn the initial components 14 KXL from the labeled samples St. These components model
the set of labels p(ylμ) = Ex=ı k= II Bernoulli (y; pla), where Tæ E (0, 1] is the normalized weight coefficient of the component However, since this is a label-only process, we are missing the connection to the input features x.
Connecting to input features. For a conditional model such CBM, the connection is through (x)). The model is still trained in an EM manner so that the predictions y can be made for each x. For conjugate model such as GP-B²M, the connection is through Tr plzzz((ii (x)) and the variational training process of the conjugate model impacting the posterior μ. As mentioned before, one issue with CBM is the disconnected prediction models of æ and μ., while the inference process of GP-B²M is too expensive. In our proposed model, we also predict both ग and je from x using two decoder networks 9r(-) and (-(-). However, we maintain the connection by using a shared encoder network e(x), as shown in the structure in Figure [ Specifically, 9r (e(x)) is an evidential model that predicts the distribution of using the output evidence parameters, which function in the following way.
Tk = p(zz x) and
Evidential weight coefficient predictor. We model the connection from input features to la- bel clusters with fine-grained uncertainty information using the evidential regression model. The target for regression to learn combination of Ti values that best reconstruct the final la- bel predictions. To this end, we place a higher-order Normal Inverse Gamma (NIG) prior
NIG p)) [r(T,, "a""aa""aa over the regression model's Gaussian output N( or.. The evidential model is trained to output the NIG parameters p = (7, v, a, 33) similar to M. In this evidential model, the Gaussian likelihood interacts with the NIG prior, leading to Student-t predictive distribution:
p(x|z, p) = J. pf
3(1+ D)
St 73
2a
(I)
va
Here, the evidential model predicts coefficients for input x as: ă = PP (.. [z] = y
The NIG parameters 7, 3 8 are outputs from the predictor branch gnr (e(x)) The evidential model, through its higher order NIG prior, can quantify the aleatoric (ALE) and epistemic (EP) uncertainty [ as ALE E[o²] = 3
a-j .EP = Var[7] p(a-13- In this evidential framework, due to the conjugacy of the NIG prior with the Gaussian likelihood, the posterior is also the NIG distribution. Moreover, in this model, after interacting with N i.i.d. data points (T1, л N) the posterior NIG parameters update as the observations increase (23] Through the pseudo-coun interpretation, the
total evidence is quantified as E aa B"
We train the model maximize The likelihood under the predictive Student-t distribution, which gives the NLL los:
LuL = Log(p(TTR(E,))
π
aalog S (a 1 2/ log (!TH - 2 + +Q) + log T(a + !)
r(a)
= log
2

---

where Ω = 28(1 + v). Additionally, we want the model's confidence/evidence for the prediction to be low when the prediction is incorrect by introducing a evidence-baee based regularization CRFG (TT - -))² E. The regularization penalizes the highly confident wrong predictions, and ensures model's s confidence is rightly placed. The overall loss is
LEVID LULL. + Areg
where A controls the impact of the regularization.
3.2 Bringing Evidential Learning into Multi-Label Active Learning
The integration of an evidential model our multi-label classification approach presents several distinct advantages, particularly in addressing the inherent complexities of active learning (AL) environments. Firstly, evidential models provide a more nuanced and sophisticated mechanism for uncertainty quantification. This is crucial in AL settings, especially when dealing with sparse and rare labels, where traditional models often struggle. By effectively capturing and quantifying uncertainty, our approach leads to more informed and strategic decisions selecting data instances for labeling, optimizing the use of limited labeling resources. Furthermore, the evidential model facilitates deeper understanding of the underlying label correlations, enabling the model to make more accurate predictions across a broad spectrum of labels, including the rare ones. This leads to significant improvement in the overall classification performance, especially in scenarios where conventional methods might overlook subtle but crucial label dependencies. Additionally, the evidential approach inherently enhances the interpretability of the model's predictions, offering insights into the confidence and reliability of these predictions. This aspect is particularly valuable in knowledge-rich domains where understanding the model's decision-making process as important as the predictions accuracy. Thus, incorporating an evidential model into our multi-label classification framework marks a substantial advancement, offering a robust, efficient, and insightful solution to the challenges posed by large and complex label spaces in active learning scenarios.
3.3 Evidential Mixture Machines
Building upon the tools above, we propose novel EMM model. Compared to existing methods, our novel contribution to model learning lies in how we connect label clusters to input features and how we learns the final labels in joint manner. The integration enables evidential uncertainty analysis through both weight coefficient predictions and final label predictions. The evidential learning of π already explained above. Here, we introduce the joint learning using actual labels y.
Joint multi-label training with label clusters. Once we have the coefficient predictor branch, we can train the full model make the final label predictions. We first freeze e(-) and 9mn, (-) to train (.) Instead of directly letting the network predict from (-)(-), we make (-)-)) output proxy
(0) (0) pseudo counts بيه and bui to be combined with the initial (H : Ogl ((()) ). This ensures that we
maintain the correlations encoded in (0) F The network outputs of dimension N K are split (0)
into à and b and added to (9) and (00) in weighted fashion: (x) a + (,, bxx(x) 610 + (x).. The new Bernoulli parameter for each instance is then computed by (x) = aga (xx(x)/ (awa(x) + bw (x)). The label prediction is ğr(x) = Ex n (x)) The model is trained using a soft margin multi-label loss:
LsoftMargin
1 L
Σ 1=1
y log
+ (1 v1) log
exp ( - it)
(3)
1 expp 91)
1 exp in)
We then adopt the evidential pseudo-count style update of the label clusters, we would have dkl (0) agt + >> (x,, yuu and (())) = ((0) + - oं
(1)
However, this update only makes the correction based the predicted weightspiæ. If we keep updating in this way, the biases build up and the popular labels will be dominant in future components. Thus, we also include a weighted update based the predicted proxy pseudo-counts, similar when we make label predictions:
(x) = Ogl + wepās((x), (x) = (()) + Wapbsa(x) This step ensures that our model mutually benefits from the coefficient predictor and the proxy pseudo-cous predictor. The initial components
are learned unsupervised and do not make up for the training of the predictors. By introducing the
joint update, we connect the two predictors more closely.
The joint training step of EMM addresses an important problem with the mixture model formulation, where the model prediction is restricted by the mixture components μ. Because the weight coefficients
5

---

5 TH S 1, the label prediction for yt can only be as great as max; Pki. If we only make updates to
(1) (0) K the mixture components using Ogd dis - > Th (Xn yus and 6 b + +Σ6- th (xx))L
K
Ynt), the rare labels will still suffer from the label imbalance which will be reflected in max Akd max; bai
Pel By incorporating the joint training, we make the prediction based on (g((
2 + (x,ūm(x), (x) = U + w,,bb((),, allowing the model to better fit the labels using instance- wise predictions (x)(x), (x). The soft margin multi-label loss effectively brings the benefits of binary relevance machines into model training because it promotes positive predictions through the multi-label one-versus-rest formulation.
Complete EMM learning process. Having established the weight coefficient training step and the joint multi-label training step, we integrate them in a complete learning process. To start with,
we have the EM-learned initial clusters (0)
P I. and the optimal 11(() which reconstructs the labels when combined with P K In one learning round, the model first goes through pre-training stage (0)
(0)
of O- (e(x)) to fit the set of (T()) optimized for the initial PkxL Then, we move on to the joint training stage where we alternate between training the coefficient predictor and jointly training the full model to fit the labels y. Each coefficient predictor training step is the same as the pre-training stage, while the joint training step is described above. The model will continually improve the quality of label clusters to maximize the ability to recreate the labels in the joint training step, each time followed by the evidential learning of optimal π to keep up with the new label clusters. The complete training process is presented in Figure T At this stage, we have combined the advantages of Bayesian mixture models, deep evidential models, and a bi-level multi-label problem formulation to obtain a powerful multi-label classification model. Next, we introduce how the evidential flavor can provide fine-grained uncertainty information and benefit active learning.
3.4 Active Learning Strategy
In order to select the most informative samples given the small initial budget, we adopt an uncertainty- oriented selection strategy. From the proposed EMM model, we can obtain uncertainty information from three sources: weight coefficient branch, proxy pseudo count predictor, and the final label predictions. The first part of the uncertainty information is from the evidential model that predicts the weight coefficients. The evidential model naturally decomposes into the aleatoric uncertainty B[o and the epistemic uncertainty EP. For AL purposes, we should target the samples that give us the most epistemic uncertainty, which can potentially improve the model's knowledge of the unknown. From the weight coefficient perspective, this criterion searches for the least confident samples based on current model. Selecting these samples will help us quickly gain knowledge of the connection between input features and the weight coefficients, which is the determining factor for predictive performance. Thus, the first part of the selection function is
(x) Are (x) 1/ (x) 1)
The second part of the uncertainty information comes from the proxy pseudo counts. We can compare the current components and the updated components when the proxy pseudo counts for x are added, and select the samples that would introduce more difference to the current model: (x) = -CosineSimilarity (1,, (x)) = (x)((x) |=]]] Because the label clusters play the most
important role in recreating the label space, we should try to capture much latent label correlation as possible. This requires sufficient exploration in the label space. Selection criterion (x) does exactly this by searching for data samples that are potentially the most different from the currently captured label space. The last part of the uncertainty information is computed over the final label prediction. Since the full model is a mixture of Bernoulli, we can compute the expected covariance of the predicted label distribution. Here, we adopt the point estimate as in existing methods:
covlý = Σ Th (diag( - 14)) + (xH4T) p())x) p()1x) T
(5)
Note that ideally we would like to use the conditional entropy H [y][]] to measure the uncertainty raised due to observing input ær However, the entropy of a mixture random variable is hard (or intractable) to compute because of the sum the expression. But under mild conditions, we can show that y, as an average of the combination of weights and components, converge to normal

---

distribution so that the cou (z)a,, which is easy to compute, can be used to quantify the uncertainties in the mixture random variables.
Let .1, be the i.i.d samples that encapsulates the coefficient-compnnene pairs where Φi =
mi
[
y1
with mean m
and covariance >. Let y/r
where EL
TiPLi/
mi
18 [T
VT
k= TAPRI. Then, according to the Multivariate Central Limit Theorem, we have N( m
)
K R
So we can leverage the dominant term in the entropy of multivariant Gaussian, In (ov(y/=)) as
surrogate measurement of H [y] since the number of components is relatively large.
The selection score is Ay (x) log (cov///x]] /x|]- This criterion captures the expected information gain evaluated on the final label predictions when including the unlabeled samples. aggregates the predictions from the weight coefficient predictor and the proxy pseudo-count predictor, and shapes the fine-grained uncertainty in a global view. By focusing on epistemic uncertainty from the evidential model for weight coefficients, differences in label clusters indicated by proxy pseudo counts, and the covariance in label predictions, we devise a comprehensive multi-source intertainty-based selection score (MSU) A(x) Ax. (x) + xx,, + m)As(x)) Compared to a single uncertainty score, the integration of these uncertainty measures facilitates a targeted exploration of the data space and enables the identification of the most informative samples within constrained budget.
Experiments
We conduct experiments both synthetic and real data demonstrate the effectiveness of EMM. We use the synthetic data experiment to show how label clusters of the mixture model can capture various label compositions and correlations. We then verify the AUC performance of EMM on real-world multi-label datasets, along with rare label analysis and ablation studies.
4.1 Synthetic Data Experiments
Feature Space
To demonstrate the effectiveness of the EMM model and the MSU sam- pling strategy, we design a synthetic dataset that can verify each of the proposed functionalities. The syn- thetic dataset contains mostly geo- metric feature related labels, along with few carefully designed labels. The input features of the data points consist of 16 clusters distributed as m-dimenniionl Gaussian's s. In other words, each point is sampled from one Gaussian cluster. The clusters have different means and universal variances such that they have slight overlaps. The geometric feature related labels indicate which Gaussian the point is sampled from. We denote all the geometric-related labels as Lyeo'ss 's (all 16 of them) for simplicity. While Lgeo's 's mainly indicate the location of the data samples feature space and do not carry correlations in themselves. we construct the labels of interest using certain groups of them to ensure complex label correlations that are also feature-rooted. These labels of interest include rare label L1, which has a frequency as low as 5% of regular label; couple of highly-correlated labels 1.2 and L33 which share similar features; and L4, which only depends on L2 and L3 and is dependent on other labels instead of input features. Specifically, L1 is assigned to data samples randomly with a low probability of 5%. L2 and L3 are randomly sampled from the same quarter of the feature space. Then, L4 is generated following the rule L4= L2 U L3. Besides these labels, we also append a set of non-geometry information- based labels Lnon- geo- which prevent the problem from being purely geometry- based. The label composition is shown Figure
Geo-based Labels
h
5%
Rare Label
30%
L
L
Labels
of Interest
Non-geo-based Labels
Figure 2: Visualization of label composition in the synthetic dataset. For visualization we show the geometric clusters in 2D, while they are high dimensional Gaussians in practice.
Capturing label correlations. In one example of our experiments, we train the EMM with label clusters. Among these clusters, one has particularly high weight (41,22) for 1.2. Simultaneously, the weight ((1,13) for 1.3 low while the weight (41,14) for 14 is high. Such behavior will ensure that the co-appearing L2 and L4 are captured during the prediction process. In the 6-cluster setting,
7

---

Table 1: The relationship between average uncertainty scores, label cardinality, and rare labels
ly|33
|y| 3
YL1
YL1
Average Ar. (x) 46.5
36.1
52.9
31.9
Average Ay
0.079
0.070
0.138
0.069
only PCL.) and (a.)) have higher weights for L2, and in both clusters ((..r4) is similarly high as M(.,123. Mcanwhile, in P(..)) we have a high weight ((3,aa) for L3, in which case (33.44) for L4 is low (< 0.01). Furthermore, in cluster [- both ((4,22) and [[4,13) have slightly higher values but ({4,24) is low (< 0.01) so L4 does falsely appear.
We further test various numbers of components and quantify the label cor- relations described in those label clus- ters. The results show that the average CosineSimilarity
Conecction
fixe
updated
(., L2), (4), is 0.91 meaning that the positive corre- lation between L2 and L4 is always captured. Meanwhile, the average sim(pt..E3). M{.,444) is as low as 0.12 showing the lack of correlation be- tween L3 and L4. More specifically, when neither (((.I22 or (...3) is in- significant, R() (CC,4)) drops to 0.31, indicating that the fine rela- tionship of "if and only if" is well- captured by the mixture model.
km5
inrlevant
Clusters with L2, L3, L4
(b) Clusters with L1
Figure 3: (a) A visualization of the labels clusters concerning L2, L3, and L4; (b) A visualization of the labels clusters concerning L1 with and without updating with proxy pseudo- counts. "fixed" is the original unsupervisedly trained 41, "updated" is an updated (1,L1 where x (x1) = 0.83 meaning 11. dominant, and "irrelevant" an updated P1, (x2) where Ti (x2) 0.18 meaning this cluster f1.
does not contribute much to the prediction of x2.
Prediction enhancement by proxy pseudo-count combination. Although the mixture model is great at capturing the label correlations with label clusters, it is not always good at predicting rare labels. For example, in the 6-cluster setting, the largest weight for LI is (1[2.113 0.016 because of the extremely imbalanced label distribution. In this case, even the model predicts solely (22...) for a sample x (T2 = 1, म. = 0, k # 2), the prediction L1 is 0.016. This small value creates difficulty in converting the predicted score positive label prediction.
Rare label and correlation discovery by uncertainty quantification. As for actively selecting data samples, we study the correlations between Ar, (x), (x(x., and the true labels of x. We show set of statisties in Table to analyze these behaviors. For Ar.. (x).. we compare the sampling score with the estimated unknown information of the corresponding pool samples. The unknown information evaluated from both the feature and label perspective, using the feature similarity and the label cardinality. The correlation between the feature similarity and the uncertainty score is -0.73, and the correlation between the label similarity and the uncertainty score is -0.41. From the label similarity we can also conclude the negative correlation between the similarity and uncertainty. For Ay (x), we specifically focus on the rare labels. On average, the samples containing less than labels have an uncertainty score 28.8% higher than the other samples and the samples containing rare labels have an uncertainty score 65.8% higher than the regular samples.
4.2 Real Data Experiments
Datasets and experiment settings. We conduct AI. experiments on representative real-world datasets including Delicious, Bibtex, Corel5k, Enron, and NUS-WIDE, covering multiple application domains [33 6). The number of labels ranges from 53 to 156, most of which are relatively rare in the entire dataset. We summarize the datasets and data preprocessing in Appendix D
Performance comparison. We compare the AL performance with competitive AL baselines:
GP-B*M uses Bayesian mixture model and conducts active sampling using the combined
predicted variance of multi-output GP and the label clusters (30]
MMC is model-adaptive (implemented with label ranking model) and involves predictive process
for the number labels. It samples instances based on the expected loss [38].
Adaptive Adaptive uses SVM margin and label cardinality inconsistency for data sampling. [18].

---

0.75
0.90
8.78
AUC
Micro AUC
0.74
Micro AUC
0.83
0.72
7.72
9.62
0.00
0.70
0.70
072
0.6
0.0
0.7g
AL round
AL round
AL round
AL rooudd
(a) Delicious
(b) Corel 5k
(c) BibТex
(d) NUS-WIDE
Figure Performances on real-world datasets (AU-ROC increases as we sample 5 rounds with 100
samples in each round)
1500
300
1000
200
AP1(%)
AP1(%)
100
500
26 Frequency
61 Frequency
(a) Corel 5k
(b) Delicious
3000
1600
1200
AP](%)
API(%)
800
1000
20
10
40
22
Frequency
Frequency
(c) BibTex
(d) NUS-WIDE
Figure 5: Average precision improvement (API) for rare labels
CVIRS combines the margin based margin ranking and label inconsistency for data sampling [26].
the area under the ROC curve the main criterion. We start with 2% initially labeled samples for datasets Delicious, Bibtex, Corel5K, and 0.03% for JUS-WIDE. The initial labeled set contains minimum of positive instance per label to ensure that binary solutions can be trained. For EMM and methods that can perform batch active learning, we sample 5 rounds with 100 samples selected in each round. For single-batch baseline methods, sample 500 rounds to obtain the same number of labels in the end. The base performance of classification models varies as some baseline methods use binary-SVMs (Adaptive, CVIRS), some use strateg--speciff models such as the label-ranking model (MMC) and the GP-B²M model.
For AL comparison,
We also include configuration EMM-entropy that the proposed EMM model and a simple entropy-based selection strategy as an additional baseline, showcasing the performance gain from the proposed sampling (MSU selection) its own. From Figure 4 model makes better predictions using the same amount initial labels compared to the SVM model, which explains the advantage the starting point. Although the label ranking model or the GP-B"M model may also have good performance at the starting point, they are restricted by specific sampling methods. To separately verify the advantage brought by the uncertainty quantification, show that our MSU selection always has an advantage in selecting better AL samples compared to a simple uncertainty-batt based selection -atMooyy.
that the EMM
For more fine-grained analysis the model performance, we also compute the average precision improvement (30]. This shows how the rare-label predictions have improved using the EMM model compared to a fully Bayesian mixture model where the label clusters are completely global.
(P.(MMM) (GP-BBB
API, (%)
100%
(6)
AP (PP-B'M)
In Figure 5
we show the metric for the rarest labels on each dataset. The improved API on label is shown by blue bar above the API 1 0 axis, while the worse API performances are shown

---

0.75
0.7
0.78
0.8
Micro AUC
0.74
0.70
Micro AUC
0.0
0.72
004
7.72
0.70
0.70
0.63
0.82
0.6
003
0.0g
0.0
AL round
AL round
AL round
AL P.roundd
(a) Delicious
(b) Corel 5k
(c) BibTex
(d) NUS-WIDE
Figure 6: Ablation study on model components
0.0
Micro AUC
Micro alc
078
0.72
Micro AUC
8:72
0:3
6:7
77
24t0 yp-
Jul 1a
0.70
0.60
Am2pp1
0.73
0:77
A+1em2
om
Am1.gm2
0.60
Jahes
0:3
Juheett
Auhost
0.67
070
06g
o8g
AL round
AL round
AL round
AL round
(a) Delicious
(b) Corel 5k
(c) BibTex
(d) HUS-WIDE
Figure 7: Ablation study on balancing parameters
by the orange bars. The axis shows the number of times each label has appeared in the testing set. We can see that EMM has significant advantage on rarer labels.
Ablation study. We conduct an ablation study on model components (weight coefficient and proxy pseudo-count predictors) and the AL sampling method (balancing parameters À and 11). Since the proposed EMM model combines the evidential weight coefficient learning and the proxy pseudo- counts learning, we compare the complete model with two weakened configurations:
EMM rY reduces the weight coefficient leaner to a simple ridge regression model, and only combines the prediction with fixed Bernoulli mixtures as the label clusters. CEMM-fired uses the evidential learning of weight coefficients with only the fixed Bernoulli
mixtures the label clusters.
From Figure we can see that the evidential regression model predicts the weight coefficients better than simple regression model such as Ridge Regression, which shows the effectiveness of the first branch EMM (e() and 9* ()). We can also see that without the proxy pseudo-count updates, performance is not good, which shows the effectiveness of the second branch of EMM (g,, ()) and the joint training of the entire model.
From Figure we can already see that the proposed evidential uncertainty-Be -baed sampling outper- forms simple metric such as Entropy. From Figure 7 we can see that our MSU sampling strategy effectively benefited from the multi-source uncertainty information compared to the single-source uncertainty 77 = 0). However, the epistemic uncertainty from the evidential model is still the most important source of uncertainty the sampling performance decreases when we increase the balancing parameters for (x. and Ay (x) too much.
Conclusion
In this work, we introduced a novel Evidential Mixture Machines (EMM) model, which integrates deep evidential learning with multi-label classification framework of AL. This approach effectively tackle large and sparse label space, particularly addressing the challenges posed by sparse and rare labels. Our model's sophisticated uncertainty quantification and improved prediction accuracy set it apart from traditional Binary Relevance Models and other existing methodologies. The effectivenesss of the EMM model is demonstrated through rigorous evaluations both synthetic and real- world datasets, showcasing its superiority in diverse labeling scenarios. This advancement not only con- tributes to the development of more efficient MIL methods but also paves the way for future research in this domain. The potential for scaling this approach to larger datasets and adapting it to various domains offers exciting opportunities for further exploration and refinement.
10

---

Acknowledgments
This research was supported in part by an NSF IIS award IIS- 1814000 The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agency. We would like thank the anonymous reviewers for their constructive comments.
References
[1] Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. Advances in Neural Information Processing Systems, 33:14927-14937. 2020.
[2] Wentao Bao, Qi Yu, and Yu Kong. Evidential deep learning for open set action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 13349-13358, October 2021.
[3] Bertrand Charpentier, Oliver Borchert, Daniel Zügner, Simon Geisler, and Stephan Günnemann. Natural posterior network: Deep bayesian uncertainty for exponential family distributions. arXiv preprint arXiv:2105.5 1005504471,0000
[4] Bertrand Charpentier, Daniel Zügner, and Stephan Günnemann. Posterior network: Uncer- tainty estimation without ood samples via density-ban -hased pseudo-counts. Advances in Neural Information Processing Systems, 33: 13:1356-1367, 2020.
[5] Junyu Chen, Shiliang Sun, and Jing Zhao. Multi-label active learning with conditional bernoulli mixtures. Pucific Rim International Conference on Artificial Intelligence, pages 954-967. Springer, 2018.
[6] Dheeru Dua and Casey Graff. Uci machine learning repository, 2017.
[7] Nengneng Gao, Sheng-Jun Huang, and Songcan Chen. Multi-label active learning by model guided distribution matching. Frontiers Computer Science, 10(5):845-855 -555 2016.
[8] Bin Gu and Victor S Sheng. A robust regularization path algorithm for y-support vector classification. IEEE Transactions on neural networks and learning systems, 2845) 2016.
[9] Bin Gu, Victor S Sheng, and Shuo Li. Bi-parameter space partition for cost-sensitive svm. In Twenty-Furrth International Joint Conference on Artificial Intelligence. Citeseer, 2015.
[10] Bin Gu, Xingming Sun, and Victor S Sheng. Structural minimax probability machine. IEEE Transactions on Neural Networks and Learning Systems, 28(7):1646-1656 2016.
[11] Francisco Herrera, Francisco Charte, Antonio J Rivera, María J Del Jesus, Francisco Herrera, Francisco Charte, Antonio J Rivera, and María J del Jesus. Multilabel classification. Springer, 2016.
[12] Yibo Hu and Latifur Khan. Uncertainty-aware reliable text classification. Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, pages 628-636. 2021.
[13] Yibo Hu, Yuzhe Ou, Xujiang Zhao, Jin-Hee Cho, and Feng Chen. Multidimensional uncertainty- aware evidential neural networks. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 35, pages 7815-7822, 2021.
[14] Sheng-Jun Huang and Zhi-Hua Zhou. Active query driven by uncertainty and diversity for incremental multi-label learning. In 2013 IEEE 13th International Conference on Data Mining, pages 1079-1084. IEEE, 2013.
[15] Yeachan Kim and Bonggun Shin. In defense of core-set: A density-aware core-set selection for active learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages
2022.
[16] Armin Lederer, Jonas Umlauft, and Sandra Hirche. Uniform error bounds for gaussian process regression with application safe control. Advances in Neural Information Processing Systems, 32, 2019.
11

---

[17] Cheng Li, Bingyu Wang, Virgil Pavlu, and Javed Aslam. Conditional bernoulli mixtures for multi-label classification. International conference on machine learning. pages 2482-2491, 2016.
[18] Xin Li and Yuhong Guo. Active learning with multi-label svm classification. In IjCAL, pages 1779-1485. Citeseer, 2013.
[19] Chunfeng Lian, Su Ruan, and Thierry Denæuxx An evidential classifier based on feature selection and two-step classification strategy. Pattern Recognition, 48(7):2318-222) 2015,
[20] Haitao Liu, Yew-Soon Ong. Xiaobo Shen, and Jianfei Cai. When gaussian process meets big data: A review of scalable gps. IEEE transactions on neural networks and learning systems, 31(11):4405-443 2020.
[21] Nis Meinert, Jakob Gawlikowski, and Alexander Lavin. The unreasonable effectiveness of deep evidential regression. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 37, pages 9142-9142, 2023.
[22] Deep Shankar Pandey and Qi Yu. Multidimensional belief quantification for label-efficient meta-learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14391-14400, June 2022.
[23] Deep Shankar Pandey and Qi Yu. Evidential conditional neural processes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 9389-93977, 2023.
[24] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[25] Guo-Jun Qi, Xian-Sheng Hua, Yong Rui, Jinhui Tang, and Hong-Jiang Zhang. Two-dimensional active learning for image classification. In 2008 IEEE conference on computer vision and pattern recognition, pages 1-8. IEEE, 2008
[26] Oscar Reyes, Carlos Morell, and Sebastián Ventura. Effective active learning strategy for multi-label learning. Neurocomputing, 273:494-508, 2018.
[27] Erik Schultheis, Marek Wydmuch, Rohit Babbar, and Krzysztof Dembezynski. On missing labels, long-tails and propensities in extreme multi-label classification. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1547-1557,
2022.
[28] Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. Advances in neurul information processing systems, 31, 2018.
[29] W. Shi, X. Liu, and Q. Yu. Correlation-awart multi-label active learning for web service tag recommendation. In 2017 IEEE International Conference on Web Services (ICWS), pages 229-236, 2017.
[30] Weishi Shi, Dayou Yu, and Qi Yu. A gaussian process-bayesian bernoulli mixture model for multi-label active learning. Advances in Neural Information Processing Systems, 34:27542- 27554, 2021.
[31] Weishi Shi and Qi Yu. Fast direct search in an optimally compressed continuous target space for efficient multi-label active learning. In International Conference on Machine Learning, pages 5779- 2019.
[32] Mohan Singh, Eoin Curran, and Pádraig Cunningham. Active learning for multi-label image annotation. Technical report, University College Dublin. School of Computer Science and Informatics, 2009.
12

---

[33] Grigorios Tsoumakas, Eleftherios Spyromitros- Xioufss, Jozef Vilcek, and Ioannis Vlahavas. Mulan: A java library for multi-label learning. The Journal of Machine Learning Research, 12:2411-2414 2011.
[34] Deepak Vasisht, Andreas Damianou, Manik Varma, and Ashish Kapoor. Active learning for sparse bayesian multilabel classification. Proceedings of the 20th ACM SIGKDD international 2014.
conference Knowledge discovery and data mining. pages
[35] Dongxia Wu, Ruijia Niu, Matteo Chinazzi, Alessandro Vespignani, Yi-An Ma, and Rose Yu. Deep bayesian active learning for accelerating stochastic simulation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2559-2569, 2023.
[36] Guoqiang Wu, Chongxuan Li, and Yilong Yin. Towards understanding generalization of macro-auc in multi-label learning. In International Conference on Machine Learning, pages 77540-37570. PMLR, 2023.
[37] Jian Wu, Anqian Guo, Victor Sheng, Pengpeng Zhao, Zhiming Cui, and Hua Li. Adaptive low-rank multi-label active learning for image classification. In Proceedings of the 25th ACM international conference on Multimedia, pages 1336-1344, 2017.
[38] Bishan Yang. Jian-Tao Sun, Tengjaa Wang, and Zheng Chen. Effective multi-label active learn- ing for text classification. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 917-926, 2009.
[39] Min-Ling Zhang and Zhi-Hua Zhou. Multilabel neural networks with applications to functional genomics and text categorization. IEEE transactions on Knowledge and Data Engineering. 18(10):1338-135) 2006.
[40] Yi Zhang. Multi-task active learning with output constraints. In Proceedings of the AAAI Conference on Artificial Intelligence volume 24, pages 667-672, 2010.
[41] Yu Zhang, Bowen Jin, Xiusi Chen, Yanzhen Shen, Yunyi Zhang, Yu Meng, and Jiawei Han. Weakly supervised multi-label classification of full-text scientific papers. KDD '23, page 3458-346 New York, NY, USA, 2023. Association for Computing Machinery.
13

---

Appendix
Organization. In this appendix, we provide additional details the work, including a summary of notations in Appendix [A] detailed evidential model interpretation in Appendix B detailed algorithm blocks C and the experiment details with additional results Appendix
Summary of Notations
Table 2: Summary of key notations with definitions
Notation
Definition
(1)......)) Wt.
Multi-label vector
Data feature vector
2t
I Bernoulli y,, pat)
L-variate Bernoulli random variable.
Pbl
Beta random variable
(agt, bbt)
Parameters of Beta distrihution.
Gaussian random variable (component coefficient).
p (7,
Parameters of inverse normal gamma distribution.
Gaussian distribution in the evidential NIG prior with mean
M(Th: +
and variance of
(a" (3)
Inverse Gamma distribution in the evidential NIG prior with shape parameter and scale parameter of 3
St(m:))))) Evidence(E)
Student distribution
The integrated model evidence
A(x)
Active sampling function
B Detailed Evidential Model Interpretation
In this section, we introduce the details of the evidential regression model that predicts л. Specifically, the Student-t predictive distribution is obtained through:
p(T|E,P) p) = / 02
P( T, o N)IGG( p)drdd
((++1))
(1+ r( - 7) (
V
I(a)
V 3 + v)
2 +v)
=St(=;
B(1 + v)
2a
(7)
va
Among the posterior NIG parameters, 7 is the expected mean of the predictive distribution:
= E Bp(s [ক]
xp(T|.. p)dr
(8)
According to [22]. the other three evidential NIG parameters v, a, and accumulate as the observa- tions increase. The updates of these posterior parameters work as follows:
N
V
N
= = N.
IN
=a
(9)
VN
VN
2
N
3N 3
na
Tn
N
7)2
(10)
2(
N
N=1
N=1
As can see, each observation impacts the confidence of the model through these three parameters. and a can seen as the pseudo-counts and directly impact the model evidence i.e., quantify the confidence on the prior mean and the prediction of target data sample, respectively. Furthermore,
14

---

a large 5 leads to low confidence in the model's prediction, which implies a lack of evidence. Additionally, each observation increases the pseudo-count of by I, and by 2. Thus, obtain the final form the overall evidence = = 2 + # and the evidence-based regularization term LREG-
C Detailed Algorithms
We provide our proposed model and AL strategy two main algorithms shown in Algorithms 1&2: the first one describes the entire AL process, while the second one describes the detailed training process of EMM.
Algorithm Active Learning with EMM Model (Outer Loop)
Input
Total number of AL. rounds: T, AL batch size lbç
Unlabeled pool: St. annotation method: h. x + y
Model at step t: fe, (x), consisting of e(x), gr (x), (x(x)
AL sampling strategy A fo, (x) x Su + IR,
Learning objective L: [x) X IR, Output: Annotated training dataset: S1, final model før Randomly select Sy
// Balanced Split Method for Initial Training
for t 1 to T do
Train preset components O, on Sr.
// E-M: Bernoulli Mixture Training // Linear Program Optimization
2. Compute preset weights HI with Oo
// LavIo f) 4. Completely train EMM model (e() and g* O, ())) to y // Alternating LEND/ Lootthergi // Mulri-source (x) // Prepare for next round
Pre-train for weight coefficient predictions (e() and 9r ())) to fit HI
Active sample b, from Sy based on A
6. Update the pool and training set St St bb, Sr. St Ub.
end
Algorithm 2 EMM Model Training (Inner Loop)
Input Modell fo, (x), consisting of e(x), In (x). (x))
Learning objective L: L: So, (x) x - (Types: LEVIo (Ho), LsottMargin (y))
Output Trained model f', updated label clusters O,, predictions ý
for to epochspretrain do Train E() and 9x(() using LEVID (II.)
end for j 1 to epochstrain do
for k = 1 to epochst do Freeze e() and ()
Train (n() using LsortHargin(y)
(x( is predicted from 9p () for each point, 0/ = -O + Σn, Wnew (xx)))
end
for to epochs, do
Train e() and 9)() using LEVID (IIo)
end
end
D Experiment Details and Additional Results
D.I Experiment settings
Our experiments are performed on clusters with NVIDIA A6000 and NVIDIA A100 graphic cards and Intel Xeon Gold 6150 CPU processors. The code is implemented with PyTorch 24 The runtime of the experiments varies depending the size of the unlabeled pool. Compared to traditional
15

---

models, the evidential deep learning model takes a longer time to train. However, compared to Bayesian models label ranking models, the inference time is significantly shorter.
For our main results, we pre-train the label clusters using an E-M algorithm and obtain initial optimal weights for the labeled training set using linear programming optimization. The evidential model is trained for around 5000 epochs to fit the weight coefficients π,, and the joint training step is trained iteratively for 100 epochs each round.
D.2 Additional Baseline Comparison
Here we show additional AL comparison with two other baselines:
AUDI uses label ranking mechanism, where dummy label is used to separate the positive and negative labels. Its sampling function is based on modified cardinality inconsistency measure [14]
CS uses a compressed sensing mechanism combined with GP predictions 31
0.76
0.80
0.75
0.78
0.76
0.73
0.74
Micro
0.71
Micr
0.72
0.70
EMM
0.70
EMM
0.69
AUDI
0.60
AUDI
0.68
0.66
0.67
0.64
AL round
AL round
(a) Delicious
(b) Corel 5k
Figure 8: Additional active learning performances comparison on smaller-sized wealwwrld datasets (the AUDI and CS baselines become computationally expensive on larger datasets)
From Figure we can see that both baselines do not perform well at the low budget, while EMM still outperforms them. The implementation of AUDI and CS BibTex and NUS-WIDE computationally too demanding and we would expect similar behavior given the labeling budget.
Most recent multi-label works also focus on AUC optimization. So, we also add discussion on the macro-AUC. For example, [36] achieves the state-of-the the--rtt macro-AUC results on bibTex, Corel 5k and Delicious. In work, due to the poor macro-AUC performance of some baselines, mainly present micro-AUC results. However, here we provide the macro-AUC results using 80% training data (except for NUS- WIDE where too expensive) compared to the methods in [36.. We can see that the macro-AUC close on common datasets. Additionally, the AUC optimization method is orthogonal to our evidential model. We could incorporate AUC-based loss regularization into our joint-label training step. Due to the complexity, leave these studies to future work. Instead, focus the AL improvements in this paper.
13,000 training samples because full- full-training is
Table 3: Performance metrics of EMM across datasets
EMM
Delicious
Corel 5k
bibTex
NUS-WIDE
micro-AUC 0.8021
0.8063 0.6613
0.8669 0.8153
0.8625
macro-AUC
0.7256
0.6396
Table 4: Repetition of EMM metrics across datasets
EMM
Delicious
Corel 5k
bibTex
JUS-WIDE
micro-AUC
0.8021 0.7256
0.8063 0.6613
0.8669 0.8153
0.8625
macro-AUC
0.6396
16

---

Table AUC-surrogate performance on selected datasets
AUC-surrogat
Delicious
Corel 5k
bibTex
Au1
0.7633
0.6645
0.8693
Aa2
0.8044
0,5703
0.9299
D.3 Additional Ablation Study
In this set of experiments, show results using more configurations of À and n.. As explained in the main paper, these balancing parameters work well with a moderately small value. With À and n both around 0.1 to 0.01, able to obtain stable AL results. However, if the values are set too large, the performance may degrade.
0.78
0.27
Micro AUC
0.75
Micro
0.88
0.0.0
0.86
01.0-1
0.72
A- 1-1-0
0.64
2 1. L.-00 0.1
A-1.010
1.
0.70
0.02
AL round
AL round
(a) Corel
(b) BibTex
Figure 9: Additional ablation study on balancing parameters: different values
0.80
0.74
Micro AUC
Micro AUC
0.76
0.70
0.74
0.68
A 0.01. 0.02
a.l.e 2000
D.ge1
0.66
A-0
1
0.70
0.64
AL round
AL round
(a) Corel 5k
(b) BibTex
Figure 10: Additional ablation study balancing parameters: single uncertainty source
When À 0, 1 1, O, get the combination of A and single source of label uncertainty A₁ or Ag- As we can see, the combination of all three works the best. In Figure we show the comparison between different K values. As we can see, extremely low number of clusters is not sufficient for achieving good model performance, while large number is much more costly and also suffers from overfitting. The latter problem might harm the AL sampling more see the K = 10 case performs even worse than K 3.
II
D.4 Additional Uncertainty Analysis
We can also evaluate the quality of uncertainty estimation using true labels of the pool samples after AL experiments. Here, we present example of the rare-label uncertainty being captured by both the cluster difference prediction and the covariance of the predicted labels. From Table (results obtained on BibTTex), can see that similar to the synthetic data case, the uncertainty metric captures rare labels well, although difference is smaller because there are more labels in total.
E Limitations, Future Works and Broader Impact
While EMM offers significant advancements in multi-label active learning, there are some limitations to consider. First, the model's complexity and computational requirements may pose challenges when
17

---

0.80
0.76
0.74
Micro AUC
0.78
0.72
0.70
0.76
0.88
0.8%
AL round
AL round
(a) Corel 5k
(b) BibTex
Figure 11: Additional ablation study number of clusters K
Table 6: The relationship between average uncertainty scores, label cardinality, and rare labels
ly|<3
yy 2 3
Average Ar, (x) 3,886
3.062
3.834
3.196
Average Ax(x)
0.021
0.064
0.018
0.013
applied to extremely large datasets in real-time applications. Second, we have not integrated the Bayesian selection for K in EMM, making the choice of K crucial. Our future works include more adaptive model structure that allows a dynamic as the number of label clusters, along with more lightweight training and adaptation process of the model. We also look to integrate adaptive testing and validation approaches for the AL process.
EMM has the potential to significantly impact various fields that rely on multi-label classification. In domains such as healthcare, bioinformatics, and environmental science, where data labeling is often expensive and time-conum,ing, our model can optimize the use of limited labeling resources, leading to more efficient and accurate predictions. This can accelerate research and development these fields by enabling the discovery of new patterns and correlations. Moreover, the model's sophisticated uncertainty quantification can improve decision-making processes in critical applications, where understanding the confidence and reliability of predictions is crucial. However, the implications of label clusters need be carefully considered applications.
Source Code
git
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main contribution of the paper is the EMM model and the AL strategy using EMM, which are clearly stated in both the abstract and the introduction.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of the work in Appendix
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: For the evidential model interpretation, we provide detailed analysis in Ap- pendix B
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: The source code is provided in Appendix F The detailed experiment settings are provided in D.I
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The source code is provided in Appendix The datasets are open access. The detailed experiment settings are provided in D.I
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The source code is provided in Appendix F are provided in D.I
The detailed experiment settings
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yss]
Justification: We repeat all experiments 3 times with random seeds. The error bars are drawn in main figures.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Question:
19

---

Answer: [Yes]
Justification: have provided the detailed information about computational resources needed to obtain the experimental results in the paper in the D.1 section of the appendix.
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our research strictly follows the NeurIPS Code Ethics. Our study does not involve human subjects. We keep transparency and honesty and as being small-data learning paradigm, the model training has minimal environmental impacts. We also complies with all relevant laws and regulations.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We address potential biases and boarder impact of the research in section B of the appendix.
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: NA
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The programming code and data used in our study are properly credited and their licenses are respected. Specifically, we used Py PyTorch for our implementations and datasets from the UC Irvine Machine Learning Repository, both of which are properly cited and used according to their respective licenses.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
20


Reviews:
[{"review_id": "pHTUQqAYTs", "reviewer": "Reviewer_dBCp", "summary": "This paper introduces the Evidential Mixture Machines (EMM) model, which addresses the multi-label active learning tasks, particularly in rare-class scenarios. EMM uses a mixture of Bernoulli distributions to capture label correlations and uses evidential learning to quantify uncertainties for more informed active sample selection. The model dynamically updates label clusters using proxy pseudo counts.", "questions": "Please see the weakness part.", "limitations": "The author has already addressed potential biases and boarder impact in their appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The usage of a mixture of Bernoulli dist. is interesting, By using it, the model effectively captures label correlations, which is crucial for multi-label classification where labels are often interdependent.\n2. The usage of evidential learning is also interesting, it can be used for predicting weight coefficients and provides fine-grained uncertainty quantification. This allows the model to differentiate between epistemic and aleatoric uncertainty.\n3. The integration of multi-source uncertainty in the active learning strategy promotes active learning performance (shown in the ablation study), it improves the selection of informative samples, leading to better performance with fewer labeled instances.", "weaknesses": "I have 2 concerns:\n1. Is the model robust to outliers? In real-life data scenarios, the existence of outliers is normal. Outliers can cause the proposed model to learn incorrect or spurious correlations between labels, (label noise issues also have a similar effect). If outliers have unusual combinations of labels, they can distort the representation of the label clusters.\n\n2. Compared with the GP-B2M paper's experiment, there are some inconsistencies, for example, on Bibtex and Corel5K datasets, GP-B2M performs significantly better than CVIRS in GP-B2M paper, but performs similarly on this manuscript, can the author discuss more about it?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing constructive comments/suggestions. Below, we provide the response to the questions and comments.\n\n**Q1: Is the model robust to outliers?**\n\nThank you for the suggestion for this interesting challenge. We agree that being robust to outliers is an important challenge in real-world applications. In active learning, the scarcity of labels may cause the model to be more brittle. Although we do not deal with outliers directly in the proposed method, there is potential for AL with the existence of outliers using our fine-grained uncertainty analysis. For example, we could derive the predicted vacuity of data samples, which indicates how little support the evidential model has for these samples, to identify potential outliers. Better utilizing the EMM model for AL with outliers could be an important future direction of this work. We will add the discussion in the revised paper as well.\n\n**Q2: Different performance using the CVIRS baseline from previous papers.**\n\nIn our experiments, the performances are slightly different because we use a larger AL batch size. If we compare the AUC after selecting 500 samples, it is similar to those in the previous paper. We will further clarify the setting in the revised paper. In the GP-B2M paper, the same baselines were implemented using either BRM or the GP-B2M model. We adapted the CVIRS baseline using the EMM model but the results are inferior to those of EMM-entropy. To be consistent, we present the results using the binary relevance machines (BRM) model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A48eNdCN8G", "reviewer": "Reviewer_FbZQ", "summary": "This paper focused on multi-label classification problems in active learning settings, where the label relationship, especially for rare labels, is hard to learn. The authors proposed an Evidential Mixture Machine (EMM) that combines the mixture of Bernoulli with a deep evidential model, which allows joint learning of the weight coefficients pseudo label counts to model the proxy label distribution for each instance. This method allows quick inference time and twofold uncertainty measurement from the evidential posterior parameters and the predicted variability in final labels. Extensive experiments on real-world and synthesised datasets certified the stated advantages of the proposed methods.", "questions": "- Figure 7 in Appendix D.2 is ambiguous about the label relationships. Why is the Geo-based labels pointing to L1 rather than the Non-geo-based labels? Also, the arrows from L3 to L4 doesn't show the not relation.\n- From the results in Figure 3, the proposed active learning strategy doesn't have noticeable advantage than the EMM-entropy in (d) and (a). Does it indicate that for some dataset, the proposed active learning strategy doesn't add much information about uncertainty compared to entropy?", "limitations": "The authors have adequately addressed the limitations and potential societal impact of their work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors clearly stated the motivation for the proposed model, where in a multi-label classification problem, it is hard to learn the dependencies between labels or the correlation of rare labels, which will be amplified in an active learning setting. \n- The authors have provided a thorough critique of current research methods, highlighting their shortcomings. Their proposed method effectively addresses the limitations of existing CBM and GP-B2M based methods.\n- The idea of combining different uncertainty measurement sources is appealing, and the results show that the strategy is beneficial comparing to the strategy based only on entropy .\n- Comprehensive experiments and ablation study which support the claims", "weaknesses": "Few typos and confusion:\n- line 103 \"reuarization\"\n- lines 339-341 stated that EMM model outperforms the others in the initial round, which only showed in Figure 3 (c) Bibtex. Also, I can't see \"the GP-B2M model may also have good performance at the starting point\" in lines 341-342", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing constructive comments/suggestions. Below, we provide the response to the questions and comments.\n\n\n**Q1: Typos and confusion.**\n\nThank you for pointing out the typos. We will correct them in the revised paper. The EMM model does perform better on Corel 5k and BibTex in the initial round as can be seen in Figures 3(b) and (c). The EMM-entropy baseline should share the same initial performance as it also uses EMM as the base model. However, we made a plotting error by plotting the EMM-fixed (from the ablation study) instead of EMM-entropy in Figure 3(c). We will correct this in the revised paper (please see the pdf file under general response for the corrected figure). The description in line 341 is inaccurate. In our experiments, the label ranking model or the BRM model may have a good starting point compared to other baselines as shown in Figure 3 (a) and (d). \n\n**Q2: Confusion about Figure 7.**\n\nWe apologize for the confusion. In Figure 7, we meant to show the generation process of the Non-geo-based labels, not the actual logical relationships between them. L1 is randomly sampled from data points with certain Geo-based labels. L2, L3 and L4 are than generated based on the process in the description in Section 4.1. We will improve the quality of the figure and clarify this in the figure caption and Appendix D.2. \n\n**Q3: Comparison with EMM-entropy.**\n\nThank you for raising an interesting point about the potential lower information advantage on certain datasets. Regarding the performance comparison, because the EMM-entropy baseline also uses the proposed EMM model, the difference only comes from the sampling method. Thus, the AL curves can be close on some datasets. The assumption that the information from other components has less impact than entropy is quite plausible. As we can see from the ablation study, the performance change from the balancing parameters in the sampling function $\\lambda$ and $\\eta$ is less obvious on Delicious and NUS-WIDE than on Corel 5k and BibTex. To better support this finding, we compare the variance of the uncertainty scores (normalized over 500 pool samples) on these datasets. From the results below, we can see that the variance of the weight coefficient uncertainty is much higher compared to the other two components on Delicious and NUS-WIDE. Thus, the weight coefficient uncertainty might have a larger impact. The EMM-entropy uses the entropy criterion from the weight coefficient side, thus might perform closer to the integrated EMM method on these datasets. We agree that this could be a very interesting factor in real-world AL applications and will further study the problem in future works.\n\n| Var($\\mathcal{A}$)      | Delicious  | Corel 5k  | bibTex    | NUS-WIDE  |\n|-------------------------|------------|-----------|-----------|-----------|\n| Var($\\mathcal{A}_\\pi$)  | 1.219e-3   | 6.074e-4  | 7.683e-4  | 1.984e-3  |\n| Var($\\mathcal{A}_\\theta$)| 2.561e-5  | 1.004e-4  | 2.635e-5  | 9.805e-5  |\n| Var($\\mathcal{A}_y$)    | 4.473e-5   | 1.028e-4  | 4.763e-5  | 1.439e-4  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dJpCLC2X26", "reviewer": "Reviewer_fLE1", "summary": "This manuscript investigates multi-label active learning, a critical issue in contemporary machine learning. To address this challenge, the authors introduce a novel evidential mixture machines (EMM) model, which provides an uncertainty-aware connection from input features to the predicted coefficients and components. The performance of the developed method is demonstrated through simulations on synthetic and real-world datasets.", "questions": "Specific comments and questions:\n1.\tHow do the authors determine the parameters in the student-t distribution?\n2.\tIn Section 4.1, the authors mention the large or small value of the weight. How does the outcome change with different weight values? Is the performance of the EMM model closely related to the weight? If so, how is the weight determined in different scenarios without prior information about the labels?\n3.\tIn Figure 6, although EMM has a significant advantage on rarer labels, several high orange bars remain in Figure 6 (b). Could the authors explain this? Additionally, please add numerical labels to the y-axis below the API=0 axis.\n4.\tWhat is the computational complexity of the EMM method compared to state-of-the-art (SOTA) methods, and how does it scale with the size of the dataset?\n5.\tPlease explain how this method can be employed in practical applications and what its strengths are compared to other SOTA methods (beyond accuracy).", "limitations": "The authors have adequately addressed the limitations of their work.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors argue that the EMM model can produce a richer multi-source uncertainty metric than simple uncertainty scores and enhance prediction accuracy. The model sounds interesting.", "weaknesses": "However, there are several typographical errors in the manuscript, and the authors should thoroughly review and revise the text. The authors should improve the presentation. Additionally, more details about the model should be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: How do the authors determine the parameters in the student-t distribution?**\n\n The parameters are a crucial component of the Bayesian nature of the evidential model. The parameters for the student-t distribution ($\\pi,\\gamma,\\frac{\\beta(1+\\nu)}{\\nu\\alpha},2\\alpha$) are all obtained from network outputs, as specified in Line 163 and the following parts. More details of the interpretation are given in Appendix B. \n\n**Q2: How does the outcome change with different weight values?**\n\nIf the question refers to the coefficient weights $\\pi$, they are obtained from the evidential network predictions. A larger $\\pi_k$ will indicate that the sample can be better explained by label cluster $k$. This is how we connect the data sample to the Bernoulli mixture label clusters. The initial label clusters are learned through E-M on training labels, and the weights are learned throught evidenital training. If the question refers to the weights of the uncertainty components, they are explained in Section 3.3 and we provide an ablation study in Section 4.2 and Appendix D.4. \n\n**Q3: Several high orange bars remain in Figure 6 (b).**\n\nThank you for the suggestion. We will improve the figure in the revised paper. According to [1], the GP-B2M has a good performance on rare labels. We show that we have improved performance on most datasets (also in Figure 9). The Delicious dataset is a more imbalanced dataset, so it might be more difficult to perform well on all rare labels. \n\n**Q4: What is the computational complexity of the EMM method?**\n\nCompared to the GP-B2M model which requires a $\\mathcal{O}(N^3)$ complexity during prediction time (due to the inference process of GP), we have a linear complexity in the number of samples. During training, the complexity is at the same scale as a standard evidential neural network. \n\n**Q5: Please explain how this method can be employed in practical applications and what its strengths are compared to other SOTA methods (beyond accuracy).**\n\n\nFirst, EMM is an end-to-end multi-label model. Thus, in real-world applications, EMM could be utilized in the same way as other models, such as binary relevance machines and label ranking models. Then, it can be applied to the AL task as demonstrated in real-world experiments, where the fine-grained uncertainty could be utilized for active sampling. The main advantages of the EMM model include the scalability as a deep evidential model compared to traditional Bayesian models and the uncertainty information compared to standard neural networks. For a good example of the employment of EMM, please also refer to the response to Q1 from Reviewer 9WcB.\n\n[1] Shi et al. \"A gaussian process-bayesian bernoulli mixture model for multi-label active learning.\" NeurIPS 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QqhdAuSSyD", "reviewer": "Reviewer_9WcB", "summary": "The paper introduces an active learning approach based on Evidential Mixture Machines (EMM) that compress the large, sparse label space of multi-label problems into a more manageable weight coefficient space. This approach combines mixtures of Bernoulli with a Deep Evidentiary models, and it leverages multiple sources of uncertainty (i.e., evidential uncertainty and the predicted label embedding covariance) to improve the active sample selection,", "questions": "- please provide a reference for the claim in. lines 52-53\n- line 61 - please explain why a single pass is sufficient\n- line 321 - the details are not presented in. Appendix D\n- lines 329-332 - instead on percentages from the original  data set, use absolute numbers; also justify the reason for choosing those particular numbers", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper introduces a novel approach to multi-class active learning. The approach appears to be original, and it improves over existing approaches on four domains. Due to reasons explained in the next section, the quality and significance of this work are difficult to judge in the paper's current form.", "weaknesses": "The paper could be improved on two main dimensions. First, the authors could make their case more compelling and intuitive by (i) introducing a motivating real-world example, and (ii) adding a section with an intuitive running examples that explains in layman terms the contribution of each of EMM's components, together with the synergy among them. Second, the experimental section should be expanded with a comprehensive analysis of the 4 domains in section 4.2 and their properties (e.g., the distribution of the multi-labels, so that one can grasp the challenges in each dataset), and by adding to each graph the best-known performance on that dataset in via fully supervised training. Without this last bit of information, it is impossible to judge the impact of this approach: the correct question is how much can AL reduce the need for labeled data while maintaining SOTA performance, rather than \"how well can I do with X examples chosen via AL\". For example, multiple algorithms reach 95+% accuracy on BIBTEX; how should we compare such results with 73% Micro AUC? To allow the reader/reviewer to estimate the significance of EMM, the authors should provide and discuss these numbers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: (i) Introducing a motivating real-world example, and (ii) adding a section with an intuitive running example that explains in layman terms the contribution of each of EMM's components, together with the synergy among them.**\n\n\nThank you for this great suggestion! For the motivation real-world example, let's consider AI facilitated medical diagnosis, where the goal is to predict the likelihood of various diseases for patients based on their medical records. Each patient can have multiple possible conditions (classes). Traditional methods may struggle due to the rarity and interdependence of certain conditions. For example, diseases like rare genetic disorders occur infrequently, making it difficult for models to learn effectively from limited labeled data. Moreover, some conditions often co-occur or influence each other, such as diabetes and high blood pressure. The proposed EMM addresses these challenges more effectively by leveraging label correlations and predicts rare conditions by utilizing a mixture model approach that considers the dependencies among diseases. \n\nAs for the contribution of each component regarding question (ii), the active learning strategy employed by EMM ensures that the most informative patient records are selected for labeling, optimizing the use of limited medical resources. This approach not only improves prediction accuracy but also provides robust uncertainty estimates, which are crucial for making reliable medical decisions. Furthermore, evidential learning plays a critical role in this process by providing robust uncertainty estimates for the predictions. This is particularly important in healthcare and other knowledge rich domains, where the cost of incorrect predictions can be high. Evidential learning allows the model to quantify its confidence in each prediction, enabling the active learning component to prioritize labeling the most uncertain and potentially informative records. This ensures that the model not only improves prediction accuracy but also provides reliable and interpretable predictions as active learning goes. Following the reviewer's suggestion, we will incorporate this example into our paper to make the proposed approach more intuitive and relatable to readers while highlighting the contribution of each key component. \n\n**Q2: How should we compare the results with full training?**\n\nWe appreciate the reviewer's insightful comment. Indeed, it is important to demonstrate how active learning (AL) with EMM reduces the need for labeled data while maintaining state-of-the-art (SOTA) performance. The complexity of SOTA metrics in multi-label problems means that accuracy can vary significantly depending on whether class imbalance is considered and how decision-making strategies are applied. Specifically, the 95+\\% accuracy mentioned by the reviewer may not be directly comparable due to different decision thresholds used by various methods. We chose the AUC metric because it focuses solely on prediction quality, independent of decision thresholds, providing a fairer assessment of multi-label classification performance.\n\nMost recent multi-label works also focus on AUC optimization. For example, [1] shows SOTA macro-AUC results on bibTex, Corel 5k and Delicious. In our paper, due to the poor macro-AUC performance of some baselines, we mainly present micro-AUC results. However, here we provide the macro-AUC results using 80\\% training data (except for NUS-WIDE where we use 13000 training samples as full-training is too expensive) compared to the methods in [1]. We can see that the macro-AUC is close on common datasets. Additionally, the AUC optimization method is orthogonal to our evidential model. We could incorporate AUC-based loss regularization into our joint-label training step. Due to the complexity, we leave these studies to future works. Instead, we focus on the AL improvements in this paper.\n\n|   EMM    | Delicious | Corel 5k | bibTex   | NUS-WIDE |\n|----------|-----------|----------|----------|----------|\n| micro-AUC| 0.8021    | 0.8063   | 0.8669   | 0.8625   |\n| macro-AUC| 0.7256    | 0.6613   | 0.8153   | 0.6396   |\n\n| AUC-surrogate | Delicious | Corel 5k | bibTex   |\n|---------------|-----------|----------|----------|\n| $A^{u1}$      | 0.7633    | 0.6645   | 0.8693   |\n| $A^{u2}$      | 0.8044    | 0.5703   | 0.9299   |\n\n**Q3: Other questions** \n\nFor the challenges of scalability and predictive ability faced by the Gaussian process due to the explosion of data size, see the survey paper [2]. While the fact that sparse Gaussian Process models are limited in their predictive ability is supported by [3]. We would like to add those references to the next version of the paper.\n\n\nIn line 61, we refer to the prediction phase of deep learning models, where only a single forward pass is required to generate predictions.\n\nWe apologize for missing the dataset summaries in Appendix D. We will include them along with the absolute samples used in the experiments in the next version of the paper. In addition, we want to assure the reviewer that all the datasets used in our experiments are publicly available, and our preprocessing steps follow standardized procedures. Therefore, the missing of this information in the current version does not critically affect the credibility of our results.\n\n**References**\n\n[1] Wu, Guoqiang, Chongxuan Li, and Yilong Yin. \"Towards understanding generalization of macro-AUC in multi-label learning.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] Liu, Haitao, et al. \"When Gaussian process meets big data: A review of scalable GPs.\" IEEE transactions on neural networks and learning systems 31.11 (2020): 4405-4423.\n\n[3] Lederer, Armin, Jonas Umlauft, and Sandra Hirche. \"Uniform error bounds for Gaussian process regression with application to safe control.\" Advances in Neural Information Processing Systems 32 (2019).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**General Response**

We would like to thank all reviewers for spending time to review our paper and providing constructive comments/suggestions. Below, we summarize some of our major responses:

- *Our contributions and how EMM could be used*: As stated in lines 83-87, our contributions include the novel integration of evidential learning and multi-label model in EMM and the corresponding active sampling method accompanying the model. Since reviewers asked about the usage of EMM, we clarify that it is first a multi-label model that can be readily used. Then, it is suitable for AL given the fine-grained uncertainty information. More detailed real-world examples are given in the specific responses.

- *Baselines and ablation studies*: Below in the specific responses, we clarify the confusion about the baselines and ablation studies and correct some minor errors. 

- *Typos and details*: We thank all reviewers for pointing out the typos and missing details. We will correct those in the revised paper and provide the details as requested.

Meta-review: Reviewers expressed mixed but generally positive views on the paper. While the novel approach to multi-label active learning was appreciated, particularly the use of Bernoulli mixtures and evidential learning, there were concerns about robustness to outliers and the clarity of the presentation. Some reviewers noted that while the model performs well, the paper could benefit from more intuitive explanations and improved experimental analysis.

As a meta-reviewer, I noticed that the paper does not explain in sufficient detail how the synthetic data were generated. There are explanations in Section 4.1 and Supplementary D.2, but neither is sufficiently detailed, making replication difficult based on the text alone. There is a link for anonymously hosted source code, but the link had expired by the time I looked at it. Despite these issues, the authors’ rebuttal addressed most concerns, and with some refinements, the paper is suitable for acceptance.
Decision: Accept (poster)