Evidential Mixture Machines: Deciphering Multi-Label Correlations for Active Learning Sensitivity

Authors: dayou  yu
weishi  shi
minghao  li
qi  yu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 21:26:59

Abstract: Multi-label active learning is a crucial yet challenging area in contemporary machine learning, often complicated by a large and sparse label space. This challenge is further exacerbated in active learning scenarios where labeling resources are constrained. Drawing inspiration from existing mixture of Bernoulli models, which efficiently compress the label space into a more manageable weight coefficient space by learning correlated Bernoulli components, we propose a novel model called Evidential Mixture Machines (EMM). Our model leverages mixture components derived from unsupervised learning in the label space and improves prediction accuracy by predicting weight coefficients following the evidential learning paradigm. These coefficients are aggregated as proxy pseudo counts to enhance component offset predictions. The evidential learning approach provides an uncertainty-aware connection between input features and the predicted coefficients and components. Additionally, our method combines evidential uncertainty with predicted label embedding covariances for active sample selection, creating a richer, multi-source uncertainty metric beyond traditional uncertainty scores. Experiments on synthetic datasets show the effectiveness of evidential uncertainty prediction and EMM's capability to capture label correlations through predicted components. Further testing on real-world datasets demonstrates improved performance compared to existing multi-label active learning methods.

Main paper: 
 introduction: 
 Active learning (AL) is a paradigm where we have access to abundant unlabeled data instances with a limited labeling budget [35] [15] . Most AL methods focus on the selection strategy that helps the machine learner achieve better performances with an informed selection of labeled data instances. However, while AL for standard classification tasks has been studied extensively, an important task that is multi-label classification has been largely overlooked. In real-world problems, each data instance may be associated with more than one labels [11, 39, 27] . For machine learning models, the difference between the multi-class problem where only one ground truth label is associated with a data instance and the multi-label classification (MLC) problem is fundamental. Classic solutions either transforming the MLC problem into multiple binary problems or directly build a joint learning problem for all labels. With the binary approach, we lose the common underlying correlations between input features and labels. The number of classifiers may be large due to the large label space, thus the separated training process can be costly. Also, many labels are relatively rare and may depend on other labels, and we can not learn such dependencies when we isolate these labels. With the joint learning approach, the biggest challenge is to combine common labels with rare labels in a balanced Active Sampling Criterion (MSU Score) + SoftMargin Loss eq(3)Decoder Figure 1 : Overview of evidential mixture machines way. For the rare labels, there might be few positive data instances. A typical model is likely to optimize the feature embedding according to popular labels or adopt the so-called "shortcut" learning, making a direct connection from the input features to the rare labels more difficult. Although we now have access to common and rare labels at the same time, it might not suffice to promote the learning of their correlations if we directly model the labels from the input features. These unique challenges of the MLC problem become more pronounced in AL settings, where the rare labels become even more scarce. They also increase the difficulty of obtaining high-quality uncertainty estimation, which is often crucial in many AL selection strategies as it allows us to know when the model "does not know" in order to make an educated decision on which data instances to label.To address these challenges, we draw inspiration from the mixture of Bernoulli model, which can model a large label space with a small number of mixture components. There are existing methods that try to capture the label correlation using such a model [17, 30] . However, to connect with the input features, they either resort to a purely conditional case, namely conditional Bernoulli mixtures (CBM) [17] or use a conjugate classification head, which is a Gaussian Process model (GP-B 2 M) [30] .For the former, a distinct set of label clusters is predicted for each data instance, meaning that the label correlations are completely separated from the learning tasks. This shortcoming makes the CBM model unsuitable for AL. The latter relies on the Gaussian Process (GP) which outputs the weights of the mixture components. For the intended task, a complete GP is too expensive [20, 16] , while a sparse GP has limited predictive ability. Unlike the CBM model, GP-B 2 M creates a set of global label clusters. However, the label prediction for rare labels remains challenging because they can only ever be as good as the best label cluster available. Furthermore, the uncertainty quantification is superficial because it only captures the approximate covariance of the label prediction and the variance of the GP predictions. This point estimate does not fully capture the unknown and is not sufficiently effective in AL.In this work, we propose to combine the mixture of Bernoulli with a deep evidential model [28] , both incorporating deep learning and enabling more fine-grained uncertainty analysis. The deep learning model needs only a forward pass during prediction time, which is much more efficient than estimating the predictive distribution in a random process model. The uncertainty analysis can now include each evidential prediction. This flexibility is crucial in improving the informativeness evaluation of multi-label data samples as mentioned above. Structurally, the proposed model is composed of a shared encoder that provides embeddings and two decoders that predict the weight coefficients and the proxy pseudo counts correction for the Posterior Beta which models the global label cluster, respectively, as shown in Figure 1 . The weight coefficient predictor is trained as a deep evidential regression model, which makes uncertainty-aware predictions of optimal assignments to the mixture of label clusters for each data instance. By using a deep evidential model as the classification head, we achieve the aforementioned improvement of efficiency during inference time. By sharing the encoder, we maintain a close connection beginning from the input features, through the weight coefficients, and to the label clusters. Compared to the CBM model, the proposed Evidential Mixture Machines (EMM) maintains a global label cluster. Compared to the GP-B 2 M model, the EMM model allows adjustment to the label clusters based on each data instance. Additionally, the uncertainty information can be captured by both the fine-grained uncertainty decomposition from the evidential posterior parameters and the predicted variability in final labels. The evidential uncertainty decomposition is implemented through a conjugate interpretation of the Normal Inverse Gamma parameters, which gives these parameters a more impactful pseudo count meaning. The label variability is assessed using the covariance approximation in predictions and the discrepancy between the global label cluster and the proxy pseudo counts. Apart from the novel sampling strategy, the model is trained using alternating loops of the evidential training step and a joint label-based training step, which strengthens the ability to accommodate individual samples.With both the demonstration on synthetic datasets and the AL evaluation on real-world datasets, we show the effectiveness of the EMM model. Our main contribution is threefold:• integration of evidential learning with multi-label classification through EMM,• principled uncertainty quantification for active sampling using EMM,• intensive evaluation showing performance gain by EMM for large label spaces, especially on sparse and rare labels. 
related works: 
 Multi-label classification. In the realm of multi-label classification (MLC) [41] [27], Binary Relevance Models (BRMs) have gained widespread use, leading to the development of various active learning (AL) models based on BRMs. Notable examples include employing the estimated reduction of a BRM loss function as an uncertainty criterion for data sampling, as demonstrated by [38] . A large portion of the multi-label active learning work do not require annotators to label all possible labels for a given data instance [25, 37, 7] . The main consideration behind these approaches is the significant reduction in annotator labeling costs. However, this strategy inevitably breaks the inherent connections between labels, making it impossible to comprehensively measure the informativeness of a data instance using label correlations. Additionally, models designed to handle partially labeled data are required, limiting the applicability of such methods. Therefore, in this paper, we will not compare our approach to these methods. Other approaches integrate the properties of the support vectors of individual support vector machines (SVM) within BRMs, using label correlation more as a means to simplify the querying process than to enhance active sampling [29, 32, 8, 9, 10] . While these methods incorporate label correlation to some extent, such as through label inconsistency [18] , label ranking [26] , or learning reularization [40] , they do not systematically capture label correlations, potentially leading to imprecise uncertainty measures in ML-AL contexts.Label correlations and multi-label AL. Some existing models attempt to explicitly capture label correlations or use latent embeddings to facilitate active multi-label sampling. For instance, the CBM model uses the approximate entropy of predicted labels for data sampling [5] , but its dependency on an external multi-class classifier for predicting component coefficients complicates AL due to the challenges in model selection and parameter tuning. Furthermore, CBM, designed primarily for MLC rather than AL, predicts distinct label clusters for each data sample without discovering global label clusters, thus limiting its effectiveness in multi-label AL [17] . Other approaches like correlationaware method for transfer learning [7] struggle with large and sparse label spaces due to their reliance on kernel functions for measuring label similarity. Compressed sensing (CS) techniques [34, 31] innovative in learning latent embeddings to capture label correlations but assume labels are drawn from a Gaussian distribution and are not efficient in AL, especially in early stages with limited training data. In [30] , a Bayesian mixture of Bernoulli model is proposed. A set of global label clusters are captured in a Bayesian manner. However, the inference process of the model is complicated and the fixed label clusters limit the predictive ability in the final label space.Evidential learning. Evidential models have been developed to enable fine-grained uncertainty quantification in deep learning (DL) models [28] [1]. These models introduce a higher order conjugate prior distribution over the likelihood distribution, and train the DL model to output the parameters of the higher order distribution [4] [3] . The higher order distribution enables the model to express the fine-grained uncertainty information. Evidential models have been successfully extended to classification [12, 19] , regression [1, 21] , action recognition [2] , OOD detection [13] , and metalearning problems [22] . We extend the evidential deep learning framework to our setting that leads to novel fine-grained uncertainty guided active-learning for multi-label classification. 
methodology: 
 Problem setting. In our multi-label active learning problem, the essential task is to predict a multivariate 0-1 label vector y = (y 1 , ..., y L ) ⊤ ∈ {0, 1} L from the input features x ∈ R M . For AL, we start with a small initial labeled set S L , and a large unlabeled poolS U (N L = |S L | ≪ N U = |S U |).The AL strategy A(x) : R M → R is a function that grades the instance x * ∈ S U according to its informativeness. The top graded samples will be labeled as a batch b t and added to S L . The building blocks of our proposed method include the Mixture of Bernoulli label clusters and the coefficient predictor which connects the input features to the label clusters.Mixture of Bernoulli. The set of label clusters contains K mixture components. Each mixture component is a L-variate Bernoulli distribution z k = L l=1 Bernoulli(y l ; µ kl ) that captures a local 'stereotype' of the complete label distribution, where Bernoulli(y l ; µ kl ) = µ y l kl (1 -µ kl ) (1-y l ) and L is the total number of labels. The Bernoulli parameter µ kl has conjugate prior Beta(µ kl ; a kl , b kl ). Initially, the mixture of Bernoulli can be found through label-only learning. Using an EM algorithm, we can learn the initial components µ (0) K×L from the labeled samples S L . These components can model the set of labels p(y|µ) = K k=1 π k L l=1 Bernoulli(y l ; µ kl ), where π k ∈ (0, 1] is the normalized weight coefficient of the component k. However, since this is a label-only process, we are missing the connection to the input features x.Connecting to input features. For a conditional model such as CBM, the connection is through π k = p(z k |x) and µ = µ(x). The model is still trained in an EM manner so that the predictions ŷ can be made for each x. For a conjugate model such as GP-B 2 M, the connection is throughπ k = p(z k |GP k (x)), and the variational training process of the conjugate model impacting the posterior µ. As mentioned before, one issue with CBM is the disconnected prediction models of π and µ, while the inference process of GP-B 2 M is too expensive. In our proposed model, we also predict both π and μ from x using two decoder networks g π (•) and g µ (•). However, we maintain the connection by using a shared encoder network e(x), as shown in the structure in Figure 1 . Specifically, g π (e(x)) is an evidential model that predicts the distribution of π using the output evidence parameters, which function in the following way.Evidential weight coefficient predictor. We model the connection from input features to label clusters with fine-grained uncertainty information using the evidential regression model. The target for regression is to learn a combination of π k values that best reconstruct the final label predictions. To this end, we place a higher-order Normal Inverse Gamma (NIG) prior NIG(τ,σ 2 |p) = N (τ |γ, σ 2 ν )Γ -1 (σ 2 |α, β) over the regression model's Gaussian output N (π|τ, σ). The evidential model is trained to output the NIG parameters p = (γ, ν, α, β) similar to [1] . In this evidential model, the Gaussian likelihood interacts with the NIG prior, leading to a Student-t predictive distribution:EQUATIONHere, the evidential model predicts coefficients for input x as:π = E p(π|x,p) [π] = γ.The NIG parameters γ, ν, α, β are outputs from the predictor branch g π (e(x)). The evidential model, through its higher order NIG prior, can quantify the aleatoric (ALE) and epistemic (EP) uncertainty [1] asALE = E[σ 2 ] = β α-1 ,EP = Var[τ ] = β ν(α-1). In this evidential framework, due to the conjugacy of the NIG prior with the Gaussian likelihood, the posterior is also the NIG distribution. Moreover, in this model, after interacting with N i.i.d. data points (π 1 , ..., π N ), the posterior NIG parameters update as the observations increase [23] . Through the pseudo-count interpretation, the total evidence is quantified as E = v + 1 2 α + 1 β . We train the model to maximize The likelihood under the predictive Student-t distribution, which gives the NLL loss:L NLL = -log(p(π k |x, p) = 1 2 log π ν -α log Ω + α + 1 2 log (π k -γ) 2 ν + Ω + log Γ(α) Γ(α + 1 2 )where Ω = 2β(1 + ν). Additionally, we want the model's confidence/evidence for the prediction to be low when the prediction is incorrect by introducing a evidence-based regularizationL REG = (π k -γ) 2 • E.The regularization penalizes the highly confident wrong predictions, and ensures model's confidence is rightly placed. The overall loss isEQUATION) where λ controls the impact of the regularization.The integration of an evidential model in our multi-label classification approach presents several distinct advantages, particularly in addressing the inherent complexities of active learning (AL) environments. Firstly, evidential models provide a more nuanced and sophisticated mechanism for uncertainty quantification. This is crucial in AL settings, especially when dealing with sparse and rare labels, where traditional models often struggle. By effectively capturing and quantifying uncertainty, our approach leads to more informed and strategic decisions selecting data instances for labeling, optimizing the use of limited labeling resources. Furthermore, the evidential model facilitates a deeper understanding of the underlying label correlations, enabling the model to make more accurate predictions across a broad spectrum of labels, including the rare ones. This leads to a significant improvement in the overall classification performance, especially in scenarios where conventional methods might overlook subtle but crucial label dependencies. Additionally, the evidential approach inherently enhances the interpretability of the model's predictions, offering insights into the confidence and reliability of these predictions. This aspect is particularly valuable in knowledge-rich domains where understanding the model's decision-making process is as important as the predictions accuracy. Thus, incorporating an evidential model into our multi-label classification framework marks a substantial advancement, offering a robust, efficient, and insightful solution to the challenges posed by large and complex label spaces in active learning scenarios.Building upon the tools above, we propose a novel EMM model. Compared to existing methods, our novel contribution to model learning lies in how we connect label clusters to input features and how we learns the final labels in a joint manner. The integration enables evidential uncertainty analysis through both weight coefficient predictions and final label predictions. The evidential learning of π is already explained above. Here, we introduce the joint learning using actual labels y.Joint multi-label training with label clusters. Once we have the coefficient predictor branch, we can train the full model to make the final label predictions. We first freeze e(•) and g π k (•) to train g µ (•). Instead of directly letting the network predict µ from g µ (•), we make g µ (•) output proxy pseudo counts âkl and bkl to be combined with the initial (µ kl + w µ bkl (x). The new Bernoulli parameter for each instance is then computed by µ kl (x) = a kl (x)/(a kl (x) + b kl (x)). The label prediction is ŷl (x) = k π k (x)µ kl (x). The model is trained using a soft margin multi-label loss:EQUATIONWe then adopt the evidential pseudo-count style update of the label clusters, we would have a(1) kl = a (0) kl + K k=1 πk (x n )y nl and b (1) kl = b (0) kl + K k=1 πk (x n )(1 -y nl ).However, this update only makes the correction based on the predicted weights μk . If we keep updating in this way, the biases build up and the popular labels will be dominant in future components. Thus, we also include a weighted update based on the predicted proxy pseudo-counts, similar to when we make label predictions:a kl (x) = a (0) kl + w up âkl (x), b kl (x) = b (0)kl + w up bkl (x). This step ensures that our model mutually benefits from the coefficient predictor and the proxy pseudo-count predictor. The initial components are learned unsupervised and do not make up for the training of the predictors. By introducing the joint update, we connect the two predictors more closely.The joint training step of EMM addresses an important problem with the mixture model formulation, where the model prediction is restricted by the mixture components µ. Because the weight coefficients 0 ≤ π k ≤ 1, the label prediction for y l can only be as great as max k µ kl . If we only make updates to the mixture components using a(1) kl = a (0) kl + K k=1 πk (x n )y nl and b (1) kl = b (0) kl + K k=1 πk (x n )(1 - y nl ), the rare labels will still suffer from the label imbalance which will be reflected in max k µ kl = max k a kl a kl +b kl . By incorporating the joint training, we make the prediction based on a kl (x) = a (0)kl + w µ âkl (x), b kl (x) = b (0)kl + w µ bkl (x), allowing the model to better fit the labels using instancewise predictions âkl (x), bkl (x). The soft margin multi-label loss effectively brings the benefits of binary relevance machines into model training because it promotes positive predictions through the multi-label one-versus-rest formulation.Complete EMM learning process. Having established the weight coefficient training step and the joint multi-label training step, we integrate them in a complete learning process. To start with, we have the EM-learned initial clusters µ (0) K×L and the optimal Π (0) which reconstructs the labels when combined with µ (0) K×L . In one learning round, the model first goes through a pre-training stage of g π (e(x)) to fit the set of Π (0) optimized for the initial µ (0) K×L . Then, we move on to the joint training stage where we alternate between training the coefficient predictor and jointly training the full model to fit the labels y. Each coefficient predictor training step is the same as the pre-training stage, while the joint training step is described above. The model will continually improve the quality of label clusters to maximize the ability to recreate the labels in the joint training step, each time followed by the evidential learning of optimal π to keep up with the new label clusters. The complete training process is presented in Figure 1 . At this stage, we have combined the advantages of Bayesian mixture models, deep evidential models, and a bi-level multi-label problem formulation to obtain a powerful multi-label classification model. Next, we introduce how the evidential flavor can provide fine-grained uncertainty information and benefit active learning.In order to select the most informative samples given the small initial budget, we adopt an uncertaintyoriented selection strategy. From the proposed EMM model, we can obtain uncertainty information from three sources: weight coefficient branch, proxy pseudo count predictor, and the final label predictions. The first part of the uncertainty information is from the evidential model that predicts the weight coefficients. The evidential model naturally decomposes into the aleatoric uncertainty E[σ 2π k ] and the epistemic uncertainty EP. For AL purposes, we should target the samples that give us the most epistemic uncertainty, which can potentially improve the model's knowledge of the unknown. From the weight coefficient perspective, this criterion searches for the least confident samples based on our current model. Selecting these samples will help us quickly gain knowledge of the connection between input features and the weight coefficients, which is the determining factor for predictive performance. Thus, the first part of the selection function isEQUATIONThe second part of the uncertainty information comes from the proxy pseudo counts. We can compare the current components and the updated components when the proxy pseudo counts for x are added, and select the samples that would introduce more difference to the current model:A µ (x) = -CosineSimilarity(µ, µ ′ (x)) = µ•µ ′ (x) ∥µ∥•∥µ ′ (x)∥ .Because the label clusters play the most important role in recreating the label space, we should try to capture as much latent label correlation as possible. This requires sufficient exploration in the label space. Selection criterion A µ (x) does exactly this by searching for data samples that are potentially the most different from the currently captured label space. The last part of the uncertainty information is computed over the final label prediction. Since the full model is a mixture of Bernoulli, we can compute the expected covariance of the predicted label distribution. Here, we adopt the point estimate as in existing methods:EQUATIONNote that ideally we would like to use the conditional entropy H[y|x] to measure the uncertainty raised due to observing input x. However, the entropy of a mixture random variable is hard (or intractable) to compute because of the sum in the expression. But under mild conditions, we can show that y, as an average of the combination of weights and components, converge to a normal distribution so that the cov(y|x), which is easy to compute, can be used to quantify the uncertainties in the mixture random variables.Let ϕ 1 , ..ϕ K be the i.i.d samples that encapsulates the coefficient-component pairs where The selection score is A ŷ(x) = log |cov[ŷ|x]|. This criterion captures the expected information gain evaluated on the final label predictions when including the unlabeled samples. It aggregates the predictions from the weight coefficient predictor and the proxy pseudo-count predictor, and shapes the fine-grained uncertainty in a global view. By focusing on epistemic uncertainty from the evidential model for weight coefficients, differences in label clusters indicated by proxy pseudo counts, and the covariance in label predictions, we devise a comprehensive multi-source uncertainty-based selection score (MSU) A(x) = A π k (x) + λA µ (x) + ηA ŷ(x). Compared to a single uncertainty score, the integration of these uncertainty measures facilitates a targeted exploration of the data space and enables the identification of the most informative samples within a constrained budget.ϕ i = π i µ 1i ... π i µ Li with mean m = m 1 ... m l = E[π i µ 1i ] ... E[π i µ Li ] 
experiments: 
 We conduct experiments on both synthetic and real data to demonstrate the effectiveness of EMM. We use the synthetic data experiment to show how label clusters of the mixture model can capture various label compositions and correlations. We then verify the AUC performance of EMM on real-world multi-label datasets, along with rare label analysis and ablation studies. To demonstrate the effectiveness of the EMM model and the MSU sampling strategy, we design a synthetic dataset that can verify each of the proposed functionalities. The synthetic dataset contains mostly geometric feature related labels, along with a few carefully designed labels. The input features of the data points consist of 16 clusters distributed as m-dimensional Gaussian's. In other words, each point is sampled from one Gaussian cluster. The clusters have different means and universal variances such that they have slight overlaps. The geometric feature related labels indicate which Gaussian the point is sampled from. We denote all the geometric-related labels as L geo 's (all 16 of them) for simplicity. While L geo 's mainly indicate the location of the data samples in feature space and do not carry correlations in themselves, we construct the labels of interest using certain groups of them to ensure complex label correlations that are also feature-rooted. These labels of interest include a rare label L1, which has a frequency as low as 5% of a regular label; a couple of highly-correlated labels L2 and L3 which share similar features; and L4, which only depends on L2 and L3 and is dependent on other labels instead of input features. Specifically, L1 is assigned to data samples randomly with a low probability of 5%. L2 and L3 are randomly sampled from the same quarter of the feature space. Then, L4 is generated following the rule L4= L2 ∪ ¬L3. Besides these labels, we also append a set of non-geometry information-based labels L non-geo , which prevent the problem from being purely geometry-based. The label composition is shown in Figure 2 .Capturing label correlations. In one example of our experiments, we train the EMM with 6 label clusters. Among these clusters, one has a particularly high weight µ {1,L2} for L2. Simultaneously, the weight µ {1,L3} for L3 is low while the weight µ {1,L4} for L4 is high. Such behavior will ensure that the co-appearing L2 and L4 are captured during the prediction process. In the 6-cluster setting, , "updated" is an updated µ 1,L1 (x 1 ) where π 1 (x 1 ) = 0.83 meaning µ 1,. is dominant, and "irrelevant" is an updated µ 1,L1 (x 2 ) where π 1 (x 2 ) = 0.18 meaning this cluster µ 1,.does not contribute much to the prediction of x 2 .We further test various numbers of components and quantify the label correlations described in those label clusters. The results show that the average CosineSimilarity(µ {.,L2} , µ {.,L4} ) is 0.91 meaning that the positive correlation between L2 and L4 is always captured. Meanwhile, the average sim(µ {.,L3} , µ {.,L4} ) is as low as 0.12 showing the lack of correlation between L3 and L4. More specifically, when neither µ {.,L2} or µ {.,L3} is insignificant, R(µ {.,L2} , µ {.,L4} ) drops to 0.31, indicating that the fine relationship of "if and only if" is wellcaptured by the mixture model.Prediction enhancement by proxy pseudo-count combination. Although the mixture model is great at capturing the label correlations with label clusters, it is not always good at predicting rare labels. For example, in the 6-cluster setting, the largest weight for L1 is µ {2,L1} = 0.016 because of the extremely imbalanced label distribution. In this case, even if the model predicts solely µ {2,.} for a sample x (π 2 = 1, π k = 0, k ̸ = 2), the prediction of L1 is 0.016. This small value creates difficulty in converting the predicted score to a positive label prediction.Rare label and correlation discovery by uncertainty quantification. As for actively selecting data samples, we study the correlations between A π k (x), A ŷ(x), and the true labels of x. We show a set of statistics in Table 1 to analyze these behaviors. For A π k (x), we compare the sampling score with the estimated unknown information of the corresponding pool samples. The unknown information is evaluated from both the feature and label perspective, using the feature similarity and the label cardinality. The correlation between the feature similarity and the uncertainty score is -0.73, and the correlation between the label similarity and the uncertainty score is -0.41. From the label similarity we can also conclude the negative correlation between the similarity and uncertainty. For A ŷ(x), we specifically focus on the rare labels. On average, the samples containing less than 3 labels have an uncertainty score 28.8% higher than the other samples and the samples containing rare labels have an uncertainty score 65.8% higher than the regular samples.Datasets and experiment settings. We conduct AL experiments on representative real-world datasets including Delicious, Bibtex, Corel5k, Enron, and NUS-WIDE, covering multiple application domains [33, 6] . The number of labels ranges from 53 to 156, most of which are relatively rare in the entire dataset. We summarize the datasets and data preprocessing in Appendix D.Performance comparison. We compare the AL performance with competitive AL baselines:• GP-B 2 M uses a Bayesian mixture model and conducts active sampling using the combined predicted variance of multi-output GP and the label clusters [30] . • MMC is model-adaptive (implemented with label ranking model) and involves a predictive process for the number of labels. It samples instances based on the expected loss [38] . • Adaptive Adaptive uses SVM margin and label cardinality inconsistency for data sampling. [18] . For AL comparison, we use the area under the ROC curve as the main criterion. We start with 2% initially labeled samples for datasets Delicious, Bibtex, Corel5K, and 0.03% for NUS-WIDE. The initial labeled set contains a minimum of one positive instance per label to ensure that binary solutions can be trained. For EMM and methods that can perform batch active learning, we sample 5 rounds with 100 samples selected in each round. For single-batch baseline methods, we sample 500 rounds to obtain the same number of labels in the end. The base performance of classification models varies as some baseline methods use binary-SVMs (Adaptive, CVIRS), some use strategy-specific models such as the label-ranking model (MMC) and the GP-B 2 M model.We also include a configuration EMM-entropy that uses the proposed EMM model and a simple entropy-based selection strategy as an additional baseline, showcasing the performance gain from the proposed sampling (MSU selection) on its own. From Figure 4 , we can see that the EMM model makes better predictions using the same amount of initial labels compared to the SVM model, which explains the advantage at the starting point. Although the label ranking model or the GP-B 2 M model may also have good performance at the starting point, they are restricted by specific sampling methods. To separately verify the advantage brought by the uncertainty quantification, we show that our MSU selection always has an advantage in selecting better AL samples compared to a simple uncertainty-based selection (EMM-Entropy).For a more fine-grained analysis of the model performance, we also compute the average precision improvement [30] . This shows how the rare-label predictions have improved using the EMM model compared to a fully Bayesian mixture model where the label clusters are completely global.EQUATIONIn Figure 5 , we show the API metric for the 50 rarest labels on each dataset. The improved API on a label is shown by a blue bar above the AP I = 0 axis, while the worse API performances are shown We can see that EMM has a significant advantage on rarer labels.Ablation study. We conduct an ablation study on model components (weight coefficient and proxy pseudo-count predictors) and the AL sampling method (balancing parameters λ and η). Since the proposed EMM model combines the evidential weight coefficient learning and the proxy pseudocounts learning, we compare the complete model with two weakened configurations:• EMM -rr reduces the weight coefficient leaner to a simple ridge regression model, and only combines the prediction with fixed Bernoulli mixtures as the label clusters. • EMM -f ixed uses the evidential learning of weight coefficients with only the fixed Bernoulli mixtures as the label clusters.From Figure 6 , we can see that the evidential regression model predicts the weight coefficients better than a simple regression model such as Ridge Regression, which shows the effectiveness of the first branch of EMM (e() and g π ()). We can also see that without the proxy pseudo-count updates, the performance is not as good, which shows the effectiveness of the second branch of EMM (g µ ()) and the joint training of the entire model.From Figure 4 , we can already see that the proposed evidential uncertainty-based sampling outperforms a simple metric such as Entropy. From Figure 7 , we can see that our MSU sampling strategy effectively benefited from the multi-source uncertainty information compared to the single-source uncertainty (λ = η = 0). However, the epistemic uncertainty from the evidential model is still the most important source of uncertainty as the sampling performance decreases when we increase the balancing parameters for A µ (x) and A ŷ(x) too much. 
conclusion: 
 In this work, we introduced a novel Evidential Mixture Machines (EMM) model, which integrates deep evidential learning with a multi-label classification framework of AL. This approach effectively tackle a large and sparse label space, particularly addressing the challenges posed by sparse and rare labels. Our model's sophisticated uncertainty quantification and improved prediction accuracy set it apart from traditional Binary Relevance Models and other existing methodologies. The effectiveness of the EMM model is demonstrated through rigorous evaluations on both synthetic and real-world datasets, showcasing its superiority in diverse labeling scenarios. This advancement not only contributes to the development of more efficient MLC methods but also paves the way for future research in this domain. The potential for scaling this approach to larger datasets and adapting it to various domains offers exciting opportunities for further exploration and refinement. models, the evidential deep learning model takes a longer time to train. However, compared to Bayesian models or label ranking models, the inference time is significantly shorter.For our main results, we pre-train the label clusters using an E-M algorithm and obtain initial optimal weights for the labeled training set using linear programming optimization. The evidential model is trained for around 5000 epochs to fit the weight coefficients π, and the joint training step is trained iteratively for 100 epochs in each round. 
 
 Appendix: Here we show additional AL comparison with two other baselines:• AUDI uses a label ranking mechanism, where a dummy label is used to separate the positive and negative labels. Its sampling function is based on a modified cardinality inconsistency measure [14] . • CS uses a compressed sensing mechanism combined with GP predictions [31] . From Figure 8 we can see that both baselines do not perform well at the low budget, while EMM still outperforms them. The implementation of AUDI and CS on BibTex and NUS-WIDE are computationally too demanding and we would expect similar behavior given the labeling budget.Most recent multi-label works also focus on AUC optimization. So, we also add a discussion on the macro-AUC. For example, [36] achieves the state-of-the-art macro-AUC results on bibTex, Corel 5k and Delicious. In our work, due to the poor macro-AUC performance of some baselines, we mainly present micro-AUC results. However, here we provide the macro-AUC results using 80% training data (except for NUS-WIDE where we use 13,000 training samples because full-training is too expensive) compared to the methods in [36] . We can see that the macro-AUC is close on common datasets. Additionally, the AUC optimization method is orthogonal to our evidential model. We could incorporate AUC-based loss regularization into our joint-label training step. Due to the complexity, we leave these studies to future work. Instead, we focus on the AL improvements in this paper. In this set of experiments, we show results using more configurations of λ and η. As explained in the main paper, these balancing parameters work well with a moderately small value. With λ and η both around 0.1 to 0.01, we are able to obtain stable AL results. However, if the values are set too large, the performance may degrade. When we set λ = 0, η = 1 or λ = 1, η = 0, we get the combination of A π k and a single source of label uncertainty A µ or A ŷ . As we can see, the combination of all three works the best. In Figure 11 , we show the comparison between different K values. As we can see, extremely low number of clusters is not sufficient for achieving good model performance, while a large number is much more costly and also suffers from overfitting. The latter problem might harm the AL sampling more as we see the K = 10 case performs even worse than K = 3.We can also evaluate the quality of uncertainty estimation using true labels of the pool samples after AL experiments. Here, we present an example of the rare-label uncertainty being captured by both the cluster difference prediction and the covariance of the predicted labels. From Table 6 (results obtained on BibTex), we can see that similar to the synthetic data case, the uncertainty metric captures the rare labels well, although the difference is smaller because there are more labels in total.While EMM offers significant advancements in multi-label active learning, there are some limitations to consider. First, the model's complexity and computational requirements may pose challenges when applied to extremely large datasets or in real-time applications. Second, we have not integrated the Bayesian selection for K in EMM, making the choice of K crucial. Our future works include a more adaptive model structure that allows a dynamic K as the number of label clusters, along with a more lightweight training and adaptation process of the model. We also look to integrate adaptive testing and validation approaches for the AL process.EMM has the potential to significantly impact various fields that rely on multi-label classification. In domains such as healthcare, bioinformatics, and environmental science, where data labeling is often expensive and time-consuming, our model can optimize the use of limited labeling resources, leading to more efficient and accurate predictions. This can accelerate research and development in these fields by enabling the discovery of new patterns and correlations. Moreover, the model's sophisticated uncertainty quantification can improve decision-making processes in critical applications, where understanding the confidence and reliability of predictions is crucial. However, the implications of label clusters need to be carefully considered in applications.https://github.com/ritmininglab/EMM.git 


Decision: Reviewers expressed mixed but generally positive views on the paper. While the novel approach to multi-label active learning was appreciated, particularly the use of Bernoulli mixtures and evidential learning, there were concerns about robustness to outliers and the clarity of the presentation. Some reviewers noted that while the model performs well, the paper could benefit from more intuitive explanations and improved experimental analysis.

As a meta-reviewer, I noticed that the paper does not explain in sufficient detail how the synthetic data were generated. There are explanations in Section 4.1 and Supplementary D.2, but neither is sufficiently detailed, making replication difficult based on the text alone. There is a link for anonymously hosted source code, but the link had expired by the time I looked at it. Despite these issues, the authors’ rebuttal addressed most concerns, and with some refinements, the paper is suitable for acceptance.

Reviews: review: paper_summary: paper_summary: This paper introduces the Evidential Mixture Machines (EMM) model, which addresses the multi-label active learning tasks, particularly in rare-class scenarios. EMM uses a mixture of Bernoulli distributions to capture label correlations and uses evidential learning to quantify uncertainties for more informed active sample selection. The model dynamically updates label clusters using proxy pseudo counts.
strength_weakness: strength_weakness: I have 2 concerns:
1. Is the model robust to outliers? In real-life data scenarios, the existence of outliers is normal. Outliers can cause the proposed model to learn incorrect or spurious correlations between labels, (label noise issues also have a similar effect). If outliers have unusual combinations of labels, they can distort the representation of the label clusters.

2. Compared with the GP-B2M paper's experiment, there are some inconsistencies, for example, on Bibtex and Corel5K datasets, GP-B2M performs significantly better than CVIRS in GP-B2M paper, but performs similarly on this manuscript, can the author discuss more about it?
questions: questions: Please see the weakness part.
limitations: limitations: The author has already addressed potential biases and boarder impact in their appendix.
paper_summary: paper_summary: This paper focused on multi-label classification problems in active learning settings, where the label relationship, especially for rare labels, is hard to learn. The authors proposed an Evidential Mixture Machine (EMM) that combines the mixture of Bernoulli with a deep evidential model, which allows joint learning of the weight coefficients pseudo label counts to model the proxy label distribution for each instance. This method allows quick inference time and twofold uncertainty measurement from the evidential posterior parameters and the predicted variability in final labels. Extensive experiments on real-world and synthesised datasets certified the stated advantages of the proposed methods.
strength_weakness: strength_weakness: Few typos and confusion:
- line 103 "reuarization"
- lines 339-341 stated that EMM model outperforms the others in the initial round, which only showed in Figure 3 (c) Bibtex. Also, I can't see "the GP-B2M model may also have good performance at the starting point" in lines 341-342
questions: questions: - Figure 7 in Appendix D.2 is ambiguous about the label relationships. Why is the Geo-based labels pointing to L1 rather than the Non-geo-based labels? Also, the arrows from L3 to L4 doesn't show the not relation.
- From the results in Figure 3, the proposed active learning strategy doesn't have noticeable advantage than the EMM-entropy in (d) and (a). Does it indicate that for some dataset, the proposed active learning strategy doesn't add much information about uncertainty compared to entropy?
limitations: limitations: The authors have adequately addressed the limitations and potential societal impact of their work.
paper_summary: paper_summary: This manuscript investigates multi-label active learning, a critical issue in contemporary machine learning. To address this challenge, the authors introduce a novel evidential mixture machines (EMM) model, which provides an uncertainty-aware connection from input features to the predicted coefficients and components. The performance of the developed method is demonstrated through simulations on synthetic and real-world datasets.
strength_weakness: strength_weakness: However, there are several typographical errors in the manuscript, and the authors should thoroughly review and revise the text. The authors should improve the presentation. Additionally, more details about the model should be provided.
questions: questions: Specific comments and questions:
1.	How do the authors determine the parameters in the student-t distribution?
2.	In Section 4.1, the authors mention the large or small value of the weight. How does the outcome change with different weight values? Is the performance of the EMM model closely related to the weight? If so, how is the weight determined in different scenarios without prior information about the labels?
3.	In Figure 6, although EMM has a significant advantage on rarer labels, several high orange bars remain in Figure 6 (b). Could the authors explain this? Additionally, please add numerical labels to the y-axis below the API=0 axis.
4.	What is the computational complexity of the EMM method compared to state-of-the-art (SOTA) methods, and how does it scale with the size of the dataset?
5.	Please explain how this method can be employed in practical applications and what its strengths are compared to other SOTA methods (beyond accuracy).
limitations: limitations: The authors have adequately addressed the limitations of their work.
paper_summary: paper_summary: The paper introduces an active learning approach based on Evidential Mixture Machines (EMM) that compress the large, sparse label space of multi-label problems into a more manageable weight coefficient space. This approach combines mixtures of Bernoulli with a Deep Evidentiary models, and it leverages multiple sources of uncertainty (i.e., evidential uncertainty and the predicted label embedding covariance) to improve the active sample selection,
strength_weakness: strength_weakness: The paper could be improved on two main dimensions. First, the authors could make their case more compelling and intuitive by (i) introducing a motivating real-world example, and (ii) adding a section with an intuitive running examples that explains in layman terms the contribution of each of EMM's components, together with the synergy among them. Second, the experimental section should be expanded with a comprehensive analysis of the 4 domains in section 4.2 and their properties (e.g., the distribution of the multi-labels, so that one can grasp the challenges in each dataset), and by adding to each graph the best-known performance on that dataset in via fully supervised training. Without this last bit of information, it is impossible to judge the impact of this approach: the correct question is how much can AL reduce the need for labeled data while maintaining SOTA performance, rather than "how well can I do with X examples chosen via AL". For example, multiple algorithms reach 95+% accuracy on BIBTEX; how should we compare such results with 73% Micro AUC? To allow the reader/reviewer to estimate the significance of EMM, the authors should provide and discuss these numbers.
questions: questions: - please provide a reference for the claim in. lines 52-53
- line 61 - please explain why a single pass is sufficient
- line 321 - the details are not presented in. Appendix D
- lines 329-332 - instead on percentages from the original  data set, use absolute numbers; also justify the reason for choosing those particular numbers
limitations: limitations: N/A

Comments: title: response
Many thanks to the authors for the detailed answers. After considering both the other reviews and the rebuttals, I maintain the original rating.
comment: Thank you for reading our rebuttal and confirming that it has addressed most of your concerns. We appreciate that you maintain the positive rating on the paper!
Thanks for the authors' response. The authors have addressed most of my concerns. I would like to maintain my score.
We sincerely appreciate the time you dedicated to reviewing our paper. We have clarified the parameters and variables settings in our model.  We have also provided additional descriptions of the characteristics of our model. We hope that our response adequately addresses your inquiries. Should you have any further questions or require additional clarifications, please let us know.
Dear Reviewer dBCp, 

Thank you for reading our rebuttal! The outlier problem in the multi-label setting is a relatively less explored topic. One potential reason is that outliers are less well-defined in multi-label problems. Even in existing multi-label datasets, the imbalance issue among labels is already severe. Also, as we explained in our motivation section, there are many underlying correlations between these labels that might be difficult to define. Thus, the design of the outliers problem itself is challenging in multi-label classification. Coupled with active learning, where labels are scarce, the problem may become more prominent. As for how our proposed method can benefit the topic, we can extend the point in our previous response about using our fine-grained uncertainty analysis to improve outlier detection. In particular, for labeled data samples where the model exhibits a relatively low epistemic uncertainty (as defined in Eq. 4), which implies that the model has a decent knowledge of the sample, if the model prediction deviates significantly from the human-provided label, it could indicate a potential outlier. In this case, we may ask the annotator to re-check the label and provide a new label if a mistake is made. Such an *active relabeling* process can be integrated into an outer active learning loop to minimize the impact of outliers on the overall learning process. While this could be an interesting research direction, thoroughly exploring this topic is out of the scope of our current work. We will extend our future work section by adding a detailed discussion on this topic.
Thank you for reading our rebuttal and maintaining a positive rating. We will incorporate your feedback into the revised paper and are more than happy to answer any further questions that you may have.
The answer was not deep enough and I decided to keep the grade.
I appreciate the authors' response to my review and their efforts to address my concerns. After carefully reviewing the feedback from the other reviewers, I am inclined to maintain my original score but remain open to reconsideration during the discussion stage with the other reviewers and the Area Chair.
Thank you for reading our rebuttal and maintaining a positive rating. Again, we appreciate your constructive feedback and will incorporate the suggestions when revising our paper.
Many thanks to the authors for the detailed answers. After considering both the other reviews and the rebuttals, I maintain the original rating.