A Gaussian Process-Bayesian Bernoulli Mixture Model for Multi-Label Active Learning
Authors: Weishi Shi
Dayou Yu
Qi Yu
Venue: NeurIPS 2021 Poster (2021)
Keywords: multi-label active learning
Bayesian Bernoulli mixture
label correlation

Abstract: Multi-label classification (MLC) allows complex dependencies among labels, making it more suitable to model many real-world problems. However, data annotation for training MLC models becomes much more labor-intensive due to the correlated (hence non-exclusive) labels and a potential large and sparse label space.  We propose to conduct multi-label active learning (ML-AL) through a novel integrated Gaussian Process-Bayesian Bernoulli Mixture model (GP-B$^2$M) to accurately quantify a data sample's overall contribution to a correlated label space and choose the most informative samples for cost-effective annotation. In particular, the B$^2$M encodes label correlations using a Bayesian Bernoulli mixture of label clusters, where each mixture component corresponds to a global pattern of label correlations. To tackle highly sparse labels under AL, the B$^2$M is further integrated with a predictive GP to connect data features as an effective inductive bias and achieve a feature-component-label mapping. The GP predicts coefficients of mixture components that help to recover the final set of labels of a data sample. A novel auxiliary variable based variational inference algorithm is developed to tackle the non-conjugacy introduced along with the mapping process for efficient end-to-end posterior inference.  The model also outputs a predictive  distribution that provides both the label prediction and their correlations in the form of a label covariance matrix. A principled sampling function is designed accordingly to naturally capture both the feature uncertainty (through GP) and label covariance (through B$^2$M) for effective data sampling. Experiments on real-world multi-label datasets demonstrate the state-of-the-art AL performance of the proposed GP-B$^2$M model. 

A Gaussian Process-Bayesian Bernoulli Mixture Model for Multi-Label Active Learning
Weishi Shi'
Dayou Yu * Qi Yu
Golisano College of Computing and Information Sciences Rochester Institute of Technology {ws7586, dy2507, qi qii.yu)@rit.edi
Abstract
Multi-lahel classification (MLC) allows complex dependencies among labels, mak- ing it suitable model many real-world problems. However, data tation for training MLC models becomes much more labor-intensive due to the correlated (hence non-exclusive) labels and potentially large and sparse label space. We propose to conduct multi-label active learning (ML-AL) through a novel integrated Gaussian Process-Bayesiie Bernoulli Mixture model (GP-B*M) to accurately quantify a data sample's overall contribution to a correlated label space and choose the most informative samples for cost-effective annotation. In particular, the B encodes label correlations using a Bayesian Bernoulli mixture of label clusters, where each mixture component corresponds a global pattern of label correlations. To tackle highly sparse labels under AL., the B is further integrated with a predictive GP to connect data features as an effective inductive cewturr-coooonentt-ieee
mapping. The GP predicts coefficients of mixture components that help to recover the final set of labels of a data sample. A novel auxiliary variable based variational inference algorithm is developed to tackle the non-conjugacy introduced along with the mapping process for efficient end-to-end posterior inference. The model also outputs predictive distribution that provides both the label prediction and their correlations in the form of a label covariance matrix. A principled sampling function is designed accordingly naturally capture both the feature uncertainty (through GP) and label covariance (through B M2 for effective data sampling. Experiments real-world multi-labe label datasets demonstrate the state-of-the-an AL performance of the proposed model.
bias and achieve
Introduction
In multi-labe classification (MLC), each data instance may be associated with more than one label. Such rich representation of labels encode more complex data-label distributions that arise in many real-world problems 2 4. As a simple yet powerful tool, binary relevance machines (BRMs) transform MLC problem into multiple binary problems and train independent binary classifiers for each label [5). Such a transformation gives BRMs the flexibility to leverage state-of-art binary classifiers (e.g.. deep neural networks and SVMs). However, applying BRMs to a correlated and potentially large label space poses key challenges. First, many real-world multi-label datasets contain large number of labels. Training one predictor per label incurs a prohibitive cost. Second, despite an overall large label space, each data instance is usually assigned limited labels. Many labels are relatively rare and their appearances depend not only on the features but also the occurrence of other labels. Predicting these "complex" labels directly using independent binary classifiers is fundamentally difficult due the limited positive data instances and weaker direct dependency on the features. Correlations among labels provide important auxiliary information to enhance multi-label
Equaa contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

Label
Descrition of conclations
Caometee losation daca, wach asigned
mesture two Gaussan's
C1-C2
CL: lobel corcinaitty 53, C2: lobel cardinllty 23
01:04
01: G1/02; oo: G7n6%; ca: G1n64; oi: G1nsss
E1-64
Lebeb related the wodanive correlation Jowfined in (b))
96
G6
PIEZ E51
E1: U
Com 2
3
Component
05
E11E2 04A.A/11111GG6004111111110000111
(a)
(b)
(c)
Figure 1: (a) Labels with geometric correlation (G1-G8); (b) Labels with cardinality (CI,C2), overlapping (01-04), and exclusive (and hierarchical) dependencies (E1-E4); (c) Definition of label correlation and learned mixture components.
prediction 6 8 However, these models heavily rely on the training data that exhibit these important label correlations. Considering the high cost in annotating a multi-label dataset, it is critical to choose the most informative data samples for cost-effective data annotation.
In this paper, we propose a novel Gaussian Process- Bayesian Bernoulli Mixture (GP-B²M) model to achieve cost-effective sampling for multi-label active learning (ML-AL). ML-AL, since labels are not mutually exclusive as in the single label setting, all the labels should be considered collectively when designing an active sampling function so that data sample's 's overall contribution to the entire label space can be accurately measured. However, since only limited training data instances are available for an ML-AL model, how to accurately model label correlations and hence quantify a data sample's 's overall informativeness using very sparse labels under AL. poses grand challenge. Existing efforts that explicitly model label correlations usually focus on limited types of correlations such as pairwise (3, conditional [10 [T], full correlation in a subset of labels 13 Consequently, those methods may miss some important label correlations. Label correlations can also be captured through latent embedding 7 g. While these methods can scale to large label space, they usually require a decent number of training labels to compute an accurate embedding, making them less suitable for ML-AL. Furthermore, the learned embedding has semantic meanings, which cannot be used to interpret the discovered label dependencies
The proposed GP-B²M model addresses the limitations of existing methods to fundamentally advance MI.AL.. In particular, the BCM encodes label correlations using a Bayesian Bernoulli mixture of label clusters. Since labels are highly sparse in ML-AL, a predictive GP is further integrated to learn a distribution of mixture coefficients that connect data features with the label clusters. Thus, the label clusters can be regarded as global pattern of label co-occurrences discovered from both the training labels and data features to address label sparsity. In this novel data features serve as inductive bias to learn accurate mixture components of labels, where data samples with similar features should be mapped to similar mixture components, which in turn lead to similar set of labels. Such an inductive bias allows the discovery of label relationships from limited labels with the support of feature relationships, which is essential for a sparse label space in ML-AL.
mapping,
Figure D shows three mixture components learned from synthetic data designed with complex label correlations, including geometric, cardinality, overlapping, and hierarchical (see Figure | (c) for definitions). For example, the hicrarchical labels (E'l E4) show some interesting but quite complicated correlations that may exist in many real-world data. If E1 represents a common disease and E2, E3 represent some less common ones that may co-occur with E1 (30% of the time), then E4 corresponds to a rather rare disease that only co-occurs with E3 but not E2. In Figure (c), both components land 2 show a high chance of E4. It is also clear that while both E1 and E3 are very likely to appear in these components, the chance of seeing E2 is much lower, reflecting its exclusive relationship with E3. Interestingly, while both components and 2 cover E4, they also show complementary information. In component 1, E4 mostly appears in the non-overlapping regions, which is indicated by smaller O1 and O4; whereas in component 2, it's more likely to
*Correlation sometimes refers to the degree to which pair of random variables are linearly related. However, in the broadest sense, both correlation and dependence refer any statistical relationship between two random variables, they are used exchangeably in the rest of the paper.

---

occur in overlapping regions O1 and O4. Finally, in component 3, it is less likely to observe E4. Consequently, the chance to see E2 is significantly higher.
The above example demonstrates that the learned mixture components accurately capture complex label correlations that are critical for active data sampling. They are also interpretable, which can help to unveil important relationships among labels. Our key contribution is threefold: (i) We propose a novel GP-Bayesian Bernoulli mixture model discover meaningful label correlations from limited labels by encoding the inductive bias from data features as Bayesian priors to learn from both labels and features while ensuring consistency. (ii) We introduce set of auxiliary latent variables to achieve fully conjugate end-to-end posterior inference. The number of mixture components is also dynamically adjusted during Bayesian inference so that the model complexity is automatically calibrated according to the size of training data, which is critical for AL.. (iii) The model outputs the predictive distribution that provides both the label prediction and their correlations in the form of a label covariance matrix. We design a novel active sampling function that integrates both feature uncertainty and label covariance to quantify data sample's 's overall contribution to a correlated label space. Extensive experiments on both synthetic and real-world multi-label data and comparison with competitive models demonstrate that the proposed GP-B achieves the state-of-the-arr active learning performance.
label mapping in the Bayesian model to support efficient
2 Related Work
Due to the wide adoption of BRMs for MLC, a number of AL models have been developed based upon BRMs. For example, the estimated reduction of BRM loss function has been used an uncertainty criterion for data sampling [1]] Uncertainty from individull SVMs in BRMs has also been integrated to compute sampling score, where label correlation is used to reduce the complexity of the active query rather than improve active sampling
[1]. Label inconsistency provides an alternative way to incorporate label correlation into BRMs [16] for data sampling. This has been further extended through label ranking 177. As discussed earlier, AL models built upon BRMs do not systematically capture label correlations, which may lead to inaccurate uncertainty measures for ML-AL.
A few existing models capture label correlations explicitly or through a latent embedding to support active multi-label sampling. For example, the approximate entropy of the predieted labels from a Bernoulli Mixtures model (CBM) is used for data sampling [18]. However, the quality of the uncertainty estimation relies on external multi-class classifier used to predict the component coefficients. Both model selection and parameter tuning for the classifier make it difficult for AL. One fundamental limitation is that CBM was originally designed for MLC (instead of AL) by predicting distinct set of label clusters for each data sample [6. Thus, different from the proposed GP-B²M model, no global label clusters are discovered to capture the label correlations, making unsuitable for multi-label AL. Gao et.al propose a correlation aware active sampling method for transfer learning task [19]. However, the method uses a kernel function to measure the label similarity between two data instances which cannot work well for large and sparse label space. Compressed sensing (CS) has been employed to learn a latent embedding of the label space to capture potential label correlations, which can then be used to design sampling function. However, since the latent space continuous, the labels are further assumed to be drawn from Gaussian distribution to ensure conjugacy, which violates the binary nature of the labels (20). Furthermore, CS requires an additional step to recover the predicted label from the latent code, which less efficient for AL. Active sampling is also sensitive to the recovery process and the recovery quality is usually low at the beginning of AL due to the lack of training data (21).
The proposed GP-B²M model systematically addresses the key limitations of existing methods through well integrated Bayesian framework that supports a fully conjugate feature-comppnent- label mapping and end-to-end posterior inference for cost-effective ML-AL.
3 GP-B*M for Multi-Label Active Learning
In this section, we first describe the proposed GP-B²M model by introducing key latent variables along with their conditional dependencies. We then present novel posterior inference process by augmenting the original model using a set of auxiliary variables to resolve the non-conjugate prior and likelihood. A principled sampling function is introduced in the end for cost-effective ML-AL.
3

---

Predict (Phase
Generate (Phase
L
K
N Data Samples
K Label clusters
Compress
Correlated Labels
(a)
Figure 2: (a) The GP-BºM framework for MI.AAI; (b) Graphical model of GP-B2M.
3.1 The Bayesian Bernoulli Mixture Model
Let X (X1, ..., X,] denote training set with .N data samples and Y 1y.,,... YN denote the labels, where yn {0, 1 The proposed GP-B²M model assumes there are K mixture components, O = {0 p- shared by all the data samples and the label vector of each sample is generated from
a mixture process. Mixture component k is result of L Bernoulli experiments governed by two parameters asg and byg 02 ~ II'= Beta( bat), where Oµ denotes the probability of assigning label to component k. The indicator variable End denotes whether component
assigned to sample Xn, where 2nk Cat( (T)), Tik h (*))) (A) is mapping function that outputs the probability of assigning X, to mixture component k, and f, = (fh ...,,... s/k))/ are GP latent functions for sample X,, with f( ( The final label vector associated with x,, is drawn from the mixture distribution given by p(y,(O) Σ t Figure () shows the graphical model of this generative process.
The GP-B-M model essentially adopts two-phase learning process, where these two phases are seamlessly integrated (see Figure a)). In phase I, it predicts the probabilistic assignment of the mixture components by learning a distribution of latent functions: F = [ where (()) (T(²) y....'' NN (( In phase II, these predicted mixture assignments are used refine the parameters of the beta distributions so that updated mixture components can best recover the true label vectors. One key innovation lies in using the latent indicator variables Z = (z,,, { to link the feature space with the label mixture components as way to encode the feature related inductive bias.
This is achieved through mapping function h. (hn ..... (K)) with / h ( (f.):
K
plznk = 1 1[f,,) = Tin = (1))
(R) ( [0, 11,
Σ
(0) n
= p(((k) = (P(( ma Ex)
(1)
where Σg = [x xmm)) is a covariance matrix and (C(--)) is a kernel function, and we set m with out losing generality. From the Bayesian perspective, this equivalent placing Dirac delta prior over TT Takk ru 0(TTnk 11()))), )), where the inductive bias is encoded by the prior distribution. By introducing (f,)) we essentially convert a multi-label problem into a multi-class problem as nr encodes the probability assigning x to each of the K components. Specifically, given the learned mixture components 0₂ ss for test data sample X., we predict the component assignments n, using the trained GP. The final labels are obtained as p(y-10) Σk T.
Posterior inference of latent variables in the two phases are jointly performed by maximizing the log marginal likelihood of the observed multiple labels for all training samples:
(np(Y = //ΣIII p (1))) (p(OR))(Zza) (tappp/eeeeeeeeeeeeoee (n)p(umllFa dFd
Directly maximizing this likelihood intractable due to the interplay the latent variables. So turn to optimizing the evidence lower bound (ELBO) of the log marginal: L(g) I (9( Z, In F. F.x. where @ Z, F) the variational distribution. However, key challenge that prevents us from using the standard mean field variational inference (MF-VI) is the term p(znk (f,), defined by the mapping function A1 in I As the most typical forms of h (c.g.. softmax) are non-conjugate with the prior distribution p( ( (f))),, which is Gaussian, the variational posterior (((()) cannot be derived analytically.
(6.2,F))

---

3.2 Auxiliary Variables based Variational Inference
We propose to resolve the non-conjugate mapping function in the complete data likelihood by introducing number of auxiliary latent variables such that the augmented complete data likelihood becomes conjugate. Auxiliary variables have been used in MCMC based inference, such as slice sampling [22] and Hamiltonian MCMC [33 with improved sampling efficiency. The basic idea of auxiliary variables based variational inference (AV-VI) is to apply the following transformation: where p(z) is target function that is difficult to compute (c.g., non- conjugate) during VI. If the conditional likelihood p(rly) still non-conjugate, this process will continue until a conjugate conditional is achieved.
p(z) = Jy (a
key identity that we leverage to achieve conditional likelihood conjugate Gaussian prior p(P(k) is convert logistic sigmoid function scale mixture of Gaussian's 24 where the mixture is defined by a Polya-Gammn Gamma distribution p(w) = PG(w)b, 0),
(el)a (1 ef) 2
/0- p(w)dw
(3)
where b 2 0, = a However, a sigmoid function only suitable for binary classification, making it infeasible for 2 mapping function that outputs the assignments for K > components. Thus, we adopt the logistic-softmax function (25) as our mapping function:
(1a+t)
(k) = p(=nk = lff,,) = Σ (FP)
(4)
To handle the summation in we introduce random variables AEN and use identity Jo e e-Aa so that
1
K
p(2nk = 1 (m, = ((f())) II (
(5)
where p(A,) x 1 (0..o),, Vn E NJ. By leveraging the moment generation function of the Poisson distribution Po()), we introduce random variables Y {VI.... UN}. where en (Un1, Unk)
to convert the exponential term in (5). which leads to
p(znk 1ffn vn) vn)= oo(((( ((((-)
j=1
where Unk ~ Po(unt/ An). Finally, using (3) and introducing the Pólya-Gamma random variables [wi,..., WN), where wn = (Wal, (W.n,,.,,W, W,., Wnk) leads
K
(Znk Unt)
(f(h²)²
p(ank = 1/fn, Unky wrk) = II
2 (emx +uma) exp
wik
2
2
where W (G (ink 0). Figure shows the graphical model with the auxiliary variables (x,''s are omitted from the graph to keep the notation uncluttered).
We proceed by defining variational distribution with auxiliary variables:
q(O) Z, F, T, = = ()q(A))(TT
(8)
The optimal variational distribution can be obtained by computing the moments of component variational distributions using some important properties of the main and auxiliary variables and iterating until convergence. The optimal variational distributions of the main latent variables are summarized in the following theorem.
Theorem 1. With the auxiliary random variables and the transformed complete conditional likelihood given in o). the optimal components of the variational distribution as specified by g are given by
Component assignments q(Z) = -I,, (z
((znk) Cat(znk (nk); Tnk exp
(aa daa + bui))]
Mak
where 2-(-) is the digamma function and mirk is te-hh element of mean of ((P()) defined in q(F).
5

---

Mni
Fack
whik
Cir
mm
812
An
but
Σ
NV
Figure 3: Graphical model with auxiliary variables
Bernoulli mixture components (()) II 11: ((xx)
@(0k)) Beta Opi
(ākl b 0 = akt
Onk:Unl: bkd = bui +
Pnk (1 yni) (10)
GP latent functions ((F) I1 (((()))
q(P(k)) = (fx in. Σx); m = 2
Sn( Egve [xx)); Ex (Fig²) diag( Fagws wz) (wx)))
(11)
where Ua (Uiky..., ..., UNK)" we (WiR,... ..., wxa) )2, and T(wE, uk) T(wx lox (7(R)) is the optimal variational distribution for these auxiliary variables.
The specific forms of the auxiliary variational distributions {()) and ((Y,)) are provided in Ap- pendix B as part of the detailed proof of the theorem.
Model interpretation. The optimal variational distributions are fairly intuitive. Interpreting these distributions can reveal some key insights on how the proposed GP-B3M model leverages the data features effective inductive bias to discover semantically coherent components from sparse label space. First, from (9. the component assignment of data sample (x,,yn) is determined by two terms: the first term indicates how all its labels yn correlated with the component and the second term reflects how likely to categorize the features X,, into to the component. Second, from (IO), since the component assignments are further utilized to compute the Bernoulli mixture components, the optimal components naturally aggregate both label and feature information to ensure semantic consistency result of using data features the inductive bias. Last, from (11, the GP latent function value on a component increases with a positive component assignment and decreases with 'negative" assignment, captured by the Poisson auxiliary variables Uk.
Time complexity. According to posterior inference of GP-B²M has the computational com- plexity of O(N*K) which is identical to training K GPs. Since each component be updated independently. we can parallelize the computation to further reduce the complexity to O We can further leverage sparse kernel machines (e.g.. Sparse GP) reduce the complexity if N is large.
3.3 Multi-Label Active Sampling
Being a Bayesian model, GP-B-M outputs the predictive distribution that provides both the label prediction and a label covariance matrix. As the covariance matrix captures both the uncertainty of individual labels and correlation of each pair of labels, it provides essential information design principled measure to quantify data sample's 's overall contribution to correlated label space.
For each testing sample x., the predictive mean can be computed using the variational distributions:
Biy-|x.]= ~ Ior........)/x./Eee..
(12)
Yfx,X,Y,x.)) F(t)) [0)) 0h. (Øgt bua
where Tak = p(=.k 1ff,) ).)()))))) is defined as logistic-softmax function given in Theorem 1 shows that q( F), which approximates the true posterior p( (F|X, Y), a Gaussian. Hence, is also a Gaussian. However, the logistic-sfftmax transformation makes predictive mean intractable
x.)

---

(a)
(b)
(c)
(d)
Figure 4: (a) Distribution of E4 samples; (b) Prediction by BRMs; (c) Mixture component assign- ments by M (d) Prediction by M
to compute. We propose to conduct Monte Carlo (MC) integration by drawing samples from p(f,|X, XX, Y, x.). perform logistic-softmal transformation, and then average.
The GP-B²M M model also allows us to compute the predicted label covariance,
cov Nov[y.|x.] = Σ E [B[AA] E [[,,]"" Ely- x. xx]]E(y.|x.
(13)
where A₂ = (E@g]([ [[Oa])) The predicted label covariance captures both individual label uncertainty (diagonal entries of the matrix) and label correlations (off- (off-diagonal entries), which is instrumental to quantify the total uncertainty of a test sample with respect to its predicted labels. Since directly computing the entropy of mixture distribution is challenging, use the log determinant of covariance matrix: In|cov[y [y, xx.].. as a proxy for uncertainty evaluation. Intuitively, this is equivalent to approximating p(y. (xy) using multivariate Gaussian, whose entropy is the log determinant of its covariance matrix plus a constant.
instead choose to
The label covariance is computed using point estimate of (T=1,..., x (one T. for each class) to quantify the total uncertainty on the label side. As Bayesian model, the proposed GP-B3M allows us quantify the variation of each πą using its predictive variance. Through MC integration as described above, we compute the predictive variance Var[T=*] of sample x, for each of the K easily show that the model would assign
class. According to the property of the GP posterior, a low variance to data samples near to the training data and a high variance to faraway samples. As a result, the predictive variance effectively captures the feature uncertainty that complements the label covariance. It allows the proposed sampling function to differentiate data samples based on their distinct contributions to model training and sample them accordingly. Our final sampling function is given by: x. = arg maxx, label covariance and predictive variance of data features. It can be dynamically updated give a higher weight in the early stage of AL to the feature variance term for better exploration of the data space and then shift the focus to the label covariance term for effective fine-tuning of decision boundaries with correct shape obtained through effective exploration.
|x.yy|xx]|++>)) Var War[T=+]/K KK, where is used to balance between
4 Experiments
We conduct extensive experiments on both synthetic and real-worke multi- data to demonstrate: (1) important properties of GP-B-M to capture complex label correlations and how they contribute to predict complex labels, (2) state-of-the-art ML-AL performance by comparing with existing competi- tive models, (3) impact of key model parameters through an ablation study, and (4) effectiveness of active sampling by examining sampled data instances.
4.1 Synthetic Data
We design synthetic dataset with 18 labels that exhibit 4 distinct types of dependencies defined in Figure T (c). In the introduction, we show that three discovered mixture components precisely capture some rather complex label dependencies (e.g., hierarchical and exclusive) while being highly interpretable. For this dataset, the model discovers 10 components in total and we show some other components in Appendix DD allonn with their interpretations. We further demonstrate how the discovered components contribute to the prediction of more complex and less frequent labels. We use E4 as example, which is located deep in the hierarchy and appears much less than other labels.
Figure 4 (a) shows the distribution of data samples whose labels contain E4. can be seen that these samples are distributed across the entire El region (roughly corresponds to the shaded area in purple).
7

---

Note that, in addition to E1, E4 also depends on the exclusive relationship: E3 AND NOT E2 (E2, E3 are not shown in the figure to keep the distribution of E4 clear). Figure (b) shows the prediction result from BRMs, which has very high false positive and negative rates. The poor performance is also reflected by a low ROC-AUC (area under the receiver operating characteristic curve) score at 0.58 (slightly better than random guessing). It appears that BRMs only predict correctly samples in an area where E4 samples are relatively dense while missing most others. This is because BRMs try to directly learn the feature-label mapping (by training independent binary predictors), which is usually weak for complex and less frequent labels, like E4.
Different from BRMs, the proposed GP-B3M learns mixture components that correctly capture the label correlations and the final labels can be recovered by combining the mixture components through their predicted coefficients. As discussed earlier, E4 has a high chance to appear in either
Figure 5: AUC on different types of labels
Label
G1-G8
C1,C2
O1-04
E1-E4
BRMs
0.79
0.68
0.72
0.58
GP-B"M
0.83
0.70
0.86
0.82
Component 1 or 2. Figure shows the predicted component assignments and the top component is highlighted. As can be seen, most E4 samples are assigned Component I or 2 as their top component. Also, samples assigned to Component 1 (shown in blue) are mostly distributed in the non-overlapping geometric regions and those assigned to Component (shown in yellow) are mostly in overlapping regions. By leveraging these components, GP-B²M achieves much better prediction results as shown F Table Summmaiies the AUC scores from BRMs and GP-B3M M on different types of labels. While both models achieve similar performance on some common labels (e.g., G), GP-B3M significantly outperforms BRMs for more complex labels, where it is essential to capture important label correlations.
4.2 Real Data
Datasets and experiment settings. We choose five representative real-world multi-label datasets, in- cluding Delicious, Enron, Bibtex, Corel5K, and NUS-WIDE, from different application domains [26] All datasets have a relatively large label space and high label sparsity (2 (2-6% 6%). Table Appendix D summarizes key properties of the pre-processed datasets. We randomly shuffle each dataset and partition them into three parts: training, testing, and candidate pool. We keep a minimum of one positive instance per label in the initial training partition as required by BRMs based AL models. To make sure each label is well represented in each partition, we remove extremely rare labels with label frequency less than 20. Since the remaining labels are still highly imbalanced, we use the ROC-AUC score to evaluate the model performance. All the baseline models share the same copy of the initial training set to make fair comparison. Active learning stops after each model selects 500 samples.
Performance comparison. We include five competitive baselines for AL performance comparison:
MMC samples instances that introduce the greatest change of the expected loss. During label
prediction, it uses logistic regression to predict the number of labels for new instance 14.
Adaptive considers both the separation margin of an SVM and the label cardinality inconsistency and combines these two parts for data sampling [16).
AUDI uses a label ranking mechanism, where dummy label is used to separate the positive and negative labels. Its sampling function is based on a modified cardinality inconsistency measure (27).
CVIRS combines the ranking on the magnitude of the difference margin in predictions and the
label vector inconsistency for active sampling [17].
CS-GP conducts active sampling in a compressed label space using a multi-output GP 21
Freure&rreports the AUC scores of all the models. For each curve, we present the average result along with the error bar from 3 trials of randomly initialized AL experiments. The proposed B model achieves better AL performance consistently on all the datasets. For multiple datasets, GP-B2M establishes a clear advantage in the early to middle stages of AL. While a few baselines eventually converge to a similar AUC score, they usually take more iterations (by consuming more labels) to reach comparable performance as GP-B*M. In addition, by comparing with random sampling with the proposed sampling function, we clearly demonstrate that the superior AL performance attributes to both the Bayesian mixture model and the effective active sampling. Note that the AUDI model runs much slower for Bibtex and Corel5K when both the number of features and candidate pool size become very large so we omit the results.
Analysis on label-wise improvement. To further justify why the proposed GP-B²M outperforms the basclines, we offer fine-grained analysis on the label-wise performance, which helps to achieve
8

---

Detcious Dataget
Enron Dataset
B Blox Dacaset
CorafSi Ducaset
639
6:3
Inac
611
CA.a
66%
C71
CSR
Number Selected Samples
Number Selected Somp es
Number Selected Samples
Number of Sellccted Samplcs
(a)
NUS-ANDEE Datase
Enron Dataget
Ponon Darewet
O0B
065
645
6.1
6.1g
kumoor of Solocttdd Samples
Number Selecced Samples
6%%
Kumber of Selected Samples
ze 10 369
(e)
(f)
Figure 6: AL performance comparison (a): Ablation study on one example dataset (Enron): (f) shows the impact of 1) and (g) shows the impact of effective components.
Delicious Dataset
NUS-WIDE Dataset
Percentage of Improvement
Percentage of AP improvement
7:0
800
500
4c0
1:c
300
1c0
30423020 166181129166
75
476
366
222
117 160
Label Frequency
Label Frequency
(a)
(b)
Figure 7: The label-wise average precision improvement over BRMs models.
a deeper insight on the performance gain. In fact, we reach a similar conclusion as the synthetic data experiments: while all models perform reasonably well on the most frequent labels, GP-B²M clearly outperforms other baselines on less frequent labels, where considering the correlations with other labels can help to improve their predictions. Thus, our fine-grained analysis, we first exclude the few most frequent labels and compute the Average Precision Improvement (API) of the i-th label,
AP
AP(( BRMs 100%
APT
B
(14)
AP(
BRMs
where (P(1) B and APl() denote the average precision of label i provided by GP-B*M and BRM, BRMs, respectively. Figure demonstrates digunredeemosstrttss the label-wise API on the NUS- LUD- and Delicious datasets as two illustrative examples. We observe that GP-B²M performs significantly better than BRMs on labels with a moderate or low frequency. This is because those labels usually do not have sufficient positive instances for BRMs learn independently. Furthermore, they may have complex correlation with other labels. GP-B"M effectively leverages label correlations to make better predictions on these labels. This further confirms the overall good performance of GP-B*M.
Ablation study. We further investigate the impact of two tunable parameters of the model: (1) n- which balances label covariance and feature uncertainty for data sampling and (2) p. which controls effective number of mixture components. Limited by space, we use the Enron dataset as an example and report other results in Appendix D Figure 6 (f) compares the performance under different y values. In early iterations, the label covariance guided sampling (n is small) slightly falls behind the feature uncertainty guided sampling (1) is large) the latter more useful to explore

---

(a) ('clouds', ... 'grass', "hills'....... (b) l'olonds',
'landscape', (C) ('clouds', 'farm,, 'landscapp', (d)
['house',
'landscape"s :
pllnnts, , 'skyy,, 'valley']
pplants', 'sky'l
'nature', pplants', 'sky". , "sun") 'plants', 'skyy"
(e) l'architecture', "buildings', (1) "clonds', 'plants', isky']
I'aamm. "Geld", "grass" (g) I'clouds', "farm". 'Blowers', (h) I'art', "clouds', "grass", "na- 'plants']
grass" 'plants", 'skyy]
ture', 'plants", 'skyy, 'tree']
Figure 8: (a)-(c) Training images; (d)-(f) Images with large feature uncertainty; (g)-(h) Images with a high label variance.
the feature space. Label covariance guided sampling gradually catches up and finally surpasses the feature uncertainty guided sampling. As shown next by the sampled instances, both criteria select informative instances that play complementary roles to improve the AL model. Figure 0 shows how the AL performance is affected by the effective number of mixture components that is automatically determined by an upper bound and component strength ratio p. For component k, we compute its total 'effective posterior observations' 28 g(Oz) El= âgl + lạg and define threshold as 9 = (p/ (o/K)Eu g(O)). The effective components only include those with g(Ox) 2 g. When the size of the training set is still small (early stage in AL), fewer components (large p) yield better results than more components (small p) by avoiding over-fitting. When more training labels are acquired, more flexible model can better explain the label correlations thus has better performance.
Examples of actively sampled instances. To demonstrate the effectiveness of the proposed sampling function, we show images sampled by GP-B2M from the JUS-WIDE dataset. To explain the distinct nature of these sampled images and how they contribute the model training, we also show some representative images from the initial training pool for comparison. These correspond to images in Figure (a)-(c with common labels: 'plants' and 'sky'. First, for sampled images with large feature uncertainty, while 'plants' and/or "sky' are predicted for those images, they look very different from the training images. particular, although the labels of the image in Figure contain both 'plants' and 'sky", there are no visible plants. For the image in Figure 8 (f), the label 'person' is not present even though a person is visible the image. These types of samples are significantly dissimilar to the initial training set, thus considered valuable to explore the feature space for effective sampling. Images Figure (g)-(h) are samples based on a high label covariance. These images look similar to the training examples but their corresponding labels can further improve the prediction of these labels and correlations there of, such as 'plants', 'grass', and 'nature'. As these images bring in additional labels, they may also help the model discover more possible correlations, such as that between 'plants' and 'flowers' or 'tree'.
somewhat different. Sampling these images
5 Conclusion
We present a novel Gaussian -ayess-n Bernoulli Mixture (GP-B²M) model for cost-effective multi-label active learning. GP-B²M extracts global patterns of label correlations by learning from both (limited) training labels and data features. The mixture components, which are accurately learned from end-to-end and fully conjugate posterior inference, are capable of encoding complex label correlations while being highly interpretable. A novel sampling function is designed by combining feature uncertainty and label covariance, both of which can be obtained from the predictive distribution of the GP-B* M model. Experiments conducted on both synthetic and real data justify the important properties of the model and its state-of- ofate-of-the-- of-the--n art AL performance.
10

---

Acknowledgements
This research was supported in part by an NSF IIS award IIS-1814450 and an ONR award N00014- 18-1-2875. The views and conclusions contained in this paper those of the authors and should not be interpreted as representing any funding agency. We would also like to thank the anonymous reviewers for their constructive comments.
References
[1] Hao Fei, Yue Zhang, Yafeng Ren, and Donghong Ji. Latent emotion memory for multi-label emotion classification. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 7692-7699, 2020.
[2] Dongkai Wang and Shiliang Zhang. Unsupervised person re-identification via multi-label classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10981-10990, 2020.
[3] Min-Ling Zhang and Zhi-Hua Zhou. Multilabel neural networks with applications to functional genomics and text categorization. IEEE transactions on Knowledge and Data Engineering, 18(10):1338-133 13511 2006.
[4] Weizhi Liao, Yu Wang, Yanchao Yin, Xiaobing Zhang, and Pan Ma. Improved sequence generation model for multi-label classification via enn and initialized fully connection. Neuro- computing, 382:188-195, 2020.
[5] Min-Ling Zhang, Yu-Kun Li, Xu- Xu-Ying Liu, and Xin Geng. Binary relevance for multi-label learning: an overview. Frontiers of Computer Science, 12(2):191-203 2018.
[6] Cheng Li, Bingyu Wang, Virgil Pavlu, and Javed Aslam. Conditional bernoulli mixtures for multi-label classification. In International conference machine learning, pages 2482-2491, 2016.
[7] Chih-Kuan Yeh, Wei-Chieh Wu, Wei-Jen Ko, and Yu-Chiang Frank Wang. Learning deep latent space for multi-label classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[8] Ya Wang, Dongliang He, Fu Li, Xiang Long, Zhichao Zhou, Jinwen Ma, and Shilei Wen. Multi- label classification with label graph superimposing. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12265-12272, 2020.
[9] André Elisseeff and Jason Weston. A kernel method for multi-labelled classification. In Advances in neural information processing systems, pages 681 1 1-78,, 2002.
[10] Nadia Ghamrawi and Andrew McCallum. Collective multi-label classification. In Proceedings of the 14th ACM international conference on Information and knowledge management, pages 195-200, 2005.
[11] Weiwei Cheng, Eyke Hüllermeier, and Krzysztof Dembezynski. Bayes optimal multilabel classification via probabilistic classifier chains. In Proceedings of the 27th international conference on muchine learning (ICML-10), pages 279-286, 2010.
[12] Jesse Read, Bernhard Pfahringer, and Geoff Holmes. Multi-label classification using ensembles of pruned sets. In 2008 eighth IEEE international conference on data mining, pages 995-1000. IEEE, 2008
[13] Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. Classifier chains for multi- label classification. Machine learning, 85(3):333, 2011.
[14] Bishan Yang, Jian-Tao Sun, Tengjiao Wang, and Zheng Chen. Effective multi-label active learn- ing for text classification. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 917-926, 2009.

---

[15] W. Shi, X. Liu, and Q. Yu. Correlation-awan multi-label active learning for web service tag recommendation. In 2017 IEEE International Conference on Web Services (ICWS), pages 229-236, 2017.
[16] Xin Li and Yuhong Guo. Active learning with multi-label svm classification. In IjCAI, pages 1479-1185.. Citeseer, 2013.
[17] Oscar Reyes, Carlos Morell, and Sebastián Ventura. Effective active learning strategy for multi-label learning. Neurocomputiing, 273:494 2018.
[18] Junyu Chen, Shiliang Sun, and Jing Zhao. Multi-label active learning with conditional bernoulli mixtures. In Pacific Rim International Conference on Artificial Intelligence, pages 954-967, Springer, 2018.
[19] Nengneng Gao, Sheng-Jun Huang, and Songean Chen. Multi-lahel active learning by model guided distribution matching. Frontiers of Computer Science, 10(5):845-8555 2016.
[20] Deepak Vasisht, Andreas Damianou, Manik Varma, and Ashish Kapoor. Active learning for sparse bayesian multilabel classification. Proceedings of the 20th ACM SIGKDD international conference Knowledge discovery and data mining. pages 472-481, 2014.
[21] Weishi Shi and Qi Yu. Fast direct search in optimally compressed continuous target space for efficient multi-label active learning. In International Conference on Machine Learning, pages 5769-5778, 2019.
[22] Radford M Neal. Slice sampling. Annals of statistics, pages 705-741, 2003.
[23] Radford M Neal al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(1 1):2, 2011.
[24] Nicholas G Polson, James G Scott, and Jesse Windle. Bayesian inference for logistic mod- els using pólya-gamma latent variables. Journal of the American statistical Association, (08(504):1339-11 2013.
[25] Théo Galy-Fajou, Florian Wenzel, Christian Donner, and Manfred Opper. Multi-class gaussian process classification made conjugate: Efficient inference via data augmentation. In Uncertainty in Artificial Intelligence, pages 755-765. PMLR, 2020.
[26] Grigorios Tsoumakas, Eleftherios Epyromitros-Xioutt Jozef Vilcek, and Ioannis Vlahavas. Mulan: A java library for multi-label learning. The Journal of Machine Learning Research, 12:2411-2414, 2011.
[27] Sheng-Jun Huang and Zhi-Hua Zhou. Active query driven by uncertainty and diversity for incremental multi-label learning. In 2013 IEEE 13th International Conference Data Mining, pages 1079-1084. IEEE, 2013.
[28] Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] For performance comparison, we tried to include all the representative basclines based on our best knowledge. However, certain baselines may be designed for a specific purpose. As a result, some baseline cannot run properly for the datasets used in our experiments to generate reasonable results. We have made this clear in the performance comparison section of our experiments. Another potential limitation is that all the labels collected are assumed to be accurate, like most other active learning models. Dealing with imperfect labels is an important direction that is out of the scope of this paper.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We provided references of real-world applications (e.g., bioinformatics and medicine) with positive societal impacts. One potential negative impact is that the active learning model could be leveraged by malicious attackers as tool manipulate the supervised learning model. Since the model will be trained using much less data, certain data samples compromised, the impact may be larger than typically trained supervised learning model.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you's 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "wt_WmvRXL6T", "reviewer": "Reviewer_9ST8", "summary": "The paper proposes a generative mixture model for multi-label classification which could handle complex label dependencies. First, the paper introduces mixture components with multi-class probabilistic assignment, which contain information about label distribution. Second,  to maximize joint likelihood the method uses variational inference with auxiliary variables. Third, an active learning strategy is proposed based on the predicted label covariance. The method is compared with other active learning algorithms on multi-label classification datasets. ", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Modeling label distribution is an actively studied problem in multi-label learning research. Because of the internal complexity of the problem, current baseline algorithms for multi-label active learning are quite limited, and can be trained and evaluated only on multi-label datasets of small dimensions (instances/labels/features). This paper makes a good step in improving such algorithms, providing a method for modeling complex label distributions. \n\nThe main idea of learning mixture components in a multi-class way using a gaussian process over data points looks quite natural: it helps to break down the method into two steps (learning components, generating label distributions). The introduced auxiliary variables for  variational inference help to come up with an adequate and interpretable algorithm for active sampling. As a disclaimer, I was not able to check all the math behind the variational inference.\n\nThe proposed method is compared with state-of-the-art baselines for multi-label active learning by three main criteria: 1) active learning AUC, 2) passive learning AUC, 3) time complexity for active sampling. While being inferior to simpler baselines (CBM) in terms of time complexity, or showing similar performance to GS-GP in the passive learning setting, the method demonstrates good overall performance.\n\nTypos: line 27 -- (BRMs), -> (BRMs); line 74: Component 1 -> Component 2 (?); line 191: digmma -> digamma.\n\n-------\nPost-Rebuttal: after reading other reviews and author replies, I decided to keep my score.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The algorithm cannot be evaluated on the large datasets for multi-label classification because of time complexity, but this is a common problem for all methods of such type.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "XvKNST6Epu5", "reviewer": "Reviewer_Fw2q", "summary": "Multi-label learning is an important yet challenging task in machine learning, particularly in extreme settings, where the label space is very large.  Despite that, quantifying the prediction uncertainty in multi-label learning for active learning is even harder, as labels can be correlated.  The paper presents a mixture model for multi-label classification that brings together the Gaussian process and Bernoulli mixture models,  which can be one of its contributions. The other contributions include the variation inference based on variable augmentation, and the use of the label covariance to compute the predictive uncertainty in the acquisition function.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* The proposed multi-label learning model is basically a simple mixture model, where each latent component is models as a distribution over all the possible labels, and each document is associated with a distribution over those latent components. The generative story is quite similar to what is often used in topic modelling. The most inserting part is to use the document feature to generate the mixture weight. Nevertheless, given the rich literature on mixture modelling, the idea is not surprising, instead, the novelty is rather limited. My further comments/concerns are as follows:\n    * Figure 1 shows different relationships between labels, which is quite illustrative to a certain degree. However, It is not clear to me how exactly the mixture model can capture those different types of label dependencies. For me, the proposed model is just a simple mixture model. \n    * In eq (2), should $y_{nk}$ be $y_{nl}$?  According to the model, it seems that the labels are generated from a mixture of latent components, I.e., $\\pi\\times \\theta_{1:K}$. So, is $z_{nk}$ associated with each label of a document? Otherwise, it is not clear how each label is generated given the mixture weight $\\theta$\n* The deviation of the poster inference using variable augmentation is neat. I have a question regards to eq (7), which could be a typo. There are two terms in $exp\\{\\}$. I wonder if $f^{(k)}_n$ should be outside of the parenthesis in the first term?\n* The model makes use of the predicted label covariance to compute the uncertainty for selecting the next sample to be annotated. It is good to be aware of methods like Bald and BatchBald, etc. An in-depth analysis of how the proposed sampling method capture both uncertainty is necessary.\n* To compute the acquisition function used in active learning, the authors proposed to use the predicted label covariance in Eq (13), based on the pointe estimated of the mixture distribution $\\pi$. However, it is good to further carry out in-depth analysis of how the proposed sampling capture the uncertainty that could linked to the expected error reduction. There is a fairly rich literate of how to quantifying uncertainty in active learning, like Bald, BatchBald [3], Weight MOCU [2], etc.\n* The proposed method is evaluated with both synthetic dataset and real-world dataset, which is good. \n    * If the major contribution of this paper is also about the proposed mixture model. It is necessary to carry out a set of experiments where the proposed model is compared with the existing multi-label learning models, to say how accurate it is.\n    * The reason why labels with frequency less than 20 are removed. However, section 4.1 seems to say that the proposed model can deal with few-shot labels.\n    * Should one consider the ranking based measures? Those measures are used more commonly for multilevel learning, see http://manikvarma.org/downloads/XC/XMLRepository.html\n    * It seems that the proposed model does not perform well as some other competitors in figure 7 when the size of the acquired sample is relatively small, and on the NUS-wide data set. Any explanation?\n    * And presumably, the acquisition size, or called batch size is one at each iteration.\n\nReferences\n\n[1] Chu et al 2018, Deep Generative Models for Weakly-Supervised Multi-Label Classification\n[2] Zhao et al 2021, Uncertainty-aware active learning for optimal Bayesian classifier\n[3] Kirsch et al 2019 BatchBALD: Efficient and diverse batch acquisition for deep Bayesian active learning\n\n--- After Review Feedback ---\nI acknowledge that the author's response to this reviewer's original comments has been reviewed and considered in the updated rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No, the authors did not address the limitations and potential negative social impact, which could be discussed from a perspective of applying the proposed active learning in practice. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": ">5 "}, "extra_text": {}}, {"review_id": "TlUqMHXzOPg", "reviewer": "Reviewer_bnVb", "summary": "The paper proposes a novel Gaussian Process-Bayesian Bernoulli Mixture (GP-B2M) model to achieve cost-effective sampling for multi-label active learning.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method encodes label correlations using a Bayesian Bernoulli mixture of label clusters. Furthermore, a predictive Gaussian Process (GP) is adopted to learn a distribution of mixture coefficients that connect data features with the label clusters.\n\nThe learning process is divided into two phases. In the first one, there is a prediction of a probabilistic assignment of the mixture components by learning a distribution of latent functions. In the second one, the predicted mixture assignments are used to refine the parameters of the beta distributions.\n\nThe description of the learning method is detailed and well described in Section 3.\n\nAs regards the experimental results, Figure 7 could be integrated with a quantitative result. In particular a table could be included reporting the numerical results. Furthermore, a statistical test could be done in order to prove that the results are statistical different.\n\nThe adopted single score (AUC) could be integrated using others common scores.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "All the labels collected are440assumed to be accurate", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "NSmGHKhez3d", "reviewer": "Reviewer_vZcB", "summary": "The authors propose a multi-label active learning approach leveraging a Gaussian process Bayesian Bernoulli mixture model to model label correlation. Each pattern of label correlation is modeled as a Bernoulli mixture. The Gaussian process predicts the coefficients of the mixture component as a means to obtain an estimate the labels of a given data sample. End-to-end posterior inference is possible due to the introduction of an auxiliary-latent-variable variational inference algorithm. The number of mixture components is estimated duting Bayesian inference. Also, uncertainty estimates for predictions produce label covariance matrix that quantify individual label correlations. Experiments on real-world multi-label datasets demonstrate the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written, easy to follow and well motivated both from an application and methodology perspective. Though the main data augmentation strategy resulting in a conjugate auxiliary variational Bayes learning is taken from elsewhere ([24] in the paper), its combination with the mixture model is interesting and sufficiently justified. The experiments, including the ablation study, passive learning and active sampling time (the last two in the SM) are extensive and convincing.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Line 74: Component 1 -> Component 2\n\nNote that the realization of the GP in (1) has zero mean function whereas in Figure 2(b) is specified as m_k (though not identified as the posterior mean function).\n\nIn (7) z_nk - v_nk f_n^k is more likely to be ( z_nk - v_nk ) f_n^k\n\nWhy there is a lambda in (6) and (7) when the formulation directly depends on nu, which blocks lambda?", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a Gaussian process Bayesian-Bernoulli mixture model and its variational inference algorithm for multiple label active learning and shows its usefulness by experiments including ablation studies. This paper has three positive reviews and one negative one. In response to the author's FB, no consensus was reached in the discussion among the reviewers. An important point of discussion is whether the proposed mixture model is novel and valid. At least for the latter, the experimental comparison shows its superiority over traditional active learning on multiple label classification datasets. As for the former, as the reviewer pointed out, the use of the Bernoulli mixture model as label distribution is natural and not a particularly novel idea. The novelty of the individual techniques themselves is subtle, but the modeling as a whole is reasonable and can be evaluated as a contribution to the advancement in the research field.
Decision: Accept (Poster)