A Gaussian Process-Bayesian Bernoulli Mixture Model for Multi-Label Active Learning
Authors: Weishi Shi
Dayou Yu
Qi Yu
Venue: NeurIPS 2021 Poster (2021)
Keywords: multi-label active learning
Bayesian Bernoulli mixture
label correlation

Abstract: Multi-label classification (MLC) allows complex dependencies among labels, making it more suitable to model many real-world problems. However, data annotation for training MLC models becomes much more labor-intensive due to the correlated (hence non-exclusive) labels and a potential large and sparse label space.  We propose to conduct multi-label active learning (ML-AL) through a novel integrated Gaussian Process-Bayesian Bernoulli Mixture model (GP-B$^2$M) to accurately quantify a data sample's overall contribution to a correlated label space and choose the most informative samples for cost-effective annotation. In particular, the B$^2$M encodes label correlations using a Bayesian Bernoulli mixture of label clusters, where each mixture component corresponds to a global pattern of label correlations. To tackle highly sparse labels under AL, the B$^2$M is further integrated with a predictive GP to connect data features as an effective inductive bias and achieve a feature-component-label mapping. The GP predicts coefficients of mixture components that help to recover the final set of labels of a data sample. A novel auxiliary variable based variational inference algorithm is developed to tackle the non-conjugacy introduced along with the mapping process for efficient end-to-end posterior inference.  The model also outputs a predictive  distribution that provides both the label prediction and their correlations in the form of a label covariance matrix. A principled sampling function is designed accordingly to naturally capture both the feature uncertainty (through GP) and label covariance (through B$^2$M) for effective data sampling. Experiments on real-world multi-label datasets demonstrate the state-of-the-art AL performance of the proposed GP-B$^2$M model. 

A Gaussian Process-Bayesian Bernoulli Mixture Model for Multi-Label Active Learning
Weishi Shi'
Dayou Yu * Qi Yu
Golisano College of Computing and Information Sciences Rochester Institute of Technology {ws7586, dy2507, qi qii.yu)@rit.edi
Abstract
Multi-lahel classification (MLC) allows complex dependencies among labels, mak- ing it suitable model many real-world problems. However, data tation for training MLC models becomes much more labor-intensive due to the correlated (hence non-exclusive) labels and potentially large and sparse label space. We propose to conduct multi-label active learning (ML-AL) through a novel integrated Gaussian Process-Bayesiie Bernoulli Mixture model (GP-B*M) to accurately quantify a data sample's overall contribution to a correlated label space and choose the most informative samples for cost-effective annotation. In particular, the B encodes label correlations using a Bayesian Bernoulli mixture of label clusters, where each mixture component corresponds a global pattern of label correlations. To tackle highly sparse labels under AL., the B is further integrated with a predictive GP to connect data features as an effective inductive cettrr--ooooonn-taalleee
mapping. The GP predicts coefficients of mixture components that help to recover the final set of labels of a data sample. A novel auxiliary variable based variational inference algorithm is developed to tackle the non-conjugacy introduced along with the mapping process for efficient end-to-end posterior inference. The model also outputs predictive distribution that provides both the label prediction and their correlations in the form of a label covariance matrix. A principled sampling function is designed accordingly naturally capture both the feature uncertainty (through GP) and label covariance (through B M2 for effective data sampling. Experiments real-world multi-labe label datasets demonstrate the state-of-the-an AL performance of the proposed model.
bias and achieve
Introduction
In multi-labe classification (MLC), each data instance may be associated with more than one label. Such rich representation of labels encode more complex data-label distributions that arise in many real-world problems 2 4. As a simple yet powerful tool, binary relevance machines (BRMs) transform MLC problem into multiple binary problems and train independent binary classifiers for each label [5). Such a transformation gives BRMs the flexibility to leverage state-of-art binary classifiers (e.g.. deep neural networks and SVMs). However, applying BRMs to a correlated and potentially large label space poses key challenges. First, many real-world multi-label datasets contain large number of labels. Training one predictor per label incurs a prohibitive cost. Second, despite an overall large label space, each data instance is usually assigned limited labels. Many labels are relatively rare and their appearances depend not only on the features but also the occurrence of other labels. Predicting these "complex" labels directly using independent binary classifiers is fundamentally difficult due the limited positive data instances and weaker direct dependency on the features. Correlations among labels provide important auxiliary information to enhance multi-label
Equaa contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

Label
Descrition of conclations
Caometee losation daca, wach asigned
mesture two Gaussan's
C1-C2
CL: lobel corcinaitty 53, C2: lobel cardinllty 23
01:04
01: G1/02; oo: G7n6%; ca: G1n64; oi: G1nsss
E1-64
Lebeb related the wodanive correlation Jowfined in (b))
96
G6
PIEZ E51
E1: U.
Com 2ompanart
3
Component
05
E11E2 04A.A/11111GG6004110111110000111
(a)
(b)
(c)
Figure 1: (a) Labels with geometric correlation (G1-G8); (b) Labels with cardinality (CI,C2), overlapping (01-04), and exclusive (and hierarchical) dependencies (E1-E4); (c) Definition of label correlation and learned mixture components.
prediction 6 8 However, these models heavily rely on the training data that exhibit these important label correlations. Considering the high cost in annotating a multi-label dataset, it is critical to choose the most informative data samples for cost-effective data annotation.
In this paper, we propose a novel Gaussian Process- Bayesian Bernoulli Mixture (GP-B²M) model to achieve cost-effective sampling for multi-label active learning (ML-AL). ML-AL, since labels are not mutually exclusive as in the single label setting, all the labels should be considered collectively when designing an active sampling function so that data sample's 's overall contribution to the entire label space can be accurately measured. However, since only limited training data instances are available for an ML-AL model, how to accurately model label correlations and hence quantify a data sample's 's overall informativeness using very sparse labels under AL. poses grand challenge. Existing efforts that explicitly model label correlations usually focus on limited types of correlations such as pairwise (3, conditional [10 [T], full correlation in a subset of labels 13 Consequently, those methods may miss some important label correlations. Label correlations can also be captured through latent embedding 7 g. While these methods can scale to large label space, they usually require a decent number of training labels to compute an accurate embedding, making them less suitable for ML-AL. Furthermore, the learned embedding has semantic meanings, which cannot be used to interpret the discovered label dependencies
The proposed GP-B²M model addresses the limitations of existing methods to fundamentally advance MI.AL.. In particular, the BCM encodes label correlations using a Bayesian Bernoulli mixture of label clusters. Since labels are highly sparse in ML-AL, a predictive GP is further integrated to learn a distribution of mixture coefficients that connect data features with the label clusters. Thus, the label clusters can be regarded as global pattern of label co-occurrences discovered from both the training labels and data features to address label sparsity. In this novel data features serve as inductive bias to learn accurate mixture components of labels, where data samples with similar features should be mapped to similar mixture components, which in turn lead to similar set of labels. Such an inductive bias allows the discovery of label relationships from limited labels with the support of feature relationships, which is essential for a sparse label space in ML-AL.
mapping,
Figure D shows three mixture components learned from synthetic data designed with complex label correlations, including geometric, cardinality, overlapping, and hierarchical (see Figure | (c) for definitions). For example, the hicrarchical labels (E'l E4) show some interesting but quite complicated correlations that may exist in many real-world data. If E1 represents a common disease and E2, E3 represent some less common ones that may co-occur with E1 (30% of the time), then E4 corresponds to a rather rare disease that only co-occurs with E3 but not E2. In Figure (c), both components land 2 show a high chance of E4. It is also clear that while both E1 and E3 are very likely to appear in these components, the chance of seeing E2 is much lower, reflecting its exclusive relationship with E3. Interestingly, while both components and 2 cover E4, they also show complementary information. In component 1, E4 mostly appears in the non-overlapping regions, which is indicated by smaller O1 and O4; whereas in component 2, it's more likely to
*Correlation sometimes refers to the degree to which pair of random variables are linearly related. However, in the broadest sense, both correlation and dependence refer any statistical relationship between two random variables, they are used exchangeably in the rest of the paper.

---

occur in overlapping regions O1 and O4. Finally, in component 3, it is less likely to observe E4. Consequently, the chance to see E2 is significantly higher.
The above example demonstrates that the learned mixture components accurately capture complex label correlations that are critical for active data sampling. They are also interpretable, which can help to unveil important relationships among labels. Our key contribution is threefold: (i) We propose a novel GP-Bayesian Bernoulli mixture model discover meaningful label correlations from limited labels by encoding the inductive bias from data features as Bayesian priors to learn from both labels and features while ensuring consistency. (ii) We introduce set of auxiliary latent variables to achieve fully conjugate end-to-end posterior inference. The number of mixture components is also dynamically adjusted during Bayesian inference so that the model complexity is automatically calibrated according to the size of training data, which is critical for AL.. (iii) The model outputs the predictive distribution that provides both the label prediction and their correlations in the form of a label covariance matrix. We design a novel active sampling function that integrates both feature uncertainty and label covariance to quantify data sample's 's overall contribution to a correlated label space. Extensive experiments on both synthetic and real-world multi-label data and comparison with competitive models demonstrate that the proposed GP-B achieves the state-of-the-arr active learning performance.
label mapping in the Bayesian model to support efficient
2 Related Work
Due to the wide adoption of BRMs for MLC, a number of AL models have been developed based upon BRMs. For example, the estimated reduction of BRM loss function has been used an uncertainty criterion for data sampling [1]] Uncertainty from individull SVMs in BRMs has also been integrated to compute sampling score, where label correlation is used to reduce the complexity of the active query rather than improve active sampling
[1]. Label inconsistency provides an alternative way to incorporate label correlation into BRMs [16] for data sampling. This has been further extended through label ranking 177. As discussed earlier, AL models built upon BRMs do not systematically capture label correlations, which may lead to inaccurate uncertainty measures for ML-AL.
A few existing models capture label correlations explicitly or through a latent embedding to support active multi-label sampling. For example, the approximate entropy of the predieted labels from a Bernoulli Mixtures model (CBM) is used for data sampling [18]. However, the quality of the uncertainty estimation relies on external multi-class classifier used to predict the component coefficients. Both model selection and parameter tuning for the classifier make it difficult for AL. One fundamental limitation is that CBM was originally designed for MLC (instead of AL) by predicting distinct set of label clusters for each data sample [6. Thus, different from the proposed GP-B²M model, no global label clusters are discovered to capture the label correlations, making unsuitable for multi-label AL. Gao et.al propose a correlation aware active sampling method for transfer learning task [19]. However, the method uses a kernel function to measure the label similarity between two data instances which cannot work well for large and sparse label space. Compressed sensing (CS) has been employed to learn a latent embedding of the label space to capture potential label correlations, which can then be used to design sampling function. However, since the latent space continuous, the labels are further assumed to be drawn from Gaussian distribution to ensure conjugacy, which violates the binary nature of the labels (20). Furthermore, CS requires an additional step to recover the predicted label from the latent code, which less efficient for AL. Active sampling is also sensitive to the recovery process and the recovery quality is usually low at the beginning of AL due to the lack of training data (21).
The proposed GP-B²M model systematically addresses the key limitations of existing methods through well integrated Bayesian framework that supports a fully conjugate feature-comppnent- label mapping and end-to-end posterior inference for cost-effective ML-AL.
3 GP-B*M for Multi-Label Active Learning
In this section, we first describe the proposed GP-B²M model by introducing key latent variables along with their conditional dependencies. We then present novel posterior inference process by augmenting the original model using a set of auxiliary variables to resolve the non-conjugate prior and likelihood. A principled sampling function is introduced in the end for cost-effective ML-AL.
3

---

Predict (Phase
Generate (Phase
L
K
N Data Samples
K Label clusters
Compress
Correlated Labels
(a)
Figure 2: (a) The GP-BºM framework for MI.AAI; (b) Graphical model of GP-B2M.
3.1 The Bayesian Bernoulli Mixture Model
Let X (X1, ..., X,] denote training set with .N data samples and Y 1y.,,... YN denote the labels, where yn {0, 1 The proposed GP-B²M model assumes there are K mixture components, O = {0 p- shared by all the data samples and the label vector of each sample is generated from
a mixture process. Mixture component k is result of L Bernoulli experiments governed by two parameters asg and byg 02 ~ II'= Beta( bat), where Oµ denotes the probability of assigning label to component k. The indicator variable End denotes whether component
assigned to sample Xn, where 2nk Cat( (T)), Tik h (*))) (A) is mapping function that outputs the probability of assigning X, to mixture component k, and f, = (fh ...,, f/k))/ are GP latent functions for sample X,, with f( ( The final label vector associated with x,, is drawn from the mixture distribution given by p(y,(O) Σ t Figure () shows the graphical model of this generative process.
The GP-B-M model essentially adopts two-phase learning process, where these two phases are seamlessly integrated (see Figure a)). In phase I, it predicts the probabilistic assignment of the mixture components by learning a distribution of latent functions: F = [ where (()) (T(²) y....'' NN (( In phase II, these predicted mixture assignments are used refine the parameters of the beta distributions so that updated mixture components can best recover the true label vectors. One key innovation lies in using the latent indicator variables Z = (z,,, { to link the feature space with the label mixture components as way to encode the feature related inductive bias.
This is achieved through mapping function h. (hn ..... (K)) with / h ( (f.):
K
plznk = 1 1[f,,) = Tin = (1))
(R) ( [0, 11,
Σ
(0) n
= p(((k) = (P(( ma Ex)
(1)
where Σg = [x xmm)) is a covariance matrix and (C(--)) is a kernel function, and we set m with out losing generality. From the Bayesian perspective, this equivalent placing Dirac delta prior over TT Takk ru 0(TTnk 11()))), )), where the inductive bias is encoded by the prior distribution. By introducing (f,)) we essentially convert a multi-label problem into a multi-class problem as nr encodes the probability assigning x to each of the K components. Specifically, given the learned mixture components 0₂ ss for test data sample X., we predict the component assignments n, using the trained GP. The final labels are obtained as p(y-10) Σk T.
Posterior inference of latent variables in the two phases are jointly performed by maximizing the log marginal likelihood of the observed multiple labels for all training samples:
(np(Y = //ΣIII p (1))) (p(OR))(Zza) (tappp/eeeeeeeeeeeeoee (n)p(umllFns dFd
Directly maximizing this likelihood intractable due to the interplay the latent variables. So turn to optimizing the evidence lower bound (ELBO) of the log marginal: L(g) I (9( Z, In F. F.x. where @ Z, F) the variational distribution. However, key challenge that prevents us from using the standard mean field variational inference (MF-VI) is the term p(znk (f,), defined by the mapping function A1 in I As the most typical forms of h (c.g.. softmax) are non-conjugate with the prior distribution p( (f(k) (f))),, which is Gaussian, the variational posterior (((()) cannot be derived analytically.
(6.2,F))

---

3.2 Auxiliary Variables based Variational Inference
We propose to resolve the non-conjugate mapping function in the complete data likelihood by introducing number of auxiliary latent variables such that the augmented complete data likelihood becomes conjugate. Auxiliary variables have been used in MCMC based inference, such as slice sampling [22] and Hamiltonian MCMC [33 with improved sampling efficiency. The basic idea of auxiliary variables based variational inference (AV-VI) is to apply the following transformation: where p(z) is target function that is difficult to compute (c.g., non- conjugate) during VI. If the conditional likelihood p(rly) still non-conjugate, this process will continue until a conjugate conditional is achieved.
p(z) = Jy (a
key identity that we leverage to achieve conditional likelihood conjugate Gaussian prior p(P(k) is convert logistic sigmoid function scale mixture of Gaussian's 24 where the mixture is defined by a Polya-Gammn Gamma distribution p(w) = PG(w)b, 0),
(el)a (1 ef) 2
/0- p(w)dw
(3)
where b 2 0, = a However, a sigmoid function only suitable for binary classification, making it infeasible for 2 mapping function that outputs the assignments for K > components. Thus, we adopt the logistic-softmax function (25) as our mapping function:
(1a+t)
(k) = p(=nk = lff,,) = Σ (FP)
(4)
To handle the summation in we introduce random variables AEN and use identity Jo e e-Aa so that
1
K
p(2nk = 1 (m, = ((f())) II (
(5)
where p(A,) x 1 (0..o),, Vn E NJ. By leveraging the moment generation function of the Poisson distribution Po()), we introduce random variables Y {VI.... UN}. where en (Un1, Unk)
to convert the exponential term in (5). which leads to
p(znk 1ffn vn) vn)= oo(((( ((((-)
j=1
where Unk ~ Po(unt/ An). Finally, using (3) and introducing the Pólya-Gamma random variables [wi,..., WN), where wn = (Wal, (W.n,,.,,W, ",., Wnk) leads
K
(Znk Unt)
(f(h²)²
p(ank = 1/fn, Unky wri) = II
2 (emx +uma) exp
wik
2
2
where W (G (ink 0). Figure shows the graphical model with the auxiliary variables (x,''s are omitted from the graph to keep the notation uncluttered).
We proceed by defining variational distribution with auxiliary variables:
q(O) Z, F, T, = = ()q(A))(TT
(8)
The optimal variational distribution can be obtained by computing the moments of component variational distributions using some important properties of the main and auxiliary variables and iterating until convergence. The optimal variational distributions of the main latent variables are summarized in the following theorem.
Theorem 1. With the auxiliary random variables and the transformed complete conditional likelihood given in o). the optimal components of the variational distribution as specified by g are given by
Component assignments q(Z) = -I,, (z
((znk) Cat(znk (nk); Tnk exp
(aa daa + bui))]
Mak
where 2-(-) is the digamma function and mirk is te-hh element of mean of ((P()) defined in q(F).
5

---

Mni
Fack
whik
Cir
mm
812
An
but
Σ.
NV
Figure 3: Graphical model with auxiliary variables
Bernoulli mixture components (()) II 11: ((xx)
@(0k)) Beta Opi
(ākl b 0 = akt
Onk:Unl: bkd = bui +
Pnk (1 yni) (10)
GP latent functions ((F) I1 (((()))
q(P(k)) = (fx in. Σx); m = 2
Sn( Egve [xx)); Ex (Fig²) diag( Fagws wz) (wx)))
(11)
where Ua (Uiky..., ..., UNK)" we (WiR,... ..., wxa) )2, and T(wE, uk) T(wx lox (7(R)) is the optimal variational distribution for these auxiliary variables.
The specific forms of the auxiliary variational distributions {()) and ((Y,)) are provided in Ap- pendix B as part of the detailed proof of the theorem.
Model interpretation. The optimal variational distributions are fairly intuitive. Interpreting these distributions can reveal some key insights on how the proposed GP-B3M model leverages the data features effective inductive bias to discover semantically coherent components from sparse label space. First, from (9. the component assignment of data sample (x,,yn) is determined by two terms: the first term indicates how all its labels yn correlated with the component and the second term reflects how likely to categorize the features X,, into to the component. Second, from (IO), since the component assignments are further utilized to compute the Bernoulli mixture components, the optimal components naturally aggregate both label and feature information to ensure semantic consistency result of using data features the inductive bias. Last, from (11, the GP latent function value on a component increases with a positive component assignment and decreases with 'negative" assignment, captured by the Poisson auxiliary variables Uk.
Time complexity. According to posterior inference of GP-B²M has the computational com- plexity of O(N*K) which is identical to training K GPs. Since each component be updated independently. we can parallelize the computation to further reduce the complexity to O We can further leverage sparse kernel machines (e.g.. Sparse GP) reduce the complexity if N is large.
3.3 Multi-Label Active Sampling
Being a Bayesian model, GP-B-M outputs the predictive distribution that provides both the label prediction and a label covariance matrix. As the covariance matrix captures both the uncertainty of individual labels and correlation of each pair of labels, it provides essential information design principled measure to quantify data sample's 's overall contribution to correlated label space.
For each testing sample x., the predictive mean can be computed using the variational distributions:
Biy-|x.]= ~ Bor........)/x./Eee..
(12)
Yfx,X,Y,x.)) F(t)) [0)) 0h. (Øgt bua
where Tak = p(=.k 1ff,) ).)()))))) is defined as logistic-softmax function given in Theorem 1 shows that q( F), which approximates the true posterior p( (F|X, Y), a Gaussian. Hence, is also a Gaussian. However, the logistic-sfftmax transformation makes predictive mean intractable
x.)

---

(a)
(b)
(c)
(d)
Figure 4: (a) Distribution of E4 samples; (b) Prediction by BRMs; (c) Mixture component assign- ments by M (d) Prediction by M
to compute. We propose to conduct Monte Carlo (MC) integration by drawing samples from p(f,|X, XX, Y, x.). perform logistic-softmal transformation, and then average.
The GP-B²M M model also allows us to compute the predicted label covariance,
cov Nov[y.|x.] = Σ E [B[AA] E [[,,]"" Ely- x. xx]]E(y.|x.
(13)
where A₂ = (E@g]([ [[Oa])) The predicted label covariance captures both individual label uncertainty (diagonal entries of the matrix) and label correlations (off- (off-diagonal entries), which is instrumental to quantify the total uncertainty of a test sample with respect to its predicted labels. Since directly computing the entropy of mixture distribution is challenging, use the log determinant of covariance matrix: In|cov[y [y, xx.].. as a proxy for uncertainty evaluation. Intuitively, this is equivalent to approximating p(y. (x,) using multivariate Gaussian, whose entropy is the log determinant of its covariance matrix plus a constant.
instead choose to
The label covariance is computed using point estimate of (T=1...., x (one T. for each class) to quantify the total uncertainty on the label side. As Bayesian model, the proposed GP-B3M allows us quantify the variation of each πą using its predictive variance. Through MC integration as described above, we compute the predictive variance Var[T=*] of sample x, for each of the K easily show that the model would assign
class. According to the property of the GP posterior, a low variance to data samples near to the training data and a high variance to faraway samples. As a result, the predictive variance effectively captures the feature uncertainty that complements the label covariance. It allows the proposed sampling function to differentiate data samples based on their distinct contributions to model training and sample them accordingly. Our final sampling function is given by: x. = arg maxx, label covariance and predictive variance of data features. It can be dynamically updated give a higher weight in the early stage of AL to the feature variance term for better exploration of the data space and then shift the focus to the label covariance term for effective fine-tuning of decision boundaries with correct shape obtained through effective exploration.
|x.yy|xx]|++>)) Var War[T=+]/K KK, where is used to balance between
4 Experiments
We conduct extensive experiments on both synthetic and real-worke multi- data to demonstrate: (1) important properties of GP-B-M to capture complex label correlations and how they contribute to predict complex labels, (2) state-of-the-art ML-AL performance by comparing with existing competi- tive models, (3) impact of key model parameters through an ablation study, and (4) effectiveness of active sampling by examining sampled data instances.
4.1 Synthetic Data
We design synthetic dataset with 18 labels that exhibit 4 distinct types of dependencies defined in Figure T (c). In the introduction, we show that three discovered mixture components precisely capture some rather complex label dependencies (e.g., hierarchical and exclusive) while being highly interpretable. For this dataset, the model discovers 10 components in total and we show some other components in Appendix (D allonn with their interpretations. We further demonstrate how the discovered components contribute to the prediction of more complex and less frequent labels. We use E4 as example, which is located deep in the hierarchy and appears much less than other labels.
Figure 4 (a) shows the distribution of data samples whose labels contain E4. can be seen that these samples are distributed across the entire El region (roughly corresponds to the shaded area in purple).
7

---

Note that, in addition to E1, E4 also depends on the exclusive relationship: E3 AND NOT E2 (E2, E3 are not shown in the figure to keep the distribution of E4 clear). Figure (b) shows the prediction result from BRMs, which has very high false positive and negative rates. The poor performance is also reflected by a low ROC-AUC (area under the receiver operating characteristic curve) score at 0.58 (slightly better than random guessing). It appears that BRMs only predict correctly samples in an area where E4 samples are relatively dense while missing most others. This is because BRMs try to directly learn the feature-label mapping (by training independent binary predictors), which is usually weak for complex and less frequent labels, like E4.
Different from BRMs, the proposed GP-B²M learns mixture components that correctly capture the label correlations and the final labels can be recovered by combining the mixture components through their predicted coefficients. As discussed earlier, E4 has a high chance to appear in either
Figure 5: AUC on different types of labels
Label
G1-G8
C1,C2
O1-04
E1-E4
BRMs
0.79
0.68
0.72
0.58
GP-B"M
0.83
0.70
0.86
0.82
Component 1 or 2. Figure shows the predicted component assignments and the top component is highlighted. As can be seen, most E4 samples are assigned Component I or 2 as their top component. Also, samples assigned to Component 1 (shown in blue) are mostly distributed in the non-overlapping geometric regions and those assigned to Component (shown in yellow) are mostly in overlapping regions. By leveraging these components, GP-B²M achieves much better prediction results as shown F Table Summmaiies the AUC scores from BRMs and GP-B3M M on different types of labels. While both models achieve similar performance on some common labels (e.g., G), GP-B3M significantly outperforms BRMs for more complex labels, where it is essential to capture important label correlations.
4.2 Real Data
Datasets and experiment settings. We choose five representative real-world multi-label datasets, in- cluding Delicious, Enron, Bibtex, Corel5K, and NUS-WIDE, from different application domains [26] All datasets have a relatively large label space and high label sparsity (2 (2-6% 6%). Table Appendix D summarizes key properties of the pre-processed datasets. We randomly shuffle each dataset and partition them into three parts: training, testing, and candidate pool. We keep a minimum of one positive instance per label in the initial training partition as required by BRMs based AL models. To make sure each label is well represented in each partition, we remove extremely rare labels with label frequency less than 20. Since the remaining labels are still highly imbalanced, we use the ROC-AUC score to evaluate the model performance. All the baseline models share the same copy of the initial training set to make fair comparison. Active learning stops after each model selects 500 samples.
Performance comparison. We include five competitive baselines for AL performance comparison:
MMC samples instances that introduce the greatest change of the expected loss. During label
prediction, it uses logistic regression to predict the number of labels for new instance 14.
Adaptive considers both the separation margin of an SVM and the label cardinality inconsistency and combines these two parts for data sampling [16).
AUDI uses a label ranking mechanism, where dummy label is used to separate the positive and negative labels. Its sampling function is based on a modified cardinality inconsistency measure (27).
CVIRS combines the ranking on the magnitude of the difference margin in predictions and the
label vector inconsistency for active sampling [17].
CS-GP conducts active sampling in a compressed label space using a multi-output GP 21
Freure&rreports the AUC scores of all the models. For each curve, we present the average result along with the error bar from 3 trials of randomly initialized AL experiments. The proposed B model achieves better AL performance consistently on all the datasets. For multiple datasets, GP-B2M establishes a clear advantage in the early to middle stages of AL. While a few baselines eventually converge to a similar AUC score, they usually take more iterations (by consuming more labels) to reach comparable performance as GP-B*M. In addition, by comparing with random sampling with the proposed sampling function, we clearly demonstrate that the superior AL performance attributes to both the Bayesian mixture model and the effective active sampling. Note that the AUDI model runs much slower for Bibtex and Corel5K when both the number of features and candidate pool size become very large so we omit the results.
Analysis on label-wise improvement. To further justify why the proposed GP-B²M outperforms the basclines, we offer fine-grained analysis on the label-wise performance, which helps to achieve
8

---

Detcious Dataget
Enron Dataset
B Blox Dacaset
CorafSi Ducaset
639
6:3
Inac
611
CA.a
66%
C71
CSR
Number Selected Samples
Number Selected Somp es
Number Selected Samples
Number of Sellcceed Samplcs
(a)
NUS-ANDEE Datase
Enron Dataget
Ponon Darewet
O0B
065
645
6.1
6.1g
kumoor of Solocttdd Samples
Number Selecced Samples
6%%
Kumber of Selected Samples
ze 10 369
(e)
(f)
Figure 6: AL performance comparison (a): Ablation study on one example dataset (Enron): (f) shows the impact of 1) and (g) shows the impact of effective components.
Percentage of Improvement
Delicious Dataset
Percentage of AP improvement
7:0
NUS-WIDE Dataset
800
500
4c0
1:c
300
1c0
30423020 186181129166
75
476
366
222
117 160
Label Frequency
Label Frequency
(a)
(b)
Figure 7: The label-wise average precision improvement over BRMs models.
a deeper insight on the performance gain. In fact, we reach a similar conclusion as the synthetic data experiments: while all models perform reasonably well on the most frequent labels, GP-B²M clearly outperforms other baselines on less frequent labels, where considering the correlations with other labels can help to improve their predictions. Thus, our fine-grained analysis, we first exclude the few most frequent labels and compute the Average Precision Improvement (API) of the i-th label,
AP
AP(( BRMs 100%
APT
B
(14)
AP(
BRMs
where (P(1) B and APl() denote the average precision of label i provided by GP-B*M and BRM, BRMs, respectively. Figure demonstrates digunredeemosssrttss the label-wise API on the NUS- LUD- and Delicious datasets as two illustrative examples. We observe that GP-B²M performs significantly better than BRMs on labels with a moderate or low frequency. This is because those labels usually do not have sufficient positive instances for BRMs learn independently. Furthermore, they may have complex correlation with other labels. GP-B"M effectively leverages label correlations to make better predictions on these labels. This further confirms the overall good performance of GP-B*M.
Ablation study. We further investigate the impact of two tunable parameters of the model: (1) n- which balances label covariance and feature uncertainty for data sampling and (2) p. which controls effective number of mixture components. Limited by space, we use the Enron dataset as an example and report other results in Appendix D Figure 6 (f) compares the performance under different y values. In early iterations, the label covariance guided sampling (n is small) slightly falls behind the feature uncertainty guided sampling (1) is large) the latter more useful to explore

---

(a) ('clouds', ... 'grass', "hills'....... (b) l'olonds',
'landscape', (C) ('clouds', 'farm,, 'landscapp', (d)
('house',
'landscape"s ;
plannts, , 'skyy,, 'valley']
pplants', 'sky'l
'nature', pplants', 'sky". , "sun") 'plants', 'skyy"
(e) l'architecture', "buildings', (1) "clonds', 'plants', isky']
I'aamm. "Geld", "grass" (g) I'clouds', "farm". 'Blowers', (h) I'art', "clouds', "grass", "na- 'plants']
grass" 'plants", 'skyy]
ture', 'plants", 'skyy, 'tree']
Figure 8: (a)-(c) Training images; (d)-(f) Images with large feature uncertainty; (g)-(h) Images with a high label variance.
the feature space. Label covariance guided sampling gradually catches up and finally surpasses the feature uncertainty guided sampling. As shown next by the sampled instances, both criteria select informative instances that play complementary roles to improve the AL model. Figure 0 shows how the AL performance is affected by the effective number of mixture components that is automatically determined by an upper bound and component strength ratio p. For component k, we compute its total 'effective posterior observations' 28 g(Oz) El= âgl + lạg and define threshold as 9 = (p/ (o/K)Eu g(O)). The effective components only include those with g(Ox) 2 g. When the size of the training set is still small (early stage in AL), fewer components (large p) yield better results than more components (small p) by avoiding over-fitting. When more training labels are acquired, more flexible model can better explain the label correlations thus has better performance.
Examples of actively sampled instances. To demonstrate the effectiveness of the proposed sampling function, we show images sampled by GP-B2M from the JUS-WIDE dataset. To explain the distinct nature of these sampled images and how they contribute the model training, we also show some representative images from the initial training pool for comparison. These correspond to images in Figure (a)-(c with common labels: 'plants' and 'sky'. First, for sampled images with large feature uncertainty, while 'plants' and/or "sky' are predicted for those images, they look very different from the training images. particular, although the labels of the image in Figure contain both 'plants' and 'sky", there are no visible plants. For the image in Figure 8 (f), the label 'person' is not present even though a person is visible the image. These types of samples are significantly dissimilar to the initial training set, thus considered valuable to explore the feature space for effective sampling. Images Figure (g)-(h) are samples based on a high label covariance, These images look similar to the training examples but their corresponding labels can further improve the prediction of these labels and correlations there of, such as 'plants', 'grass', and 'nature'. As these images bring in additional labels, they may also help the model discover more possible correlations, such as that between 'plants' and 'flowers' or 'tree'.
somewhat different. Sampling these images
5 Conclusion
We present a novel Gaussian -ayess-n Bernoulli Mixture (GP-B²M) model for cost-effective multi-label active learning. GP-B²M extracts global patterns of label correlations by learning from both (limited) training labels and data features. The mixture components, which are accurately learned from end-to-end and fully conjugate posterior inference, are capable of encoding complex label correlations while being highly interpretable. A novel sampling function is designed by combining feature uncertainty and label covariance, both of which can be obtained from the predictive distribution of the GP-B* M model. Experiments conducted on both synthetic and real data justify the important properties of the model and its state-of- ofate-of-the-- of-the--n art AL performance.
10

---

Acknowledgements
This research was supported in part by an NSF IIS award IIS-1814450 and an ONR award N00014- 18-1-2875. The views and conclusions contained in this paper those of the authors and should not be interpreted as representing any funding agency. We would also like to thank the anonymous reviewers for their constructive comments.
References
[1] Hao Fei, Yue Zhang, Yafeng Ren, and Donghong Ji. Latent emotion memory for multi-label emotion classification. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 7692-7699, 2020.
[2] Dongkai Wang and Shiliang Zhang. Unsupervised person re-identification via multi-label classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10981-10990, 2020.
[3] Min-Ling Zhang and Zhi-Hua Zhou. Multilabel neural networks with applications to functional genomics and text categorization. IEEE transactions on Knowledge and Data Engineering, 18(10):1338-133 13511 2006.
[4] Weizhi Liao, Yu Wang, Yanchao Yin, Xiaobing Zhang, and Pan Ma. Improved sequence generation model for multi-label classification via enn and initialized fully connection. Neuro- computing, 382:188-195, 2020.
[5] Min-Ling Zhang, Yu-Kun Li, Xu- Xu-Ying Liu, and Xin Geng. Binary relevance for multi-label learning: an overview. Frontiers of Computer Science, 12(2):191-203 2018.
[6] Cheng Li, Bingyu Wang, Virgil Pavlu, and Javed Aslam. Conditional bernoulli mixtures for multi-label classification. In International conference machine learning, pages 2482-2491, 2016.
[7] Chih-Kuan Yeh, Wei-Chieh Wu, Wei-Jen Ko, and Yu-Chiang Frank Wang. Learning deep latent space for multi-label classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[8] Ya Wang, Dongliang He, Fu Li, Xiang Long, Zhichao Zhou, Jinwen Ma, and Shilei Wen. Multi- label classification with label graph superimposing. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12265-12272, 2020.
[9] André Elisseeff and Jason Weston. A kernel method for multi-labelled classification. In Advances in neural information processing systems, pages 681 1 1-78,, 2002.
[10] Nadia Ghamrawi and Andrew McCallum. Collective multi-label classification. In Proceedings of the 14th ACM international conference on Information and knowledge management, pages 195-200, 2005.
[11] Weiwei Cheng, Eyke Hüllermeier, and Krzysztof Dembezynski. Bayes optimal multilabel classification via probabilistic classifier chains. In Proceedings of the 27th international conference on muchine learning (ICML-10), pages 279-286, 2010.
[12] Jesse Read, Bernhard Pfahringer, and Geoff Holmes. Multi-label classification using ensembles of pruned sets. In 2008 eighth IEEE international conference on data mining, pages 995-1000. IEEE, 2008
[13] Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. Classifier chains for multi- label classification. Machine learning, 85(3):333, 2011.
[14] Bishan Yang, Jian-Tao Sun, Tengjiao Wang, and Zheng Chen. Effective multi-label active learn- ing for text classification. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 917-926, 2009.

---

[15] W. Shi, X. Liu, and Q. Yu. Correlation-awan multi-label active learning for web service tag recommendation. In 2017 IEEE International Conference on Web Services (ICWS), pages 229-236, 2017.
[16] Xin Li and Yuhong Guo. Active learning with multi-label svm classification. In IjCAI, pages 1479-1185.. Citeseer, 2013.
[17] Oscar Reyes, Carlos Morell, and Sebastián Ventura. Effective active learning strategy for multi-label learning. Neurocomputiing, 273:494 2018.
[18] Junyu Chen, Shiliang Sun, and Jing Zhao. Multi-label active learning with conditional bernoulli mixtures. In Pacific Rim International Conference on Artificial Intelligence, pages 954-967, Springer, 2018.
[19] Nengneng Gao, Sheng-Jun Huang, and Songean Chen. Multi-lahel active learning by model guided distribution matching. Frontiers of Computer Science, 10(5):845-8555 2016.
[20] Deepak Vasisht, Andreas Damianou, Manik Varma, and Ashish Kapoor. Active learning for sparse bayesian multilabel classification. Proceedings of the 20th ACM SIGKDD international conference Knowledge discovery and data mining. pages 472-481, 2014.
[21] Weishi Shi and Qi Yu. Fast direct search in optimally compressed continuous target space for efficient multi-label active learning. In International Conference on Machine Learning, pages 5769-5778, 2019.
[22] Radford M Neal. Slice sampling. Annals of statistics, pages 705-741, 2003.
[23] Radford M Neal al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(1 1):2, 2011.
[24] Nicholas G Polson, James G Scott, and Jesse Windle. Bayesian inference for logistic mod- els using pólya-gamma latent variables. Journal of the American statistical Association, (08(504):1339-11 2013.
[25] Théo Galy-Fajou, Florian Wenzel, Christian Donner, and Manfred Opper. Multi-class gaussian process classification made conjugate: Efficient inference via data augmentation. In Uncertainty in Artificial Intelligence, pages 755-765. PMLR, 2020.
[26] Grigorios Tsoumakas, Eleftherios Epyromitros-Xioutt Jozef Vilcek, and Ioannis Vlahavas. Mulan: A java library for multi-label learning. The Journal of Machine Learning Research, 12:2411-2414, 2011.
[27] Sheng-Jun Huang and Zhi-Hua Zhou. Active query driven by uncertainty and diversity for incremental multi-label learning. In 2013 IEEE 13th International Conference Data Mining, pages 1079-1084. IEEE, 2013.
[28] Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] For performance comparison, we tried to include all the representative basclines based on our best knowledge. However, certain baselines may be designed for a specific purpose. As a result, some baseline cannot run properly for the datasets used in our experiments to generate reasonable results. We have made this clear in the performance comparison section of our experiments. Another potential limitation is that all the labels collected are assumed to be accurate, like most other active learning models. Dealing with imperfect labels is an important direction that is out of the scope of this paper.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We provided references of real-world applications (e.g., bioinformatics and medicine) with positive societal impacts. One potential negative impact is that the active learning model could be leveraged by malicious attackers as tool manipulate the supervised learning model. Since the model will be trained using much less data, certain data samples compromised, the impact may be larger than typically trained supervised learning model.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you's 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "yMvxEhN_h1-", "reviewer": "Reviewer_dgB5", "summary": "The paper proposes a novel model for multi-label active learning that could learn label correlations from both limited labels and data features. This is achieved through a two-phase learning process that uses a Bayesian Bernoulli mixture of label clusters and the Gaussian Process to predict coefficients of mixture components. The paper also proposes a sampling function to consider both feature uncertainty and label covariance obtained by the model.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation for the proposed methods is well-explained and the paper is well-written overall. \n\nThe paper addresses multi-label active learning by combining a Bayesian Bernoulli mixture of label clusters and the Gaussian Process that predicts coefficients of mixture components. The predictive distribution could provide feature uncertainty and label covariance for the sampling function. The proposed methods are theoretically sound.\n\nExperiments on the synthetic and real multi-label data demonstrate the strength of the model. The label-wise improvement in the appendix shows that the model performs better than BRMs on labels with a lower frequency. The authors argue that this is because tail labels do not have sufficient examples for BRMs and the proposed model could use the label correlations. It should be noted that the AP deterioration of the proposed model on tail labels is also more significant. This could be worth further investigation.\n\n**Typo:**\n\n- Line 24  more than one label", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": " The paper should not pose any negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5.5"}, "extra_text": {}}, {"review_id": "qB643DkermH", "reviewer": "Reviewer_iRRM", "summary": "A Bayesian model for multi-label assignment is introduced.\nThe paper is about training the model with active learning, and this is its main goal.\nThough, the model usefulness is prediction (classification).\nTherefore, although not mentioned or not made clear in the manuscript, there are two goals in this paper:\npresent a new model for multi-label prediction, and present a new way to do active learning.\nThis second task is developed for the given model.\n\nThe model is a Bayesian model composed of Gaussian latent variables and a finite mixture of\nof multi-label clusters. The Gaussian latent variables are use to estimate the probabilities\nof items being generated by specific components of the multi-label mixture. In this sense, this model\nlooks like an expert mixture model.\n\nThe active learning is done by selecting samples that show two kinds of uncertainty in the classification:\nlabel uncertainty, measure by a proxy of label entropy, and classification uncertainty, measured by assignment probability\nvariance.\n\nThe performance of the model is compared with five other alternative models.\nThe proposed model performs well, presenting high efficacy early on the sampling;\nby the time 500 active learning samples have been shown to the methods, one can see\nthat other methods may perform slightly better, but still comparable to the proposed method.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nORIGINALITY\n\nThe model is a Bayesian model composed of Gaussian latent variables and a finite mixture of\nof multi-label clusters. The Gaussian latent variables are use to estimate the probabilities\nof items being generated by specific components of the multi-label mixture. In this sense, this model\nlooks like an expert mixture model. The authors should comment on this similarity.\n\nThe active learning is done by selecting samples that show two kinds of uncertainty in the classification:\nlabel uncertainty, measure by a proxy of label entropy, and classification uncertainty, measured by assignment probability\nvariance.\n\n\nSIGNIFICANCE\n\nActive learning is developed for the given model. However, I think that the objective function used to\ndo active learning could be used for any other probabilistic model for multi-labeling.\n\nLines 46 to 53, in this paragraph, several critiques are given to existing methods for active learning:\npairwise correlation, full correlation (??) on subset of labels; latent embedding requires a large number of training labels;\nlatent embedding has no semantic meaning.\nHowever, the model presented in this paper introduces latent embeddings (through Gaussian distributions), and\nuses pairwise label correlations (correlations are between pairs of labels) in a covariance matrix.\nHow is then the proposed model addressing the limitations mentioned above?\n\nLines 77 to 91, the paper mentions \"label correlations\" many times. I would think that the right way of thinking about\nthe relation among labels is \"label dependencies\". It is a bit strange to talk about\nlabel correlations, when labels are nominal variables.\nAt first, I thought \"label dependencies\" is what the authors meant; but by looking\nat the final output, I see that all what is measured is label correlations.\n\n\nQUALITY\n\nA drawback from the proposed method is its computational cost, which following the authors' claim is of\norder O( [K= number of components] x [N= number of initial data]^3 ). How does this compare with the five\nalternative methods used in the experiments?\n\nThe paper mention a Gaussian process, but these are just Gaussian random variables associated to\nmixture components. The process is hidden in the covariance matrix which is a Gram matrix from a kernel.\nBut not much is said about this kernel. How is this kernel chosen?\nAlso, it is not clear if the Gram matrix is evaluated in the covariates or the points?\n\nLine 123, it is stated that \"A principled sampling function is introduced in the end for cost-effective ML-AL\"\nWhy is that function \"cost-effective\". Please justify your claims.\n\nLine 140, it is stated \"One key innovation lies in using the latent indicator variables Z...\"\nThis is far from innovative; this is the usual manner of dealing with finite mixtures.\n\nAbout variational inference: Why putting so much emphasis on using conjugate priors or hyper-prior (by augmenting the data) if at the end (line 220) you still cannot obtain analytical expressions for the predictive posterior?\n\nAbout time complexity (lines 209 to 211), O(K N^3) is huge, even if using parallelization, O(N^3) is huge.\nCould you display the training time of the methods compared in Section 4?\nAlso, one of the advantages of using Bayesian models is the shrinking produced by sharing data in a hierarchical model.\nThe claim that the algorithm may be parallelized would indicate that not all data information is being passed over the\nwhole model. Some comments on this should be added.\n\n\nCLARITY\n\nThe Bayesian model is presented equation by equation. This makes it hard to understand.\nThe authors should describe the model clearly as it is usually done for Bayesian models,\nspecifying the different distributions associated with the different variables in a hierarchical manner.\n\nPage 2, there is a nice try to show a simple example of the multiple label problem. Unfortunately, the example\nis not very clear. I would have preferred the authors show an example based on real labels and correlations between labels.\nThis would make the problem, and perhaps the model as well,  more interesting and intuitive.\n\nLine 82, point (i) What do you mean with \"ensuring consistency\"? What consistency are you referring to?\n\nLine 85, point (ii) What do you mean with \"calibration\" and \"automatic calibration\" here?\n\nLines 87 to 89, it is a bit too early to talk about \"label covariance\" in the introduction without having seen the model.\nAlso, What is a \"correlated label space\"\n\nLines 107 to 115, I feel the competitor models are being unfairly criticized. I agree that CBM is different from the proposed model, but that does not makes it necessarily \"unsuitable\" for active learning. There is always the possibility of using some measure of uncertainty to choose samples, given a model. Also, the critique to CS for using Gaussian latent embedding seems odd, since\nyour own model uses Gaussian latent embeddings. You said that this \"violates the binary nature of the labels\"; but many statistical models uses this technique for modeling categorical data by means of thresholds.\nAlso, Why is CS \"less efficient\" for active learning, say than the proposed model?\n\nSection 3.1, first paragraph, the model is being introduced here, but it is not very clear. For example, the components\nof the mixture (which are supposed to be distributions) are being confused with the parameters of the components {theta_k}.\nThen in line 133, one sees p(y) = sum_k pi_nk theta_nk. First, the notation p(y) has never been defined in the paper;\nsecondly, sum_k pi_nk theta_nk is just a weighted average of the parameters. My guess is that p(y) is the mixture\ndistribution of y, and that the sum should be over the distributions, something like sum_k pi_nk p( y | theta_nk),\nwhere p( y | theta_nk ) is a product of independent Bernoulli distributions with success probabilities given by\nthe vector theta_nk.\n\nEquation (1), Where is the role of X in these equations?\nYou write p(f^(k) |  X) = N( f^(k) |  0, Sigma_k ). I guess one should write Sigma_k (X).\nAlso, you have to define the notation N( | 0, Sigma), i.e., say explicitly that this is a multivariate normal (of what dimension?).\n\nLine 160, What is the \"non-conjugate mapping function\"?\n\nLine 166, Please define \"VI\". I guess this stands for variational inference. However, in this sentence it might be\nbetter to talk about variational estimation instead of variational inference.\n\nEquation (12), I guess on the first line of this equation, the equality sign should be changed to approximation sign, since\nthe right-hand-side seems to be using the variational approximation q().\nIn the second line, Where did the variable Y go in the integrand?\n\nLines 235, 236, it says \"By assigning a low variance to data samples near to the training data and a high variance to faraway samples, the predictive variance effectively captures the feature uncertainty...\"\nWho assigns these values to the samples? Are you just stating that this is what your model does? If so, Where is the argument and the justification for this claim?\n\nLine 316, Where is the quantity \"rho\" defined? Could you show the equation where it is defined?\n\n\n\n\nOTHER\n\nLine 24, change \"labels.\" to \"label.\"\n\nLine 112, change \"the labels are further assumed to draw from a Gaussian to ensure...\" to\n\"the labels are further assumed to BE drawN from a Gaussian DISTRIBUTION to ensure...\"\n\nLines 116 to 118, the last two line of this sentence are repeated from point (ii) above in Page 3.\n\nLine 133, I would suggest changing \"the final label vector for x_n is formed by\" by\n\"the final label vector associated with x_n is drawn from the mixture distribution\"\n\nLine 165, the notation p(x) has already been used before for p(y).\n\nLines 253, 254, I still do not quite understand this example, so I would avoid saying that the result is \"highly interpretable\".\n\nLine 260, What are the symbols &! representing here?\n\nFigure 7, the caption and curves are too small to be seen clearly.\n\n\nREBUTTAL\n\nI have read the authors' rebuttal and the other colleagues' reviews. I thank the authors for trying to clarify some issues.\nHowever, the answers have not made me change my mind. In the first place, there were not too many issues in the original manuscript.\nI find the proposed method interesting but overly complex, involving many parameters, and hyper-parameters, but, still producing very similar results to other competing methods.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nSOCIETAL IMPACT\nNone mentioned in the paper. But they are describe in the Checklist.\n\nLIMITATIONS\nNone mentioned in the paper. But they are describe in the Checklist.\n\nBoth Societal impact and Limitations should be moved to the main paper.\n", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "14"}, "extra_text": {}}, {"review_id": "VIYsUMuUOxY", "reviewer": "Reviewer_AzUM", "summary": "This paper introduces a Gaussian Process Bayes Bernoulli mixture model based solution for cost-effective multi-label active learning. The proposed graphical model can account for diverse label correlations by mixture components and by the feature-component-label mapping. The proposed method gives better results than other compared methods and show the Bernoulli mixture model can better capture the label correlations.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Multi-label active learning have been popularly studied in recent years and many solutions have been proposed, including the graphical model based ones. Although the authors highlight the distinction of the proposed method and clarify the difference of the proposed one to related ones. In addition, they also show better results than other compared methods. I still have some comments for the authors:\n1. There are related methods that used sample-label pairs that can make use of label correlations for multi-label active learning and to save the query the cost, but the authors do not well recognize them and criticize there are few works consider the label correlations for multi-label active learning, which downgrade the technical Quality and Originality.\n2. The technical Clarity is clear but the experimental results are not so well analyzed. For example:\n(1). The eta in Figure 6 do not have clear effect on its expected effect as the author stated, this fact also applies for rho, it is hard to clearly see the contribution factor. The results are not well analyzed yet, it is still not so clear how the proposed methods give better results than other compared methods, and the abaltion study is not so informative and supportive.\n(2). Most active learning methods have a large variance in Figure 7, there is not clear pattern that the proposed method gives better results.\n3. AUC-ROC is not a so good class-imbalance evaluation metric. AUC-PRC is more suitable for class balance classification.\n4. More recent cost-effective mutli-label active learning method that use sample-label pair selection are omitted. For your information, some of them are as follows:\n(1). Multi-instance multi-label active learning. IJCAI 2017.\n(2). Multi-label active learning by model guided distribution matching. Frontiers Comput. Sci, 2016.\n(3). Transfer Learning with Active Queries from Source Domain. IJCAI 2016\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission considers the problem of active learning for multi-label classification, and propose a two phase Bayesian approach that takes label correlation into account.

Three reviewers carefully considered the submission, and found several weaknesses in the presentation and proposed idea. In particular, reviewer iRRM provided a detailed set of suggestions to the authors for improving the paper. The author rebuttal was considered by all the reviewers, and its contents was briefly discussed post rebuttal. The reviewers all agreed after considering each others' reviews as well as the rebuttal that there were remaining issues with the submission. Unfortunately, due to the large number of high quality submissions at NeurIPS, this submission cannot be accepted in its current state.
Decision: Accept (Poster)