Title: A Gaussian Process-Bayesian Bernoulli Mixture Model for Multi-Label Active Learning
Authors: Weishi Shi
Dayou Yu
Qi Yu
Venue: NeurIPS 2021
Keywords: multi-label active learning
Bayesian Bernoulli mixture
label correlation

Abstract: Multi-label classification (MLC) allows complex dependencies among labels, making it more suitable to model many real-world problems. However, data annotation for training MLC models becomes much more labor-intensive due to the correlated (hence non-exclusive) labels and a potential large and sparse label space.  We propose to conduct multi-label active learning (ML-AL) through a novel integrated Gaussian Process-Bayesian Bernoulli Mixture model (GP-B$^2$M) to accurately quantify a data sample's overall contribution to a correlated label space and choose the most informative samples for cost-effective annotation. In particular, the B$^2$M encodes label correlations using a Bayesian Bernoulli mixture of label clusters, where each mixture component corresponds to a global pattern of label correlations. To tackle highly sparse labels under AL, the B$^2$M is further integrated with a predictive GP to connect data features as an effective inductive bias and achieve a feature-component-label mapping. The GP predicts coefficients of mixture components that help to recover the final set of labels of a data sample. A novel auxiliary variable based variational inference algorithm is developed to tackle the non-conjugacy introduced along with the mapping process for efficient end-to-end posterior inference.  The model also outputs a predictive  distribution that provides both the label prediction and their correlations in the form of a label covariance matrix. A principled sampling function is designed accordingly to naturally capture both the feature uncertainty (through GP) and label covariance (through B$^2$M) for effective data sampling. Experiments on real-world multi-label datasets demonstrate the state-of-the-art AL performance of the proposed GP-B$^2$M model. 

TL;DR: We propose a novel integrated Gaussian process-Bayesian Bernoulli mixture model to accurately quantify a data sample's overall contribution to a correlated label space for cost-effective multi-label active learning.

Review discussion:
Multi-label active learning have been popularly studied in recent years and many solutions have been proposed, including the graphical model based ones.
Although the authors highlight the distinction of the proposed method and clarify the difference of the proposed one to related ones.
In addition, they also show better results than other compared methods. I still have some comments for the authors:
1. There are related methods that used sample-label pairs that can make use of label correlations for multi-label active learning and to save the query the cost, but the authors do not well recognize them and criticize there are few works consider the label correlations for multi-label active learning, which downgrade the technical Quality and Originality.
2. The technical Clarity is clear but the experimental results are not so well analyzed.
For example:
(1). The eta in Figure 6 do not have clear effect on its expected effect as the author stated, this fact also applies for rho, it is hard to clearly see the contribution factor. The results are not well analyzed yet, it is still not so clear how the proposed methods give better results than other compared methods, and the abaltion study is not so informative and supportive.
(2). Most active learning methods have a large variance in Figure 7, there is not clear pattern that the proposed method gives better results.
3. AUC-ROC is not a so good class-imbalance evaluation metric. AUC-PRC is more suitable for class balance classification.
4. More recent cost-effective mutli-label active learning method that use sample-label pair selection are omitted. For your information, some of them are as follows:
(1). Multi-instance multi-label active learning. IJCAI 2017.
(2). Multi-label active learning by model guided distribution matching. Frontiers Comput. Sci, 2016.
(3). Transfer Learning with Active Queries from Source Domain. IJCAI 2016
No
We would like to thank the reviewer for the valuable and thoughtful
comments. We summarize our response as follows:
**Q1: relationship with related methods that used sample-label pairs for
multi-label active learning.**
We thank the reviewer for this suggestion and for providing some related
works. We will cite and discuss these works.
Meanwhile, we would also
like to highlight some fundamental differences from the proposed model.
First, as mentioned in the abstract of our paper, the proposed work
focuses on addressing a relatively large label space. This is also
reflected in the real-world datasets used in our experiments, most of
which have more than 100 labels.
While a large label space poses a key
challenge, it also allows the proposed model to fully leverage the
potential label correlations by exploring a large and diverse set of
labels to clearly demonstrate its novelty and effectiveness.
Meanwhile,
since models that use sample-label pairs as queries need to consider the
potential combination of a data instance with all labels, a large label
space will introduce significant overhead for both active sampling and
human annotators. After checking the related works suggested by the
reviewer, it turns out that these works usually consider a much smaller
label space (around 20 labels), which confirms our analysis above and is
more suitable to use sample-label pair queries.
Second, while these
related works also developed active learning models, they are either
proposed for multi-instance learning (reference 1) or transfer learning
(references 2 and 3). We will update the revised paper to include a
discussion of these works.
Meanwhile, we would like to clarify that
multi-instance learning has a special assumption that data instances are
organized into bags, and there is a special relationship between
instance-level and bag-level labels. Transfer learning assumes that
training data and unlabelled/testing data are from different
distributions. Those assumptions do not hold in our setting, which
focuses on developing a general active learning model from a large and
sparse label space.
**Q2: experimental results are not so well analyzed.**
First, we would like to clarify that we expect the proposed model not to
be very sensitive to some specific hyper-parameters (e.g., $\eta$ and
$\rho$). The primary reason is that since we are fundamentally limited
by the availability of labels in active learning, it is usually infeasible to
conduct systematic cross-validation to choose a highly sensitive
hyper-parameter. Our experiments helped verify this behavior of our
model.
In particular, different $\eta$ leads to only slightly different
model performances. This is because the proposed model automatically
balances between the feature variance (second term in the sampling
function) and label covariance (first term in the sampling function) as
more labeled data samples are added to the training set. The underlying
reason is that feature variance will continue to decrease as more data
instances are exposed to the GP, whose predictive variance will continue
to decrease. Robustness wrt $\rho$ benefits from Bayesian model
selection. The significant components are automatically assigned much
larger 'effective posterior observations', which are sufficient to make
them selected by the model without much help from $\rho$.
In summary,
both $\eta$ and $\rho$ only provide a supporting role, and the model can
automatically adjust the sampling function and select the significant
mixture components mostly due to the Bayesian nature of the model (i.e.,
GP on the feature side and Bayesian Bernoulli mixture on the label
side).
**Q3: provide an alternative evaluation metric to AUC.**
In addition to the AUC-ROC scores as reported in the main paper, we also
conducted a fine-grained analysis on the label-wise performance, which
helps to achieve a deeper insight on the performance gain. In
particular, we compute the average precision (equivalent AUC-PRC as
suggested by the reviewer) improvement rate (referred to as API) of
individual labels. Figure 10 in Appendix D.3 shows that GP-B$^2$M
performs significantly better than BRMs on labels with a moderate or low
frequency, which further confirms the overall good performance of
GP-B$^2$M as shown in the main paper.
ORIGINALITY
The model is a Bayesian model composed of Gaussian latent variables and a finite mixture of
of multi-label clusters. The Gaussian latent variables are use to estimate the probabilities
of items being generated by specific components of the multi-label mixture. In this sense, this model
looks like an expert mixture model. The authors should comment on this similarity.
The active learning is done by selecting samples that show two kinds of uncertainty in the classification:
label uncertainty, measure by a proxy of label entropy, and classification uncertainty, measured by assignment probability
variance.
SIGNIFICANCE
Active learning is developed for the given model.
However, I think that the objective function used to
do active learning could be used for any other probabilistic model for multi-labeling.
Lines 46 to 53, in this paragraph, several critiques are given to existing methods for active learning:
pairwise correlation, full correlation (??) on subset of labels; latent embedding requires a large number of training labels;
latent embedding has no semantic meaning.
However, the model presented in this paper introduces latent embeddings (through Gaussian distributions), and
uses pairwise label correlations (correlations are between pairs of labels) in a covariance matrix.
How is then the proposed model addressing the limitations mentioned above?
Lines 77 to 91, the paper mentions "label correlations" many times. I would think that the right way of thinking about
the relation among labels is "label dependencies". It is a bit strange to talk about
label correlations, when labels are nominal variables.
At first, I thought "label dependencies" is what the authors meant;
but by looking
at the final output, I see that all what is measured is label correlations.
QUALITY
A drawback from the proposed method is its computational cost, which following the authors' claim is of
order O( [K= number of components] x [N= number of initial data]^3 ). How does this compare with the five
alternative methods used in the experiments?
The paper mention a Gaussian process, but these are just Gaussian random variables associated to
mixture components. The process is hidden in the covariance matrix which is a Gram matrix from a kernel.
But not much is said about this kernel. How is this kernel chosen?
Also, it is not clear if the Gram matrix is evaluated in the covariates or the points?
Line 123, it is stated that "A principled sampling function is introduced in the end for cost-effective ML-AL"
Why is that function "cost-effective". Please justify your claims.
Line 140, it is stated "One key innovation lies in using the latent indicator variables Z..."
This is far from innovative; this is the usual manner of dealing with finite mixtures.
About variational inference: Why putting so much emphasis on using conjugate priors or hyper-prior (by augmenting the data) if at the end (line 220) you still cannot obtain analytical expressions for the predictive posterior?
About time complexity (lines 209 to 211), O(K N^3) is huge, even if using parallelization, O(N^3) is huge.
Could you display the training time of the methods compared in Section 4?
Also, one of the advantages of using Bayesian models is the shrinking produced by sharing data in a hierarchical model.
The claim that the algorithm may be parallelized would indicate that not all data information is being passed over the
whole model. Some comments on this should be added.
CLARITY
The Bayesian model is presented equation by equation. This makes it hard to understand.
The authors should describe the model clearly as it is usually done for Bayesian models,
specifying the different distributions associated with the different variables in a hierarchical manner.
Page 2, there is a nice try to show a simple example of the multiple label problem. Unfortunately, the example
is not very clear. I would have preferred the authors show an example based on real labels and correlations between labels.
This would make the problem, and perhaps the model as well,  more interesting and intuitive.
Line 82, point (i) What do you mean with "ensuring consistency"? What consistency are you referring to?
Line 85, point (ii) What do you mean with "calibration" and "automatic calibration" here?
Lines 87 to 89, it is a bit too early to talk about "label covariance" in the introduction without having seen the model.
Also, What is a "correlated label space"
Lines 107 to 115, I feel the competitor models are being unfairly criticized. I agree that CBM is different from the proposed model, but that does not makes it necessarily "unsuitable" for active learning. There is always the possibility of using some measure of uncertainty to choose samples, given a model.
Also, the critique to CS for using Gaussian latent embedding seems odd, since
your own model uses Gaussian latent embeddings. You said that this "violates the binary nature of the labels";
but many statistical models uses this technique for modeling categorical data by means of thresholds.
Also, Why is CS "less efficient" for active learning, say than the proposed model?
Section 3.1, first paragraph, the model is being introduced here, but it is not very clear.
For example, the components
of the mixture (which are supposed to be distributions) are being confused with the parameters of the components {theta_k}.
Then in line 133, one sees p(y) = sum_k pi_nk theta_nk.
First, the notation p(y) has never been defined in the paper;
secondly, sum_k pi_nk theta_nk is just a weighted average of the parameters. My guess is that p(y) is the mixture
distribution of y, and that the sum should be over the distributions, something like sum_k pi_nk p( y | theta_nk),
where p( y | theta_nk ) is a product of independent Bernoulli distributions with success probabilities given by
the vector theta_nk.
Equation (1), Where is the role of X in these equations?
You write p(f^(k) |  X) = N( f^(k) |  0, Sigma_k ). I guess one should write Sigma_k (X).
Also, you have to define the notation N( | 0, Sigma), i.e., say explicitly that this is a multivariate normal (of what dimension?).
Line 160, What is the "non-conjugate mapping function"?
Line 166, Please define "VI". I guess this stands for variational inference.
However, in this sentence it might be
better to talk about variational estimation instead of variational inference.
Equation (12), I guess on the first line of this equation, the equality sign should be changed to approximation sign, since
the right-hand-side seems to be using the variational approximation q().
In the second line, Where did the variable Y go in the integrand?
Lines 235, 236, it says "By assigning a low variance to data samples near to the training data and a high variance to faraway samples, the predictive variance effectively captures the feature uncertainty..."
Who assigns these values to the samples? Are you just stating that this is what your model does? If so, Where is the argument and the justification for this claim?
Line 316, Where is the quantity "rho" defined? Could you show the equation where it is defined?
OTHER
Line 24, change "labels." to "label."
Line 112, change "the labels are further assumed to draw from a Gaussian to ensure..." to
"the labels are further assumed to BE drawN from a Gaussian DISTRIBUTION to ensure..."
Lines 116 to 118, the last two line of this sentence are repeated from point (ii) above in Page 3.
Line 133, I would suggest changing "the final label vector for x_n is formed by" by
"the final label vector associated with x_n is drawn from the mixture distribution"
Line 165, the notation p(x) has already been used before for p(y).
Lines 253, 254, I still do not quite understand this example, so I would avoid saying that the result is "highly interpretable".
Line 260, What are the symbols &! representing here?
Figure 7, the caption and curves are too small to be seen clearly.
REBUTTAL
I have read the authors' rebuttal and the other colleagues' reviews. I thank the authors for trying to clarify some issues.
However, the answers have not made me change my mind.
In the first place, there were not too many issues in the original manuscript.
I find the proposed method interesting but overly complex, involving many parameters, and hyper-parameters, but, still producing very similar results to other competing methods.
No
We would like to thank the reviewer for the valuable comments.
**Q1: connection to expert mixture model.**
We appreciate this insight from the reviewer. By integrating GPs with
label clusters resulted from the B$^2$M model, the proposed model
essentially partitions the problem space into $k$ regions, each
corresponding to a label cluster. In this regard, each GP acts as an
expert machine and the gating is achieved by the logistic-softmax
function given in Eq. (4). We will include this connection in model
interpretation in the revised paper. We also want to emphasize that
achieving a unified model that seamlessly integrates data features with
label clusters to discover more accurate mixture components to capture
label correlations from a very sparse label space is both novel and
highly nontrivial. To this end, we propose a unique end-to-end fully
conjugate posterior inference process to learn accurate and
interpretable mixture components of labels. A novel sampling function is
also designed by combining feature uncertainty and label covariance,
both of which can be obtained from the predictive distribution of our
model.
**Q2: the objective function.**
While the latent variables and their dependencies are unique to the
proposed model, the overall objective function given in Eq. (2) is a
general one for probabilistic models, which aims to maximize the (log)
marginal likelihood of the observed labels.
Thus, it can be used for
other probabilistic models for multi-label classification, where the key
latent variables need to be changed based on the chosen model. We would
also like to clarify that the proposed model is designed specifically
for the AL setting, where the labeled data samples are limited.
Hence,
the available labels are much more sparse than the general multi-label
learning problems. To tackle the high label scarcity, the proposed model
is uniquely designed to discover meaningful label correlations from
limited labels by encoding the inductive bias from data features as
Bayesian priors to learn from both labels and features.
**Q3: address limitation of existing models**
In our model, the latent embedding is performed on the feature side
(instead of labels) through GP kernels. The label correlations, on the
other hand, are captured through the B$^2$M model, which results in a
set of label clusters that encode the co-occurrence relationship among
labels. Each mixture component can be conveniently interpreted by
checking the most probable labels that co-occur within the component
(see Figure 1.c). By integrating the GP with the B$^2$M model, data
features are used along with limited labels to more accurately discover
meaningful label correlations, which is more advantageous than existing
models.
**Q4: label correlations vs label dependencies.**
Based on the standard definition in statistics: \"correlation or
dependence is any statistical relationship, whether causal or not,
between two random variables or bivariate data. In the broadest sense,
correlation is any statistical association, though it commonly refers to
the degree to which a pair of variables are linearly related.\" To avoid
the confusion that correlation is commonly used to refer to the linear
relationship as the reviewer may indicate, we will clarify this in the
revised paper.
**Q5: computational cost.**
Under the typical active learning setting, the number of labeled data
samples available for model training is usually small, as the goal is to
minimize the annotation cost. In our experiments, $N$ is set to be less
than 600 (initial pool plus 500 AL selected samples) in all real
datasets to mimic common AL scenarios.
As a result, computational cost
is not a major concern. We report the execution time (in seconds) for a
complete active sampling iteration in Table 4 of Appendix D.6. The
comparison shows that the proposed model is more efficient than most
baselines. It is also worth noting that for a larger $N$ value, we can
further leverage sparse kernel machines (e.g., Sparse GP), to reduce the
complexity.
**Q6: kernel selection.**
One main reason for choosing GP to predict the mixture weights is to
give the proposed method flexibility to deal with AL problems in
different domains. Being a Bayesian model, we can optimize the kernel
parameters through type-II maximum likelihood. In our experiments, we
find a standard RBF kernel with a fixed length scale provides
sufficiently good performance in all datasets.
**Q7: cost-effective property of the proposed method.**
The cost refers to data annotation cost, which can be pretty high,
especially in specialized domains (e.g., medicine, bioinformatics,
etc.). This is the major objective that active learning models aim to
minimize.
**Q8: latent indicator variables $Z$ is the usual manner of dealing
with finite mixtures.**
We would like to clarify that the novelty of latent variables $Z$ is not
because they are indicators for the mixture component assignments.
Instead, the novelty is to use them to bridge between the feature space
and the label space. In essence, by introducing latent variables $Z$, we
convert a multi-label problem into a multi-class problem. The posterior
inference is also designed to overcome the non-conjugacy introduced by
the mapping function defined on $Z$ in order to integrate features and
labels.
**Q9: emphasis on using conjugate priors**
Choosing conjugate priors will ensure that the posteriors take the same
form as the prior distributions so that the posterior inference can be
conducted much more efficiently.
While the predictive distribution in
Eq. (12) cannot be analytically evaluated, it is important to note that
having an analytical (approximate) posterior $q(F)$ allows us to
conveniently sample from it to efficiently evaluate the predictive
distribution.
**Q10: algorithm may be parallelized**
Running the algorithm in parallel benefits from the proposed auxiliary
variables based variational inference, which fully factorizes among the
$k$ mixture components, as indicated by Eq. (27) in Appendix B.
**Q11: consistency in line 82.**
Consistency refers to the assignment of a data instance to the mixture
component from the feature side (through GP) and the label side (through
the Bernoulli mixture-based label clustering) should agree with each
other.
**Q12: calibration in Line 85.**
The proposed Bayesian Bernoulli mixture model can automatically
determine the number of mixture components, which has the effect of
calibrating the overall model complexity to fit the data.
**Q13: correlated label space?**
It refers to the original label space, where many labels may co-occur
with each other, indicating that seeing one such label implies a higher
chance of seeing another. We will make this clear in the revised paper.
**Q14: competitor models, including CBM and CS.**
Regarding CBM, we agree with the reviewer that it is possible to use
some uncertainty measure to choose samples given a model, and we provide
one such example (ref \[18\]) in our related work section.
However, it
is also important for such a model to perform well when the labeled data
is very limited so that uncertainty can be accurately measured for
active sampling. As we reported in Table 3 of Appendix D, CBM cannot
perform well when the labeled data is limited, which is usually more than 10% worse than our model. Data sampling based on such an inaccurate
model is much less effective.
As for the CS model, embedding is
performed on both the label side (through compressed sensing) and the
feature side (through GP kernels), which is different from our model.
CS-based model is less efficient for active learning because it requires
additional steps to recover from the latent label space. In Table 4 of
Appendix D, we show that our model is consistently more efficient than
the CS-based model.
**Q15: Model presentation**
Thank you for the suggestion. In the formulation, $p({\bf y}_n)$ is the
marginal distribution of ${\bf y}_n$ and we skip the mixture conditional
distribution
$p({\bf y}_n={\bf 1}|\boldsymbol{\theta}_k)=\boldsymbol{\theta}_k$,
which causes the confusion. We will update this to make it clear.
**Q16: role of $X$ in Eq (1); non-conjugate mapping function in line
106; VI in line 166; explain Eq. (12)**
Here, we follow the conventional definition of a Gaussian prior, where
${\bf X}$ is used to construct the covariance matrix of the
multivariate Gaussian. This implies that the dimension of the prior is
$N$, which we also explicitly noted previously by writing out
$\textbf{f}^{(k)}$. We will clarify this. The mapping function refers to
$h$ in Eq (1). As mentioned in line 156, the typical form of this
function is softmax, which is non-conjugate to the Gaussian prior. The
abbreviation for VI is first defined in line 155. We will clarify this.
As for Eq. (12), the reviewer is correct in both cases. It should be an
approximation, and we will add ${\bf Y}$ in the integrand.
**Q17: the claim in Line 235.**
In line 235, we are explaining the active sampling behavior of the
proposed model. Assigning a low variance to data samples near the
training data and a high variance to faraway samples is the property of
the predicted variance of a GP, which can be verified by checking the
analytical form of the predictive covariance matrix. We will further
clarify this in the revised paper.
**Q18: rho in line 316**
$\rho$ is first introduced in Line 329 and further explained in lines 330-333.
We will also follow other suggestions provided by the reviewer to
further improve the presentation of the revised paper.
The motivation for the proposed methods is well-explained and the paper is well-written overall.
The paper addresses multi-label active learning by combining a Bayesian Bernoulli mixture of label clusters and the Gaussian Process that predicts coefficients of mixture components. The predictive distribution could provide feature uncertainty and label covariance for the sampling function. The proposed methods are theoretically sound.
Experiments on the synthetic and real multi-label data demonstrate the strength of the model. The label-wise improvement in the appendix shows that the model performs better than BRMs on labels with a lower frequency. The authors argue that this is because tail labels do not have sufficient examples for BRMs and the proposed model could use the label correlations.
It should be noted that the AP deterioration of the proposed model on tail labels is also more significant. This could be worth further investigation.
**Typo:**
- Line 24  more than one label
No
We would like to thank the reviewer for the valuable comments.
**Q1: Explain the AP deterioration on tail labels.**
The reviewer might refer to the result in Figure 10 from the Appendix.
As mentioned, the proposed method performs better than BRMs and other
baselines on labels with a moderate or relatively low frequency. This is
because the proposed model leverages the label correlation in addition
to features to make the prediction.
But AP deterioration is still
possible as not all relatively rare labels may exhibit correlations with
other labels.

Meta-review:
role: PC
data: comment: The submission considers the problem of active learning for multi-label classification, and propose a two phase Bayesian approach that takes label correlation into account.

Three reviewers carefully considered the submission, and found several weaknesses in the presentation and proposed idea. In particular, reviewer iRRM provided a detailed set of suggestions to the authors for improving the paper. The author rebuttal was considered by all the reviewers, and its contents was briefly discussed post rebuttal. The reviewers all agreed after considering each others' reviews as well as the rebuttal that there were remaining issues with the submission. Unfortunately, due to the large number of high quality submissions at NeurIPS, this submission cannot be accepted in its current state.

Decision: Accept (Poster)