Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a novel model for multi-label active learning that could learn label correlations from both limited labels and data features. This is achieved through a two-phase learning process that uses a Bayesian Bernoulli mixture of label clusters and the Gaussian Process to predict coefficients of mixture components. The paper also proposes a sampling function to consider both feature uncertainty and label covariance obtained by the model.
main_review: The motivation for the proposed methods is well-explained and the paper is well-written overall. 

The paper addresses multi-label active learning by combining a Bayesian Bernoulli mixture of label clusters and the Gaussian Process that predicts coefficients of mixture components. The predictive distribution could provide feature uncertainty and label covariance for the sampling function. The proposed methods are theoretically sound.

Experiments on the synthetic and real multi-label data demonstrate the strength of the model. The label-wise improvement in the appendix shows that the model performs better than BRMs on labels with a lower frequency. The authors argue that this is because tail labels do not have sufficient examples for BRMs and the proposed model could use the label correlations. It should be noted that the AP deterioration of the proposed model on tail labels is also more significant. This could be worth further investigation.

**Typo:**

- Line 24  more than one label
limitations_and_societal_impact:  The paper should not pose any negative societal impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: A Bayesian model for multi-label assignment is introduced.
The paper is about training the model with active learning, and this is its main goal.
Though, the model usefulness is prediction (classification).
Therefore, although not mentioned or not made clear in the manuscript, there are two goals in this paper:
present a new model for multi-label prediction, and present a new way to do active learning.
This second task is developed for the given model.

The model is a Bayesian model composed of Gaussian latent variables and a finite mixture of
of multi-label clusters. The Gaussian latent variables are use to estimate the probabilities
of items being generated by specific components of the multi-label mixture. In this sense, this model
looks like an expert mixture model.

The active learning is done by selecting samples that show two kinds of uncertainty in the classification:
label uncertainty, measure by a proxy of label entropy, and classification uncertainty, measured by assignment probability
variance.

The performance of the model is compared with five other alternative models.
The proposed model performs well, presenting high efficacy early on the sampling;
by the time 500 active learning samples have been shown to the methods, one can see
that other methods may perform slightly better, but still comparable to the proposed method.

main_review: 
ORIGINALITY

The model is a Bayesian model composed of Gaussian latent variables and a finite mixture of
of multi-label clusters. The Gaussian latent variables are use to estimate the probabilities
of items being generated by specific components of the multi-label mixture. In this sense, this model
looks like an expert mixture model. The authors should comment on this similarity.

The active learning is done by selecting samples that show two kinds of uncertainty in the classification:
label uncertainty, measure by a proxy of label entropy, and classification uncertainty, measured by assignment probability
variance.


SIGNIFICANCE

Active learning is developed for the given model. However, I think that the objective function used to
do active learning could be used for any other probabilistic model for multi-labeling.

Lines 46 to 53, in this paragraph, several critiques are given to existing methods for active learning:
pairwise correlation, full correlation (??) on subset of labels; latent embedding requires a large number of training labels;
latent embedding has no semantic meaning.
However, the model presented in this paper introduces latent embeddings (through Gaussian distributions), and
uses pairwise label correlations (correlations are between pairs of labels) in a covariance matrix.
How is then the proposed model addressing the limitations mentioned above?

Lines 77 to 91, the paper mentions "label correlations" many times. I would think that the right way of thinking about
the relation among labels is "label dependencies". It is a bit strange to talk about
label correlations, when labels are nominal variables.
At first, I thought "label dependencies" is what the authors meant; but by looking
at the final output, I see that all what is measured is label correlations.


QUALITY

A drawback from the proposed method is its computational cost, which following the authors' claim is of
order O( [K= number of components] x [N= number of initial data]^3 ). How does this compare with the five
alternative methods used in the experiments?

The paper mention a Gaussian process, but these are just Gaussian random variables associated to
mixture components. The process is hidden in the covariance matrix which is a Gram matrix from a kernel.
But not much is said about this kernel. How is this kernel chosen?
Also, it is not clear if the Gram matrix is evaluated in the covariates or the points?

Line 123, it is stated that "A principled sampling function is introduced in the end for cost-effective ML-AL"
Why is that function "cost-effective". Please justify your claims.

Line 140, it is stated "One key innovation lies in using the latent indicator variables Z..."
This is far from innovative; this is the usual manner of dealing with finite mixtures.

About variational inference: Why putting so much emphasis on using conjugate priors or hyper-prior (by augmenting the data) if at the end (line 220) you still cannot obtain analytical expressions for the predictive posterior?

About time complexity (lines 209 to 211), O(K N^3) is huge, even if using parallelization, O(N^3) is huge.
Could you display the training time of the methods compared in Section 4?
Also, one of the advantages of using Bayesian models is the shrinking produced by sharing data in a hierarchical model.
The claim that the algorithm may be parallelized would indicate that not all data information is being passed over the
whole model. Some comments on this should be added.


CLARITY

The Bayesian model is presented equation by equation. This makes it hard to understand.
The authors should describe the model clearly as it is usually done for Bayesian models,
specifying the different distributions associated with the different variables in a hierarchical manner.

Page 2, there is a nice try to show a simple example of the multiple label problem. Unfortunately, the example
is not very clear. I would have preferred the authors show an example based on real labels and correlations between labels.
This would make the problem, and perhaps the model as well,  more interesting and intuitive.

Line 82, point (i) What do you mean with "ensuring consistency"? What consistency are you referring to?

Line 85, point (ii) What do you mean with "calibration" and "automatic calibration" here?

Lines 87 to 89, it is a bit too early to talk about "label covariance" in the introduction without having seen the model.
Also, What is a "correlated label space"

Lines 107 to 115, I feel the competitor models are being unfairly criticized. I agree that CBM is different from the proposed model, but that does not makes it necessarily "unsuitable" for active learning. There is always the possibility of using some measure of uncertainty to choose samples, given a model. Also, the critique to CS for using Gaussian latent embedding seems odd, since
your own model uses Gaussian latent embeddings. You said that this "violates the binary nature of the labels"; but many statistical models uses this technique for modeling categorical data by means of thresholds.
Also, Why is CS "less efficient" for active learning, say than the proposed model?

Section 3.1, first paragraph, the model is being introduced here, but it is not very clear. For example, the components
of the mixture (which are supposed to be distributions) are being confused with the parameters of the components {theta_k}.
Then in line 133, one sees p(y) = sum_k pi_nk theta_nk. First, the notation p(y) has never been defined in the paper;
secondly, sum_k pi_nk theta_nk is just a weighted average of the parameters. My guess is that p(y) is the mixture
distribution of y, and that the sum should be over the distributions, something like sum_k pi_nk p( y | theta_nk),
where p( y | theta_nk ) is a product of independent Bernoulli distributions with success probabilities given by
the vector theta_nk.

Equation (1), Where is the role of X in these equations?
You write p(f^(k) |  X) = N( f^(k) |  0, Sigma_k ). I guess one should write Sigma_k (X).
Also, you have to define the notation N( | 0, Sigma), i.e., say explicitly that this is a multivariate normal (of what dimension?).

Line 160, What is the "non-conjugate mapping function"?

Line 166, Please define "VI". I guess this stands for variational inference. However, in this sentence it might be
better to talk about variational estimation instead of variational inference.

Equation (12), I guess on the first line of this equation, the equality sign should be changed to approximation sign, since
the right-hand-side seems to be using the variational approximation q().
In the second line, Where did the variable Y go in the integrand?

Lines 235, 236, it says "By assigning a low variance to data samples near to the training data and a high variance to faraway samples, the predictive variance effectively captures the feature uncertainty..."
Who assigns these values to the samples? Are you just stating that this is what your model does? If so, Where is the argument and the justification for this claim?

Line 316, Where is the quantity "rho" defined? Could you show the equation where it is defined?




OTHER

Line 24, change "labels." to "label."

Line 112, change "the labels are further assumed to draw from a Gaussian to ensure..." to
"the labels are further assumed to BE drawN from a Gaussian DISTRIBUTION to ensure..."

Lines 116 to 118, the last two line of this sentence are repeated from point (ii) above in Page 3.

Line 133, I would suggest changing "the final label vector for x_n is formed by" by
"the final label vector associated with x_n is drawn from the mixture distribution"

Line 165, the notation p(x) has already been used before for p(y).

Lines 253, 254, I still do not quite understand this example, so I would avoid saying that the result is "highly interpretable".

Line 260, What are the symbols &! representing here?

Figure 7, the caption and curves are too small to be seen clearly.


REBUTTAL

I have read the authors' rebuttal and the other colleagues' reviews. I thank the authors for trying to clarify some issues.
However, the answers have not made me change my mind. In the first place, there were not too many issues in the original manuscript.
I find the proposed method interesting but overly complex, involving many parameters, and hyper-parameters, but, still producing very similar results to other competing methods.


limitations_and_societal_impact: 
SOCIETAL IMPACT
None mentioned in the paper. But they are describe in the Checklist.

LIMITATIONS
None mentioned in the paper. But they are describe in the Checklist.

Both Societal impact and Limitations should be moved to the main paper.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper introduces a Gaussian Process Bayes Bernoulli mixture model based solution for cost-effective multi-label active learning. The proposed graphical model can account for diverse label correlations by mixture components and by the feature-component-label mapping. The proposed method gives better results than other compared methods and show the Bernoulli mixture model can better capture the label correlations.
main_review: Multi-label active learning have been popularly studied in recent years and many solutions have been proposed, including the graphical model based ones. Although the authors highlight the distinction of the proposed method and clarify the difference of the proposed one to related ones. In addition, they also show better results than other compared methods. I still have some comments for the authors:
1. There are related methods that used sample-label pairs that can make use of label correlations for multi-label active learning and to save the query the cost, but the authors do not well recognize them and criticize there are few works consider the label correlations for multi-label active learning, which downgrade the technical Quality and Originality.
2. The technical Clarity is clear but the experimental results are not so well analyzed. For example:
(1). The eta in Figure 6 do not have clear effect on its expected effect as the author stated, this fact also applies for rho, it is hard to clearly see the contribution factor. The results are not well analyzed yet, it is still not so clear how the proposed methods give better results than other compared methods, and the abaltion study is not so informative and supportive.
(2). Most active learning methods have a large variance in Figure 7, there is not clear pattern that the proposed method gives better results.
3. AUC-ROC is not a so good class-imbalance evaluation metric. AUC-PRC is more suitable for class balance classification.
4. More recent cost-effective mutli-label active learning method that use sample-label pair selection are omitted. For your information, some of them are as follows:
(1). Multi-instance multi-label active learning. IJCAI 2017.
(2). Multi-label active learning by model guided distribution matching. Frontiers Comput. Sci, 2016.
(3). Transfer Learning with Active Queries from Source Domain. IJCAI 2016


limitations_and_societal_impact: Yes
review_title: null

Meta-review: The submission considers the problem of active learning for multi-label classification, and propose a two phase Bayesian approach that takes label correlation into account.

Three reviewers carefully considered the submission, and found several weaknesses in the presentation and proposed idea. In particular, reviewer iRRM provided a detailed set of suggestions to the authors for improving the paper. The author rebuttal was considered by all the reviewers, and its contents was briefly discussed post rebuttal. The reviewers all agreed after considering each others' reviews as well as the rebuttal that there were remaining issues with the submission. Unfortunately, due to the large number of high quality submissions at NeurIPS, this submission cannot be accepted in its current state.
Decision: Accept (Poster)