Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed Recurrent Bayesian Classifier Chains (RBCC) for exact multi-label classification. The proposed method learned a Bayesian network of class dependencies to improve the Recurrent Classifier Chains (RCCs). The comparative experiments were performed with RCC ([15], 2017, Nam, etc.), TS-RCC ([15], 2017, Nam, etc.), OF-RCC ([4], 2018 , Chen, etc.), and BCC ([23], 2014, Sucar). And the results demonstrated the effectiveness of the RBCC method.
main_review: - The RBCC incorporates a Bayesian network into the recurrent classifier chain system.
- A new algorithm is proposed for training and inference.
- Experiments show that this method improves the performance of the mentioned methods.

limitations_and_societal_impact: ## Weakness:

- In equation 1:
The left side of equation is a function of x, while the right side is the expectation on x, that means the right side is irrelevant to x.

- line 16, ‘we outperform the state of the art methods’
More experiments should be conducted to compare with some latest methods proposed in 2019-2020.

- page 5-6, Table 1-3
The improvement of this method is marginal.

## Minor issues: 
	- line 100:
	The reference [22] needs to be mentioned only once.

## Suggestion:
- Some figures could be demonstrated to help the reader to understand the method and the advantage of the method, such as, the algorithm flow chart and the resulting figure which can illustrate the superiority of the proposed algorithm.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose Recurrent Bayesian Classifier Chains (RBCCs), a new method for exact multi-label classification. It combines the idea of ordering and conditioning labels based on a Bayesian network, utilized previously in Bayesian Classifier Chains (BCCs), and using a recurrent neural network as in Recurrent Classifier Chain (RCC). Usage of recurrent architecture allows modeling that condition on previously predicted classes instead of using many separate binary models originally used in the Classifiers Chain. The proposed approach uses an established approach to construct a Bayesian network. The novelty of the approach lies in the training and inference procedure. Instead of performing many to many classification, where labels are predicted one after another as in the case of standard RCCs, in RBCCs, many to one classification is performed for each class. Each class is conditioned only on features and labels it depends on in the obtained Bayesian network. The author validates the attractiveness of their method in empirical study on three popular benchmark datasets with a small number of labels. In all three cases, the proposed approach significantly improves in terms of subset accuracy over the baselines.
main_review: Strengths:
+ ~~The organization of this paper is good, and it's easy to read and understand.~~
+ The proposed method seems to be an intuitive and neat improvement to BCCs/RCCs formula.
+ The RBCCs consistently outperform the benchmark in the empirical evaluation.
+ Comparison of different Bayesian network construction technique and their impact on the predictive performance of RBCs.
+ Comprehensive description of implementation details in the appendix.

Weaknesses:
- Empirical evaluation was conducted on only three datasets, all with a small number of labels - this is the only clear issue I have. With so many multi-label datasets from varying domains and a varying number of labels. I believe that a much more extensive empirical evaluation should be performed.

Other comments:
* The simplest baseline that directly optimizes argmax E[P(Y|x)] is the well-known labels power set method. Why not include it as a simple baseline, especially when comparing datasets with small labels?
* "If the parents have not yet been predicted, we recursively call the inference algorithm on the unpredicted parent classes." - this is not necessary since the Bayesian network is a DAG, you just need to predict classes in their topological order.
* There should be an equal sign instead of unequal in line 122.

---

Edit after the Authors' response: Since the authors addressed my main issue by conducting additional experiments, I'm raising my rating (to 6).

---

Edit after further discussion with AC and the Reviewers: Further discussion highlighted several issues with notation and clarity of the paper that led to differences in understanding of the proposed method among the Reviewers. After consideration, I think the paper requires major revision, and despite attractive results, should not be accepted to the conference. Because of that, I'm reverting my rating to the original one (5).
limitations_and_societal_impact: The limitations were addressed, no issues with negative social impact were found.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a new method for optimizing the subset zero-one loss in multi-label classification. The new method is an adaptation of probabilistic classifier chains. The central idea is to replace the chain by a dependence structure learned using Bayesian networks. In theory the order of the labels in a probabilistic classifier chain does not matter, but in practice one can observe performance drops if the chosen order leads to propagation of errors. By replacing a random order with the dependency structure of a Bayesian neural network, the authors intend to solve the problem of choosing an appropriate order. 

As underlying models to fit probabilities, recurrent neural networks are used. Experimental results are reported for three benchmark datasets. 
main_review: Overall this is a nicely-written paper that presents a novel idea. Indeed, it is the case that in practice the order of labels in a probabilistic classifier chain matters. Learning the dependency structure by means of a Bayesian network is therefore a useful thing to do. Intuitively, I have the feeling that the presented approach makes sense. On the other hand, the presented approach is not very novel, since in reference [30] something similar was done, albeit without a focus on subset zero-one loss and without recurrent neural networks as underlying models. However, I do believe there is enough novelty to justify a new paper on this topic. 

The existing literature is covered quite well, and in the experiments the choice of baselines is appropriate to my opinion. With only three datasets, the number of datasets in the experiments is quite small, especially because there are much more datasets in the public domain, see e.g. the MULAN datasets. I believe that it is important to justify why only those three datasets were analyzed in order to avoid being accused of cherry picking. I can imagine that those datasets were chosen because they have more complex feature spaces than the simple MULAN datasets, but it would be good to state that explicitly. 

At some points the mathematical notation is a bit sloppy. In general it is important to formally define all notations, while being consistent in the use of vectors, components of vectors, sets and random variables. An example of an awkward notation is line 159, where C_ki = g_k(x_i) + E_ki. The model g is here a classifier that probably outputs zero or one, but C_ki is a pointer to class k. It might be better to use the notation Y_i, as defined on page 2, and refer to the k-th entry in Y_i. Also in Lemma 1, the quantities E_1,...,E_L are used but they are not formally defined. Most likely they allude to the vectors that originate by grouping E_i,k for all i. Another example of an incorrect notation is Eqn. 1, because the expectation over X is not needed there. If you want to refer to parameter fitting, please parameterize the distribution P by theta and formulate the optimization problem as the minimization of the log loss over a training dataset. Eqn. 1 looks very awkward for the moment. 

I don't find the title very appropriate, and I would suggest to change it. When I downloaded the pdf, I thought the paper would be about a Bayesian variant of probabilistic classifier chains, in which a distribution of models is learned instead of a single model, similar to Bayesian methods such as Gaussian processes and Bayesian neural networks. However, the term Bayesian here comes from the fact that Bayesian networks are used to learn dependencies between labels. For example, a title like "Learning class dependencies for probabilistic classifier chains" or something similar would be more appropriate. 
   
limitations_and_societal_impact: They have a paragraph about this in the paper. No concerns from my side.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new method, Recurrent Bayesian Classifier Chains (RBCC), for exact multi-label classification problems (maximizing subset accuracy). RBCC involves three steps: 1. Intra-class Dependency Network to learn a Bayes net to the classes; 2. Recurrent Conditional Dependency Model to summarize the information of features and the corresponding parent classes learned in the previous step; 3. Bayesian-Conditioning Classifier to predict. RBCC is shown to outperform several state-of-the-art (STOA) algorithms on some real-world multi-label data sets with respect to subset accuracy.
main_review: Overall I think the method (RBCC) proposed in this paper is relatively novel and its performance is good. The description of the method is fairly clear, and enough background is provided. My main concern is about the experiments (see below). The following are my comments on each of the dimension.

Originality.
The method (RBCC) is new. Although each component is known, the combination is relatively novel, for example, how to better incorporate a Bayes net into the recurrent classifier chain system. The related work is adequate and some STOA algorithms are described and compared.

Quality.
The work is sound. The methods for designing RBCC are appropriate. However, there are some issues with the experimental design, which I list below.

1) Why some data sets (Reuters-21578, RCV1-v2 and BioASQ) in paper [15] are not used in the experiments? Both paper [15] and this paper aim to maximize subset accuracy, and methods in both papers are based on Recurrent Classifier Chain (RCC), so it is reasonably expected the authors should use one of the data sets from [15] or at least provide explanations for not using.

2) I might be missing something, but I could not find the number of repeated experiments in the paper, given that the error bars are reported.

3) Training time and inference time are not discussed. It looks that since RBCC has slightly more complicated structures than RCCs, it might take longer to train. Unfortunately this info is lack in the paper. The authors should discuss the computational costs.

Clarity.
Overall this paper is clear. Enough background is provided. The descriptions of the proposed method is detailed and clear. Nevertheless, some terms are not clearly defined and there are several typos (some of which are listed below). If this paper is accepted, the authors should proofread carefully, add necessary explanations and fix typos. Question regarding some terms.

1)  What is "non-noisy"? I can get a sense of the meaning after reading the whole paper, but I could not find where this is defined.

2) In the contributions, it is stated that a novel training algorithm is proposed. Which part of the paper does this "training algorithm" refer to?

Significance.
The method (RBCC) proposed in this work is useful for researchers and practitioners working on multi-label learning. It also advances STOA in exact multi-label classification problems.

---
Extra question (the authors do not need to respond):

1) Could the authors also report metric F1 (instance-averaged) in the experiments?
---
Typos:

Page 3, line 100, duplicate citations [22]

Page 3, line 113, "strongly dependent class" -> "strongly dependent on class"

Page 3, line 122, $P(C_k | x) \ne P(C_k | C_n, x)$ -> $P(C_k | x) = P(C_k | C_n, x)$

Page 5, line 187, $V = $ -> $V_t =$

Page 6, line 243, "in the onstruction" -> "in the construction"

---

After rebuttal: My main concerns (1 and 3 in the Quality section) are addressed. I keep my original rating for this paper.

---
Citations:
[15] Jinseok Nam, Eneldo Loza Mencía, Hyunwoo J Kim, and Johannes Fürnkranz. 2017. Maximizing subset accuracy with recurrent neural networks in multi-label classification. In Advances in Neural Information Processing Systems. 5413–5423.

limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The algorithm introduced by the authors is sound and obtains promising results as given in the experimental results. The main problems pointed out by reviewers are its clarity and readability. All those problems are mainly caused by the used notation and typos in equations. Nevertheless, the authors were very responsive, clarified almost all issues, and shared with reviewers intended changes to be incorporated into the final version of the paper. Taking this into account we lean toward acceptance of the paper. 

Additional comments:
- Please also correct text and equations in lines 217-218: I suppose it should be $[0, 1]^L$ instead of $\\{0,1\\}^L$; it is better to define a loss function using all its parameters (true labels and predicted probabilities). 
Decision: Accept (Poster)