Recurrent Bayesian Classifier Chains for Exact Multi-Label Classification
Authors: Walter Gerych
Thomas Hartvigsen
Luke Buquicchio
Emmanuel Agu
Elke Rundensteiner
Venue: NeurIPS 2021 Poster (2021)
Keywords: multilabel classification
recurrent classifier chains
classifier chains
recurrent networks
exact classification
subset accuracy

Abstract: Exact multi-label classification is the task of assigning each datapoint a set of class labels such that the assigned set exactly matches the ground truth. Optimizing for exact multi-label classification is important in domains where missing a single label can be especially costly, such as in object detection for autonomous vehicles or symptom classification for disease diagnosis. Recurrent Classifier Chains (RCCs), a recurrent neural network extension of ensemble-based classifier chains, are the state-of-the-art exact multi-label classification method for maximizing subset accuracy. However, RCCs iteratively predict classes with an unprincipled ordering, and therefore indiscriminately condition class probabilities. These disadvantages make RCCs prone to predicting inaccurate label sets. In this work we propose Recurrent Bayesian Classifier Chains (RBCCs), which learn a Bayesian network of class dependencies and leverage this network in order to condition the prediction of child nodes only on their parents. By conditioning predictions in this way, we perform principled and non-noisy class prediction.  We demonstrate the effectiveness of our RBCC method on a variety of real-world multi-label datasets, where we routinely outperform the state of the art methods for exact multi-label classification. 

Recurrent Bayesian Classifier Chains for Exact Multi-Label Classification
Walter Gerych, Thomas Hartvigsen, Luke Buquicchio, Emmanuel Agu, Elke Rundensteiner Worcester Polytechnic Institute
Worcester, MA
(ugerych, twhartvigsen, 1jbuquicchio, emmanuel, nundenst)@upi.coom edu
Abstract
Exact multi-label classification is the task of assigning each datapoint set of class labels such that the assigned set exactly matches the ground truth. Optimizing for exact multi-label classification is important in domains where missing a single label can be especially costly, such as in object detection for autonomous vehicles or symptom classification for disease diagnosis. Recurrent Classifier Chains (RCCs), a recurrent neural network extension ensemble-based classifier chains, are the state- of-the-art exact multi-label classification method for maximizing subset accuracy. However, RCCs iteratively predict classes with an unprincipled ordering, and therefore indiscriminately condition class probabilities. These disadvantages make RCCs prone to predicting inaccurate label sets. In this work we propose Recurrent Bayesian Classifier Chains (RBCCs), which learn Bayesian network of class dependencies and leverage this network in order to condition the prediction child nodes only their parents. By conditioning predictions in this way, perform principled and non-noisy class prediction. We demonstrate the effectiveness our RBCC method variety of real-wor world multi-label datasets, where routinely outperform the state of the art methods for exact multi-label classification.
Introduction
Exact multi-label classification (E-MLC) is the task of assigning multiple labels single instance, and is critical for variety of of domains. For example, when detecting health conditions, one person will often experience multiple symptoms concurrently. Additionally, missing the detection of even one symptom can have drastic effects and so the entire label set must be predicted exactly correct. Missing the symptom Loss of Taste, for instance, might cause downstream classifier misdiagnose COVID-19 as the common cold (3. While explicitly optimizing for exact classification is inordinately challenging, a successfulll-traanee model highly rewarding as guaranteed to capture the joint probabilities between the features and all labels.
There has been increasing interest in developing multi-label classification methods for E-MLC 19 [8 13 28 15 224]. Most are based on classifier chains, which are ensembles of models where each classifier predicts single class and the predictions from the i-th model are used as input into the i + 1-th model in order to learn inter-class dependencies. As the classifier chain requires an ordering of the classes a priori 19 there are variations of classifier chains that utilize different class orderings [1] [6], such as Bayesian classifier chains (BCCs) 30 23 which utilize Bayesian network to learn class orderings. key disadvantage of these classier chain approaches that they all require ensemble of individual classifiers both requiring large number of parameters and learning different and independent latent representation of the input data per classifier. This makes such ensemble methods particularly prone to overfitting (00]
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Recurrent Classifier Chains
Recurrent Bayesian Classifier Chains
Learned Bayesian Network
Reset Memory
Reset Memory
CIF
CIF
ClF
R
cif
Cif
Cff
CIF
Cif
Cx
%
Ad boc class orderinn
Figure 1: A comparison of the difference between Recurrent Classifier Chains and our proposed
Recurrent Bayesian Classifier Chains.
The leading extension of classifier chains is the recurrent classifier chain (RCC), which models inter-class dependencies by using a recurrent network to iteratively predict each class, such that each class prediction is conditioned on the predictions of all previously predicted classes. The
order in which classes are predicted is likewise important for RCCs [19 15
To address this, the
ordering usually ad hoc with popular choices being predict classes either frequent-to-rare or rare-to-frequent [5] while some very recent RCCs attempt to learn an ordering for the classes 4 244.
However, there is no direct feedback on the optimal class ordering, and attempting to learn through reinforcement alone is a challenging task that is also prone to overfitting [29]. Additionally, RCCs inherently condition their class predictions on noisy inputs, since the predictions are conditioned on
all previously preedicced classes even when most classes are independent of all other classes.
The task of E-MLCC is to train model such that f can predict the exact label sets of previously- unseen instances. Specifically, our goal is to use training data to obtain a classifier that can maximize the Subset Accuracy of unseen testing data, the label set most likely to be exactly correct the risk minimizer for Subset Accuracy. Thus, maximizing Subset Accuracy directly corresponds to optimizing for E-MLC. Subset Accuracy is a strict multi-label metric that returns if the entire label set is correct, and 0 otherwise.
There are four major challenges for E-MLC:
Critical label dependencies. Inter-class dependencies must be learned and modeled in order to perform E-MLC (7].
Cascading class errors. When class predictions are made sequentially, early mistakes are likely lead to additional errors for subsequent classes.
Strict metric. Subset Accuracy is only nonzero if the predicted label set is exactly correct, which means that a mostly-correct prediction is penalized as harshly as a completely incorrect solution during evaluation. This non- non-smooth metric is naturally challenging maximize.
Exponential label set growth. The number of possible label sets grows exponentially with the number of classes. Thus, finding the label set that is exactly correct akin to finding a needle in haystack for large numbers of classes.
We propose the Recurrent Bayesian Classifier Chain (RBCC), the first recurrent classifier chain that avoids conditioning on noisy inter-class relationships. thereby learning highly-principled ordering for class predictions. The RBCC consists of three core components. First, the RBCC explicitly learns inter-class dependencies using the Inter-class Dependency Network, which finds a Bayesian network that models the relationships between the classes. Second, this network structure used by the Recurrent Conditional Dependency Model, which for each class produces latent vector representation that summarizes the input features along with the classes upon which depends. Classifier, which produces the
This representation is then passed on to the probability of observing each class. Importantly, the Bayesian- Conditioning Classifier's predictions for given class are only conditioned conditionally independent (which we refer to as the parent classes). Altogether, the RBCC thus produces accurate label sets, optimizing for exact multi-label classification by mitigating cascading errors.
the input features and the classes for which it is not
Contributions. The contributions of this work are follows:
The RBCC the first recurrent classifier chain method that utilizes conditional independence to perform non-noisy class conditioning.

---

We propose a novel training and inference algorithm for this approach.
Experimentally. we show that RBCCs significantly outperform state-of-the art methods for E-MLC variety of real-world datasets.
2 Methodology
2.1 Problem Definition
Formally, we define the problem of E-MLC as follows: Let (X, CI,,.,,Ci Pxx XxxCs XCI where X is the random variable of features and C; is the ith class, such that there are L possible classes. If the observed value of C. Cj. is 1, then the ith class applies. Otherwise, G; 0 and the class does not apply. Let D {x, y) } be dataset of i.i.d samples from Pxxc, xc.. such that each I is an observed value X and y is vector of length of observed values for all classes, and thus [y]: (the ith element of vector y) is equal to Our goal is to find a classifier fe that maximizes the Subset Accuracy of unobserved test data. The Subset Accuracy (S of a prediction fo( is defined as:
(A(fo()), y) = ly = fo(x))]
where [:I is the conditional function such that [P] returns if the predicate P is true, and is otherwise.
The risk minimizer for SA is given by:
h" = arg max P(C1 = (191,.... CL = [y]2|X = x)
(I)
y
Thus, our classifier fe should aim to return the Maximum A Posteriori of the class probabilities given observations
2.2 Proposed method: RBCC
We propose the Recurrent Bayesian Classifier Chain (RBCC), which improves upon RCCs by performing class prediction in a principled order with non-noisy class conditioning during prediction. We achieve this in part by learning Bayesian network representation of inter-class dependencies. We note here that the focus of this work is not on class inter-dependencies, focus is how to better incorporate a Bayesian network found through these existing methods into the recurrent classifier chain system.
method to construct a Bayesian network of
which there are many existing methods 22 6 57. Instead, the
At high level, RBCC aims to find to find the parameters of classification model such that the predicted label vector the most likely to exactly match the true label vector y. Thus, we are aiming to return the risk minimizer given in Equation 2 and are as such optimizing for the exact goal of E-MLC.
This is achieved through finding model of P(C Cr(XX) parameterized by 0:
0 arg max
logPo(æ:; ();;0))
(2)
Then, the prediction for y, can be found be searching for the maximum of Po(æ;;0);
At glance, to model ([C1,..,C2||X would be to factorize it into product of per-class probabilities:
= pz) II P(C.(C2)))) $=2
C
(3)
as is done by [5]. However, there are two issues apparent with the above factorization. First, while in theory we can order the classes in the above product in any way if the exact true conditional probabilities are known, in practice the order classes has an immense effect on classification performance [9]. For instance, if Cm is strongly dependent class C,, then the prediction for Go should be conditioned on that class during training for more accurate predictions. However, depending on the ordering of the classes, Cm may not be conditioned c, in the factorization above. Second,

---

Symbol
Meaning
Feature random variable
C
Random variable of ktth class
Observed value for kth class. If = 1 the class applies;
otherwise, C₁ = 0 and the class does not apply
a
Observed feature value for particular instance
Vector of class observations; [e1, C2, C2,... cL]
[(-]]s
kth entry in vector (-)
g
Bayesian network of class dependencies
Unbolded capital letter
random variable
Unbolded lowercase
Observed value of random variable
Bold uppercase letter
A matrix
Bold lowercase letter
A vector
Table 1: Meaning of the most import notation used in this work.
each class Cj is conditioned on all of the preceding classes C'=k. even if C₃ is independent from most of these classes. Approaches that use the factorization shown in Equation 3 condition the prediction of c on noise, risking overfiuting to false correlations between classes during training.
We can solve these two issues by assuming that the joint dependencies among all classes C₁ C2 ... Ce and features can factorized as a directed acyclic graph (DAG) g. Each node in the graph corresponds to either class C; or the features X, and the edges encode the dependencies between nodes. As we make the standard multi-label assumption that all classes depend on the input features X, X is parent of all other nodes in G, other than itself. Let Pacyx (Cx) be the parents of Ck. other than the universal parent X. We can thus simplify P(C1, Cr. (X) into the following:
P(CI,C,...,CuIX)) CLI) = II
(C)),
The above equation does not have the flaws of the factorization in Equation 3 as each class is now only conditioned all of the classes upon which it depends.
RBCC thus involves three primary steps to encode this key idea into one unified multi-labes model: For each class we need to first identify its parent classes, summarize the information of the observations of the parent classes, and then make a prediction for that class based on the observations of its parents.
In the first step, we learn the structure of the inter-class dependencies using the Inter-class Dependency Network, which fits a Bayesian network to the classes. This network does not tell us the strength of the dependencies between classes and does not alone allow us to infer which classes are associated with given instance. Instead, the learned network identifies which classes given class dependent on (the parent classes), and which classes it is independent of given the parent classes. This allows us to condition a down-stream classification model on only the required parent classes of each class, reducing cascading emrors from incorrect class predictions.
The input features and observations of the parent classes are summarized by the Recurrent Conditional Dependency Model, recurrent neural network baasrr--based architecture, before this summarized repre- sentation is passed into the down stream classifier, which we refer to as the Bayesian-Conditionin Conditioning Classifier. The CoveesinnCCndiitionn that the predictions are conditioned only on the observed features x and the parent classes of Ck. according to the network structure learned from the Inter-class Dependency Network.
Classifier model makes predictions for each class Cs. such

---

2.2.1 Inter-class Dependency Network.
The Inter-class Dependency Network learns for each class both which other classes it depends on and which classes it is independent from when conditioned on X. Thus, for class Ck. the Inter-class Dependency Network finds Pa(Ca the set of classes that such that P(Cx + P(C X) V C2 E Pa(Cz).
We accomplish this by assuming that the inter-class dependencies can be modeled by a Bayesian network. Thus. the inter-class dependencies are modeled by directed graph structure such that each node represents a class, and the edge from the node for Ce to Cj means that Ce E Pa( C+)). Furthermore, the Bayesian network by definition acyclic, implying that if C. E Pa(Cs). then C& & Pa(C)). This implies that we can obtain a principled ordering for class predictions, such that each class can always be predicted after its parent classes.
Note that we are interested in the parents of each class conditioned on the features X. This is an important distinction, because the structure of the inter-class dependencies may differ from the structure of the classes when conditioned on the observed features. Thus we can not learn the desired conditional class inter-dependencies by simply fitting Bayesian network structure the classes.
To model the correct conditional dependencies, we follow the approach of Zhang et al. [30] and identify the residuals of each class. To this end, assume that the relationship between each class and X can be expressed as
C 9%x Eg, such that g (X) is the component of C₁ that is predictable from X, and Fxx is the irreducible error of C& that is not predictable from X. Note that if there are inter-class dependencies, Eg will be in part predictable given some other class.
Lemma 1. The Bayesian network structure of the classes conditioned X is the same as the network structure of {E1, E2..., EL}.
Lemma proven in [30 implies we can learn the inter-class dependency structure by fitting Bayesian network to the residuals of each class. For each class Cs we train classifier ga X - {0, 1) which predicts e given æ. We then obtain Fg. as the residual of the predictions of 9k-
The final step fit a Bayesian network to the residuals. As finding the Bayesian network structure is NP-hard, we must estimate this structure. There are many algorithms to approximate fit Bayesian networks 22 6 27 [1. We use hill climbing [2]] for our experiments due its low computational cost. After fitting the Bayesian network, we obtain the graph structure Gg and determine the parents Pagn (Ex) for every Eg. Note that from this we can obtain Pagy x (C&) for each Ck. because
E, E Page (Fx) then C2 Pagyx (Ck).
2.2.2 Recurrent Conditional Dependency Model.
With knowledge of each class's 's parents in hand, we next perform classification (i.e., we want to find P(C₂ || PaPa x, X x) for each class Cg). However, the number of parents will vary for each class. This implies that class prediction must be performed with a variable number of inputs. This remains standing challenge for state-of-the-ar machine learning, most methods require constant input sizes. For this reason, we leverage recent successes in representation learning, summarizing the input features æ and the observed valuyes of the set of parent classes Pagi, (CC). This is achieved using our proposed Recurrent Conditional Dependency Model.
The Recurrent Conditional Dependency Model utilizes a recurrent network that learns joint feature representation that summarizing the input features and observed parent classes for each class C2 of a given instance. This representation is learned by reading in the observed features and parent classes sequentially. To mitigate effects of vanishing gradients, we compute use gated recurrent unit (GRU) for the recurrent network to create the representation. Thus, the representation a is determined by:
T: = o (W,v + U,h + br)
= aW
(W .. U-hh111+ bz)
8: = + W.a v + U Un(r; 2 (2.--1) ba)
a = 2, C - + (1 2x) 2 St
5

---

such that W r.s.a and U r,s, are the weight matrices of the GRU, and =< 2, a, b > the concatenation of and and enc(c, vector encoding of ct. the sigmoid function and is the hyperbolic tangent function. In general, the choice of recurrent architecture is modular and can be substituted with some other architecture, such as an L.TTM.
2, enc(c))) >, where
The above process is repeated for all C₁ in Pagyx (C²).. The final representation ag for class Cj is obtained after reading the observed value for the final parent class. If the class C₃ has no parents, then we set v =< 2, 0 > and compute the above equations only once each.
2.2.3
Classifier.
We now the summarized information of the parent classes and input features to condition classifier to produce P(Ck| for each class Ck,, as required for Equation ??. We model this probability using the Cayesian-Conditionn Classifier.
We thus pass said representation ag into a feed forward network m. such that
m(ax) o (WHa:)) ) + bи:
where WH E R/.aa and by E R are the weight and bias vector of m, respectively. Thus, m maps ag into a vector of length L, where [m(as)), is our model of P(C₂ I|Pa(C,), x).
To avoid inadvertently conditioning on noise by conditioning our prediction for a given class C on classes other than its parents, we reset the hidden representation ag after predicting each class. Thus, we perform many-to-one prediction for each class, where we read the sequence of parents and then predict binary value for the class. This differs from the approach taken by standard RCCs, as they perform many-lo-many classification by reading in sequence of previously predicted class and predicting a sequence of successive classes.
This strategy poses a challenge when multiple classes have the exact same parents. For instance, in if Pagix(Cs) (Cx) Pa(Ct), then a₃ ag. However, we do not necessarily want P(Cs 1|Pa(C*), X æ) = P(C transforms the latent representation into a prediction probability does not map each as into [o. I]. Instead, m maps as to [0. 1|4, and we take one value of the L-dimensional vector (i.e., [m((a)]ss to represent P C))a = x). This way, even if Cas = Car, we can still have P(C₂ 1|Pa(CC), x) # P = P( (m())))e.
= æ). For this reason, the feed forward network m that
x) [m[a:)] & is not required to equal
As all classes that apply to a given instance are observed during training, we can train the Recurrent Conditional Dependency Model and Bayesian-Conditioninn Conditioning Classifier by simply minimizing the binary cross entropy (BCE) between the predicted values for each class and whether not the class applies the given instance. Let ÿ E [0, 1] be our probabilistic predicted label set for æ, such that [U]= [m(ax)]a. Thus, our loss function C((]}= is:
1
[yl log([m(a))))) - (1 log(1 [m({a)]s))
(5)
During testing, no classes are observed. We thus require inference strategy that predicts the value of all parent classes before their child classes. To this end, we construct a recursive algorithm that predicts each class given observations for its parent classes. If the parents have not yet been predicted, we recursively call the inference algorithm on the umpredicted parent classes.
We note that this results in greedy estimate of the most likely label vector for each instance. Alternate searches for the most likely label vector, such as an A search or c-approximate search, could likewise be performed [19]. We apply greedy search as it is computationally inexpensive and the simplest solution, while still leading to high accuracy as we demonstrate in the following section.
Experiments
3.1 Datasets
We evaluate our method on three commonly used benchmark multi-label datasets 9 these we use the provided train/test partitions and are not modified.
2
21
for all of

---

This standard multi-labe image dataset consists of 9.963 natural images with 20 possible classes. Images were originally collected from "flickr". Each image was featurized by extracting the features of a pretrained ResNet-18 [10] model into 51 512-dimensional vectors. Specifically, ResNet-18 18 model was pretrained on ImageNet and the feature representations were extracted from the final average pooling layer.
PASCAL VOC 2007 [9]:
Scene This dataset contains 2407 scenery images with 6 possible labels. The features are 294- mone-imnnionall vectors corresponding to the spatial color moments in the LUV space.
Yelp [11]: This dataset consists of over 10,000 reviews of business submitted by users of Yelp. There are 7 classes, though? 3 are mutually exclusive. Each review is featurized into 671 tokens.
The Scene and Yelp datasets contain no personally identifiable information. PASCAL VOC was scraped from Flikr, and therefor might have identifiable information. However, all of the PASCAL VOC data was factorized with a non-invertable transformation (Resnet-18 model), and thus no longer will contain identifiable information.
To the best of our knowledge, no explicit consent obtained from the people whose data was collected beyond the terms of service of the websites from which the data was scraped. However, these are standard benchmark datasets. We had no part in the onstruction of these datasets.
3.2 Compared methods
We compare our RBCC method against the following state-of-the-art and baseline E-MLCC multi-label classification approaches:
Recurrent Classifier Chain (RCC) 15:: This method uses unified RNN model to iteratively predict the classes that apply to each instance. The classes are ordered according to their frequency, with the most frequent classes being predicted before the rarest. The intuition is that the more frequent classes will be easier than the rarer classes, and thus the model will make fewer predictions early on if the frequent classes are predicted first. This would then mitigate cascading label errors. This class ordering is standard for classifier chains [9]. Note that there are three RCC variants proposed in [[3] RNN RN NM and Ence Dec. We refer to the RN No approach as RCC, and is the method we compare against. We forego direct comparison with R.N N as it's main difference is that it predicts all positive classes first. This is likewise done by the extension of this method, OF RCC, which we do compare against. Lastly, we do not compare against Enc - Dec as it is specific to sequence data, which is not what we focus on in this work.
Topological-Sone -orted Recurrent Classifier Chain (TS-RCC) 15
This method likewise uses a unified RNN model to iteratively predict classes. However, before training the model Bayesian network is fit to the class residuals, as done for RBCC and is described in Section 2.2.1 Then, we perform topological sort on the classes according the the Bayesian network, such that each class is placed after all of its parents. This ordering is then used for the class prediction of the RCC. While this method has not been proposed in the literature, it is the natural way of combining Bayesian networks with recurrent classifier chains.
The OF-RCC uses the same base RNN architecture as the RCC and TS-RCC. However, instead of specifying an order for the classes to be predicted in, the OF-RCC learns the ordering itself. We compare against this method to validate that our RBCC's approach of predicting the classes in a principled order produces better results than relying on the model to learn an optimal ordering.
Order-Free Recurrent Classifier Chain (OF-RCC) 4:
Bayesian Classifier Chain (BCC) 23 Unlike the RCC models, the BCC utilizes an ensemble of individual models rather than unified RNN classifier. Each class is given corresponding classifier, such that the classifier takes in the observed features as well as the predictions for the parents of that class. The parents of each class are found by fitting Bayesian network to the class residuals, as described in Section 2.2.
license: PDDL
http://wwww...ooooo.t.tllllsosss license: https: melis//3mmedia...m fww.elleel....mmm com/
agreement.

---

Methods
Evaluation Metrics
RBCC (Ours)
RCC
TS-RCC
OF-RCC
BCC
BD
Subset Accuracy
0.240 + 0.008
0.212 = 0.002
0.192 + 0.010
0.169 1 0.009
0.210 = 0.000
0.202 + 0.002
Hamming Loss 1
0.186 + 0.003
0.204 = 0.001
0.209 + 0.004
0.218 : 0.004
0.199 = 0.001
0.189 = 0.000
Macro-F1
0.556 0.008
0.526 + 0.004
0.506
0.004
0.569
1 0.004
0.551 0.005
0.517 0.008
Micro-F1
0.670 0.006
0.639 + 0.002
0.628 0.004
0.662 + 0.004
0.653 0.003
0.638 0.003
Table 2: Results for the Yelp dataset. Bolded is the best performer, underlined is second best.
Methods
Evaluation Metries
RBCC (Ours)
RCC
TS-RCC
OF-RCC
BCC
BD
Subset Accuracy
0.653 + 0.007
0,548 = 0.006
0.545 + 0.007
0.610 # 0.007
0.527 = 0.008
0.479 = 0.013
Hamming Loss
0.097 0.002
0.116 + 0.001
0.134 + 0.002
0.102 + 0.001
0.110 + 0.002
0.105 + 0.002
Macro-FI
0.727
0.004
0.610 + 0.005
0.576 + 0.010
0.742 + 0.004
0.660
0.009
0.650 0.010
Micro-F1
0.722 0.005
0.650 0.004
0.613 0.005
0.735 0.004
0.672 0.006
0.652 0.010
Table 3; Results for the Scene dataset. Bold is the best performer, underlined is second best.
Binary Decomposition (BD) g 14] Binary Decomposition is the process of transforming a multi-label classification problem into set of binary classification problems. Specifically, separate binary classifier constructed per each class. Despite the fact that this method does not capture inter-class dependencies, it is competitive approach for multi-label classification depending on the evaluation metric [8]. performance is measured as a function of how many individual classes are predicted correctly, rather than if the label predicted label set is exactly correct, then BD is expected to do well.
Unlike the RCC models, the BCC and BD approaches utilize ensemble of individual models rather than a unified RNN classifier. Each class is given a corresponding classifier such that the classifier takes in the observed features as well the predictions for the parents of that class for the BCC, and only the observed features for BD. The parents of each class are found by fitting a Bayesian network to the class residuals, as described in Section 2.2.1
3.3 Implementation Details
RBCC, RCC, TS-RCC, and OF-RCC all utilize the same RNN architecture (i.e., same number of layers and hidden sizes) and the same architecture for the classification layer. As the BCC and BD methods require an ensemble of models rather than a unified recurrent architecture, we replace the RNN with feed-forward network for BCC and BD. However, the feed-forwart network has comparable number of parameters to the RNN and classifier used by the other methods in order to perform fair comparison.
Experiments were performed on a computing cluster, using a Intel(R) Xeon(R) Platinum 263 8160 CPU @ 2. 10GHz CPU, an NVIDIA Tesla V100 SXM2 GPU, and 128 GB of RAM.
Full details, including our specific architectural choices, are available in the Reproducibility Appendix. More information
The code for RBCC is available at http://thub..cmmmaaleeeerrggggg/ available in our supplemental materials.
3.4 Metrics
To evaluate the performance on each dataset, we use four standard multi-label metrics: subser accuracy, Hamming Loss, macro FI, and micro Fl. Subset accuracy is the metric we explicitly optimize for as directly corresponds to exact multi-label classification.
3.5 Classification Performance
We begin by demonstrating that our RBCC outperforms 5 state-of-the-art methods for multi-label classification. Results for the Yelp, Scene, and PASCAL VOC 2007 datasets are shown in Tables 2 and respectively. Notably, our RBCC outperforms all other methods for each dataset for the
8

---

Methods
Evaluation Metrics
RBCC (Ours)
RCC
TS-RCC
OF-RCC
BCC
BD
Subset Accuracy
0.394 0.009
0.457 = 0.057
0.454 + 0.008
0,575 1: 0.007
0.581 = 0.011
0.578 1 0.002
Hamming Loss 1
0.028
0.000
0.045 = 0.006
0.046 + 0.000
0.030 4 0.001
0.035 = 0.001
0.029 = 0.000
Macro-F1
0.718
0.016
0.495 + 0.092
0.577 0.011
1 0.007
0.629
0.746
0.003
Micro-F1
0.787 0.006
0.646 0.054
0.662 0.006
0.782 0.004
0.728 0.005
0.781 0.003
Table 4: Results for the PASCAL VOC 2007 dataset. Bold is the best, underlined is second best.
Coatraks Bardd
Seler Leen Mobel
Figure 2: Observing the effect of the network learning algorithm RBCC's performance. Results obtained from Scene dataset.
strict subset accuracy metric. This is important, this metric corresponds our problem definition (predicting the exactly correct label set).
The second- best method in terms of subset accuracy is usually of the RCC methods, which is as expected as RCCs are the leading approach for maximizing subset accuracy. For the PASCAL VOC 2007 dataset, the BCC performs second best, despite utilizing parameter sharing during class prediction. This could be explained by PASCAL VOC 2007 having more classes than the other datasets. Thus, the RCC approaches must perform inference over longer sequence of classes, as they condition class prediction on all previously predicted classes. However, the BCC shares RBCC's approach of only conditioning the predictions for a given class on the parents of that class, resulting in the method not having to do inference over longer sequences even as the number of classes grows.
Our RBCC likewise performs the best each dataset in terms of Hamming Loss. Unsurprisingly, binary decomposition performs second-boo best in terms of this metric. This is as expected, as Hamming Loss measures classification performance a per class, per instance basis, rather than the label-set as a whole. Thus, it is reasonable that the binary decomposition approach of maximizing the probability for each class individually results in good performance.
Lastly, RBCC performs well on the Micro-Fl metric (the best on two datasets, and second-best on the third). but is second-best in terms of Macro-Fl for two datasets and third-best for one. This discrepancy in fl-scores is likely explained by Macro-Fl being more sensitive to class imbalance. Even if class appears very infrequently, it is weighted equally to more frequent classes. This is in contrast to our primary target of subset accuracy.
3.6 Effect of Network Learning Algorithm
Using the Inter-class Dependency Network component of RBCC, we fit Bayesian network of class inter-dependencies to the observed data. There exist several choices for the learning algorithm used to fit the Bayesian network. In this experiment, study the effect of the choice of learning algorithm on RBCC's classification performance. Specifically, we test the performance with three standard Bayesian network approaches: Hill Climbing 22 Constraint Based 22 27), and the Chow Liu algorithm (5).
The Hill Climbing approach is a greedy local search [22]. Starting with a disconnect DAG, single- edges are added or removed in order improve the likelihood score of the model. Once local optimum is obtained, the search ceases.
The Constraint Based approach 22 27 identifies in in-dependencies in the data using Chi- Shuured hypothesis test. The Bayesian network is then constructed according to this independence test.
The Chow-Liu algorithm finds the moximum-llkelihoon graph such that each node (class) has at most one parent. First, the mutual information for class pairs is calculated. Next, the maximum weight spanning tree is found according to the obtained mutual information. Lastly, random node is picked

---

to be the root node, with directions assigned to each edge radiating outwards from the root node, obtaining a directed graph from the undirected maximum weight spanning tree.
The results of this experiment on the Scene dataset are shown in Figure Importantly, the perfor- mance of RBCC is very similar for each network learning approach, as the performance difference between methods is typically within the margin of error of each other. This indicates that RBCC is robust the choice of this algorithm.
There is a slight performance increase (roughly 0.5%) for the Hill Climbing approach in terms aiming to
of Subset Accuracy, indicating that it might be the more appropriate choice maximize this metric in particular. Indeed, the Hill Climbing approach is the default for RBCC, as described in Section 2.2.1
Interestingly, the Chow Liu algorithm outperforms the other approaches by a small margin in terms of the other metrics (Hamming Loss, Macro-F1, and Micro-F1), despite performing the worst for subset aceuracy, it being the more restrictive network learning approach. This might be explained by the fact that the Chow Liu algorithm assigns fewer parents to each class on average than the other approaches (with an average 0.86 parents per class for Chow Liu and 1.67 for Hill Climbing). Thus, this approach might learn less-accurate inter-class dependencies, explaining why it gives the lowest subset accuracy score. However, for the metrics where capturing inter-class dependencies is less important, having fewer parents results in shorter sequences to perform inference over and may lead to better predictions on individual classes rather than label sets.
Limitations
RBCCs faces the same limitation that nearly every exact classification suffers from: Predicting the label set exactly correct becomes exponentially difficult as the size of the set of possible labels grows. Thus, performance will degrade in settings where there are very many possible labels.
Negative societal impact. Our method improves over standard recurrent classier chains, which commonly used in many object detection and image recognition systems. Image recognition systems that recognize individuals or members of certain minority groups have recently raised ethical concerns [26]. However, such systems are not the focus of this work, and RBCCs are not any more prone to ethical misuse than are any other classifier chain or multi-label classification approaches.
5 Conclusion
In this paper we propose RBCC, an effective method for exact multi-label classification that improves on recurrent classifier chains' ability to model inter-class dependencies using principled class ordering and by conditioning the predictions for each class on only its parents. Our principled class ordering and the parent classes themselves are found using by fitting a Bayesian network to the classes. Through extensive experiments we show that RBCC consistently outperforms the leading methods for exact multi-label classification. In particular, RBCC outperforms the other methods in all cases with respect to the strict Subset Accuracy metric, which is the metric that directly measures exact classification. Additionally, we find that RBCC is robust to the choice of network learning method. By showing the benefit of combining recurrent classifier chains with Bayesian networks, which has to-date been unstudied, we believe results can inspire additional research combining class network models with recurrent classifier chains.
6 Acknowledgements
This material is based on research sponsored by DARPA under agreement number FA8750- 18-20077 The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of DARPA or the U.S. Government, Additionally, we would like thank the WASH Research Group at WPI as well as the DAISY Lab at WPI for the invaluable feedback during the research process.
10

---

References
[1] Jonathan Baxter. 1997. A Bayesian/information theoretic model of learning to learn via multiple task sampling. Machine learning 28, (1997). 7-39.
[2] Matthew R Boutell, Jiebo Luo, Xipeng Shen, and Christopher M Brown. 2004. Learning multi-label scene classification. Pattern Recognition 37, 9 (2004), 1757-1771.
[3] CDC. 2020. Similarities and differences between flu and COVID-19. Retrieved July 13 (2020).
[4] Shang-Fu Chen, Yi-Chen Chen, Chih- Kuan Yeh, and Yu-Chiang Frank Wang. 2018. Order-free rn with visual attention for multi-label classification. In Proceedings of the Thirty-Second
AAAI Conference on Artificial Intelligence.
[5] Chow and Cong Liu. 1968. Approximating discrete probability distributions with dependence trees. IEEE transactions on Information Theory 14, 3 (1968), 462-467.
[6] Rónán Daly and Qiang Shen. 2007. Methods to accelerate the learning of bayesian network structures. In Proceedings of the 2007 UK Workshop on Computational Intelligence.
[7] Krzysztof Dembezynski. Weiwei Cheng, and Eyke Hüllermeier. 2010. Bayes optimal multilabel classification via probabilistic classifier chains. In ICML.
[8] Krzysztof Dembezyáski, Willem Waegeman, Weiwei Cheng, and Eyke Hüllermeier. 2012. On label dependence and loss minimization in multi-label classification. Machine Learning 88, 1-2 (2012), 5-45.
[9] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. [n.d.]. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal wttphhahleeeeeeooooe///////ooooa
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. 770-778.
[11] Xie Jun, Yu Lu, Zhu Lei, and Duan Guolun. 2019. Conditional entropy based classifier chains for multi-label classification. Neurocomputing 335 (2019), 185-194.
[12] Arthur B Kahn. 1962. Topological sorting of large networks. Commun. ACM 5, 11 (1962), 358-562.
[13] Tomasz Kajdanowiez and Przemyslaw Kazienko. 2009. Hybrid repayment prediction for debt portfolio. In International Conference on Computational Collective Intelligence. Springer, 850-857.
[14] Oscar Luaces, Jorge Díez, José Barranquero, Juan José del Coz, and Antonio Bahamonde. 2012. Binary relevance efficacy for multilabel classification. Progress in Artificial Intelligence 1, 4 (2012), 303-313.
[15] Jinscok Nam, Eneldo Loza Meneía, Hyunwoo Kim, and Johannes Fürnkranz. 2017. Maximiz- ing subset accuracy with recurrent neural networks in multi-label classification. In Advances in Neural Information Processing Systems. 5413-5423.
[16] Yue Peng, Ming Fang, Chongjun Wang, and Junyuan Xie. 2015. Entropy chain multi-label classifiers for traditional medicine diagnosing Parkinson's 's disease. In 2015 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 856-862.
[17] Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. 2009. Classifier chains for multi-label classification. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 254-269.
[18] Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. 2011. Classifier chains for multi-label classification. Machine learning 85, (201 1), 333.
[19] Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. 2019. Classifier Chains: A Review and Perspectives. arXiv preprint 1xX1v:1912.13405 (2019).
[20] Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:17066 05098 (2017).
[21] Hitesh Sajnani, Vaibhav Saini, Kusum Kumar, Fugenia Gabrielova, Pramit Choudary, and Cristina Lopes. 2012. Classifying yelp reviews into relevant categories.
11

---

[22] Marco Scutari. 2009. Learning Bayesian networks with the bnlear R package. arXiv preprint PxXiv:0908.8817 (2009).
[23] L. Enrique Sucar, Concha Bielza, Eduardo F Morales, Pablo Hernandez-Leal, Julio H Zaragoza, and Pedro Larrañaga. 2014. Multi-label classification with Bayesian network-based chain classifiers. Paltern Recognition Letters 41 (2014), 14-22.
[24] C.-P. Tsai and H.-Y. Lee. 2020. Order-free Learning Alleviating Exposure Bias in Multi-label Classification. In AAAI.
[25] Grigorios Tsoumakas and loannis Katakis. 2007. Multi-label classification: An overview. International Journal of Data Warehousing and Mining (IJDWM) 3, 3 (2007). 1-13.
[26] Richard Van Noorden. 2020. The ethical questions that haunt facial-recognition research. Nature 587, 7834 (2020), 354-358.
[27] Thomas Verma and Judea Pearl. 1991. Equivalence and synthesis of causal models. UCLA, Computer Science Department.
[28] Julio Cesar Zaragoza, Enrique Sucar, Eduardo Morales, Concha Bielza, and Pedro Larranaga. 2011. Bayesian chain classifiers for multidimensional classification. In Twenty-second interna- tional joint conference on artificial intelligence. Citeseer.
[29] Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. 2018. study on overfitting in deep reinforcement learning, arXiv preprint arXiv:1804.066: (2018).
[30] Min-Ling Zhang and Kun Zhang. 2010. Multi-label learning by exploiting label dependency. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining. 999-1008.
12


Reviews:
[{"review_id": "rAzLzpZ1MiL", "reviewer": "Reviewer_JRCf", "summary": "This paper presents a new method for optimizing the subset zero-one loss in multi-label classification. The new method is an adaptation of probabilistic classifier chains. The central idea is to replace the chain by a dependence structure learned using Bayesian networks. In theory the order of the labels in a probabilistic classifier chain does not matter, but in practice one can observe performance drops if the chosen order leads to propagation of errors. By replacing a random order with the dependency structure of a Bayesian neural network, the authors intend to solve the problem of choosing an appropriate order. \n\nAs underlying models to fit probabilities, recurrent neural networks are used. Experimental results are reported for three benchmark datasets. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall this is a nicely-written paper that presents a novel idea. Indeed, it is the case that in practice the order of labels in a probabilistic classifier chain matters. Learning the dependency structure by means of a Bayesian network is therefore a useful thing to do. Intuitively, I have the feeling that the presented approach makes sense. On the other hand, the presented approach is not very novel, since in reference [30] something similar was done, albeit without a focus on subset zero-one loss and without recurrent neural networks as underlying models. However, I do believe there is enough novelty to justify a new paper on this topic. \n\nThe existing literature is covered quite well, and in the experiments the choice of baselines is appropriate to my opinion. With only three datasets, the number of datasets in the experiments is quite small, especially because there are much more datasets in the public domain, see e.g. the MULAN datasets. I believe that it is important to justify why only those three datasets were analyzed in order to avoid being accused of cherry picking. I can imagine that those datasets were chosen because they have more complex feature spaces than the simple MULAN datasets, but it would be good to state that explicitly. \n\nAt some points the mathematical notation is a bit sloppy. In general it is important to formally define all notations, while being consistent in the use of vectors, components of vectors, sets and random variables. An example of an awkward notation is line 159, where C_ki = g_k(x_i) + E_ki. The model g is here a classifier that probably outputs zero or one, but C_ki is a pointer to class k. It might be better to use the notation Y_i, as defined on page 2, and refer to the k-th entry in Y_i. Also in Lemma 1, the quantities E_1,...,E_L are used but they are not formally defined. Most likely they allude to the vectors that originate by grouping E_i,k for all i. Another example of an incorrect notation is Eqn. 1, because the expectation over X is not needed there. If you want to refer to parameter fitting, please parameterize the distribution P by theta and formulate the optimization problem as the minimization of the log loss over a training dataset. Eqn. 1 looks very awkward for the moment. \n\nI don't find the title very appropriate, and I would suggest to change it. When I downloaded the pdf, I thought the paper would be about a Bayesian variant of probabilistic classifier chains, in which a distribution of models is learned instead of a single model, similar to Bayesian methods such as Gaussian processes and Bayesian neural networks. However, the term Bayesian here comes from the fact that Bayesian networks are used to learn dependencies between labels. For example, a title like \"Learning class dependencies for probabilistic classifier chains\" or something similar would be more appropriate. \n   ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They have a paragraph about this in the paper. No concerns from my side. ", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "b53M2HmiUtM", "reviewer": "Reviewer_RLfS", "summary": "This paper proposes a new method, Recurrent Bayesian Classifier Chains (RBCC), for exact multi-label classification problems (maximizing subset accuracy). RBCC involves three steps: 1. Intra-class Dependency Network to learn a Bayes net to the classes; 2. Recurrent Conditional Dependency Model to summarize the information of features and the corresponding parent classes learned in the previous step; 3. Bayesian-Conditioning Classifier to predict. RBCC is shown to outperform several state-of-the-art (STOA) algorithms on some real-world multi-label data sets with respect to subset accuracy.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I think the method (RBCC) proposed in this paper is relatively novel and its performance is good. The description of the method is fairly clear, and enough background is provided. My main concern is about the experiments (see below). The following are my comments on each of the dimension.\n\nOriginality.\nThe method (RBCC) is new. Although each component is known, the combination is relatively novel, for example, how to better incorporate a Bayes net into the recurrent classifier chain system. The related work is adequate and some STOA algorithms are described and compared.\n\nQuality.\nThe work is sound. The methods for designing RBCC are appropriate. However, there are some issues with the experimental design, which I list below.\n\n1) Why some data sets (Reuters-21578, RCV1-v2 and BioASQ) in paper [15] are not used in the experiments? Both paper [15] and this paper aim to maximize subset accuracy, and methods in both papers are based on Recurrent Classifier Chain (RCC), so it is reasonably expected the authors should use one of the data sets from [15] or at least provide explanations for not using.\n\n2) I might be missing something, but I could not find the number of repeated experiments in the paper, given that the error bars are reported.\n\n3) Training time and inference time are not discussed. It looks that since RBCC has slightly more complicated structures than RCCs, it might take longer to train. Unfortunately this info is lack in the paper. The authors should discuss the computational costs.\n\nClarity.\nOverall this paper is clear. Enough background is provided. The descriptions of the proposed method is detailed and clear. Nevertheless, some terms are not clearly defined and there are several typos (some of which are listed below). If this paper is accepted, the authors should proofread carefully, add necessary explanations and fix typos. Question regarding some terms.\n\n1)  What is \"non-noisy\"? I can get a sense of the meaning after reading the whole paper, but I could not find where this is defined.\n\n2) In the contributions, it is stated that a novel training algorithm is proposed. Which part of the paper does this \"training algorithm\" refer to?\n\nSignificance.\nThe method (RBCC) proposed in this work is useful for researchers and practitioners working on multi-label learning. It also advances STOA in exact multi-label classification problems.\n\n---\nExtra question (the authors do not need to respond):\n\n1) Could the authors also report metric F1 (instance-averaged) in the experiments?\n---\nTypos:\n\nPage 3, line 100, duplicate citations [22]\n\nPage 3, line 113, \"strongly dependent class\" -> \"strongly dependent on class\"\n\nPage 3, line 122, $P(C_k | x) \\ne P(C_k | C_n, x)$ -> $P(C_k | x) = P(C_k | C_n, x)$\n\nPage 5, line 187, $V = $ -> $V_t =$\n\nPage 6, line 243, \"in the onstruction\" -> \"in the construction\"\n\n---\n\nAfter rebuttal: My main concerns (1 and 3 in the Quality section) are addressed. I keep my original rating for this paper.\n\n---\nCitations:\n[15] Jinseok Nam, Eneldo Loza Mencía, Hyunwoo J Kim, and Johannes Fürnkranz. 2017. Maximizing subset accuracy with recurrent neural networks in multi-label classification. In Advances in Neural Information Processing Systems. 5413–5423.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "EHc5EsaJyT3", "reviewer": "Reviewer_DcQS", "summary": "The authors propose Recurrent Bayesian Classifier Chains (RBCCs), a new method for exact multi-label classification. It combines the idea of ordering and conditioning labels based on a Bayesian network, utilized previously in Bayesian Classifier Chains (BCCs), and using a recurrent neural network as in Recurrent Classifier Chain (RCC). Usage of recurrent architecture allows modeling that condition on previously predicted classes instead of using many separate binary models originally used in the Classifiers Chain. The proposed approach uses an established approach to construct a Bayesian network. The novelty of the approach lies in the training and inference procedure. Instead of performing many to many classification, where labels are predicted one after another as in the case of standard RCCs, in RBCCs, many to one classification is performed for each class. Each class is conditioned only on features and labels it depends on in the obtained Bayesian network. The author validates the attractiveness of their method in empirical study on three popular benchmark datasets with a small number of labels. In all three cases, the proposed approach significantly improves in terms of subset accuracy over the baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ ~~The organization of this paper is good, and it's easy to read and understand.~~\n+ The proposed method seems to be an intuitive and neat improvement to BCCs/RCCs formula.\n+ The RBCCs consistently outperform the benchmark in the empirical evaluation.\n+ Comparison of different Bayesian network construction technique and their impact on the predictive performance of RBCs.\n+ Comprehensive description of implementation details in the appendix.\n\nWeaknesses:\n- Empirical evaluation was conducted on only three datasets, all with a small number of labels - this is the only clear issue I have. With so many multi-label datasets from varying domains and a varying number of labels. I believe that a much more extensive empirical evaluation should be performed.\n\nOther comments:\n* The simplest baseline that directly optimizes argmax E[P(Y|x)] is the well-known labels power set method. Why not include it as a simple baseline, especially when comparing datasets with small labels?\n* \"If the parents have not yet been predicted, we recursively call the inference algorithm on the unpredicted parent classes.\" - this is not necessary since the Bayesian network is a DAG, you just need to predict classes in their topological order.\n* There should be an equal sign instead of unequal in line 122.\n\n---\n\nEdit after the Authors' response: Since the authors addressed my main issue by conducting additional experiments, I'm raising my rating (to 6).\n\n---\n\nEdit after further discussion with AC and the Reviewers: Further discussion highlighted several issues with notation and clarity of the paper that led to differences in understanding of the proposed method among the Reviewers. After consideration, I think the paper requires major revision, and despite attractive results, should not be accepted to the conference. Because of that, I'm reverting my rating to the original one (5).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations were addressed, no issues with negative social impact were found.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues were found.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "8fLypehTKSq", "reviewer": "Reviewer_Bp9h", "summary": "This paper proposed Recurrent Bayesian Classifier Chains (RBCC) for exact multi-label classification. The proposed method learned a Bayesian network of class dependencies to improve the Recurrent Classifier Chains (RCCs). The comparative experiments were performed with RCC ([15], 2017, Nam, etc.), TS-RCC ([15], 2017, Nam, etc.), OF-RCC ([4], 2018 , Chen, etc.), and BCC ([23], 2014, Sucar). And the results demonstrated the effectiveness of the RBCC method.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The RBCC incorporates a Bayesian network into the recurrent classifier chain system.\n- A new algorithm is proposed for training and inference.\n- Experiments show that this method improves the performance of the mentioned methods.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "## Weakness:\n\n- In equation 1:\nThe left side of equation is a function of x, while the right side is the expectation on x, that means the right side is irrelevant to x.\n\n- line 16, ‘we outperform the state of the art methods’\nMore experiments should be conducted to compare with some latest methods proposed in 2019-2020.\n\n- page 5-6, Table 1-3\nThe improvement of this method is marginal.\n\n## Minor issues: \n\t- line 100:\n\tThe reference [22] needs to be mentioned only once.\n\n## Suggestion:\n- Some figures could be demonstrated to help the reader to understand the method and the advantage of the method, such as, the algorithm flow chart and the resulting figure which can illustrate the superiority of the proposed algorithm.\n\n", "review_text": "", "extra_scores": {"ethical_concerns": "na", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The algorithm introduced by the authors is sound and obtains promising results as given in the experimental results. The main problems pointed out by reviewers are its clarity and readability. All those problems are mainly caused by the used notation and typos in equations. Nevertheless, the authors were very responsive, clarified almost all issues, and shared with reviewers intended changes to be incorporated into the final version of the paper. Taking this into account we lean toward acceptance of the paper. 

Additional comments:
- Please also correct text and equations in lines 217-218: I suppose it should be $[0, 1]^L$ instead of $\\{0,1\\}^L$; it is better to define a loss function using all its parameters (true labels and predicted probabilities). 
Decision: Accept (Poster)