Recurrent Bayesian Classifier Chains for Exact Multi-Label Classification

Exact multi-label classification is the task of assigning each datapoint a set of class labels such that the assigned set exactly matches the ground truth . Optimizing for exact multi-label classification is important in domains where missing a single label can be especially costly , such as in object detection for autonomous vehicles or symptom classification for disease diagnosis . Recurrent Classifier Chains ( RCCs ) , a recurrent neural network extension of ensemble-based classifier chains , are the stateof-the-art exact multi-label classification method for maximizing subset accuracy . However , RCCs iteratively predict classes with an unprincipled ordering , and therefore indiscriminately condition class probabilities . These disadvantages make RCCs prone to predicting inaccurate label sets . In this work we propose Recurrent Bayesian Classifier Chains ( RBCCs ) , which learn a Bayesian network of class dependencies and leverage this network in order to condition the prediction of child nodes only on their parents . By conditioning predictions in this way , we perform principled and non-noisy class prediction . We demonstrate the effectiveness of our RBCC method on a variety of real-world multi-label datasets , where we routinely outperform the state of the art methods for exact multi-label classification . 1 Introduction . Exact multi-label classification ( E-MLC ) is the task of assigning multiple labels to a single instance , and is critical for a variety of of domains . For example , when detecting health conditions , one person will often experience multiple symptoms concurrently . Additionally , missing the detection of even one symptom can have drastic effects and so the entire label set must be predicted exactly correct . Missing the symptom Loss of Taste , for instance , might cause a downstream classifier to misdiagnose COVID-19 as the common cold [ 3 ] . While explicitly optimizing for exact classification is inordinately challenging , a successfully-trained model is highly rewarding as it is guaranteed to capture the joint probabilities between the features and all labels . There has been increasing interest in developing multi-label classification methods for E-MLC [ 17 , 19 , 18 , 13 , 7 , 28 , 15 , 4 , 24 ] . Most are based on classifier chains , which are ensembles of models where each classifier predicts a single class and the predictions from the i-th model are used as input into the i + 1-th model in order to learn inter-class dependencies . As the classifier chain requires an ordering of the classes a priori [ 19 ] , there are variations of classifier chains that utilize different class orderings [ 11 , 16 ] , such as Bayesian classifier chains ( BCCs ) [ 30 , 23 ] which utilize a Bayesian network to learn class orderings . A key disadvantage of these classier chain approaches is that they all require an ensemble of individual classifiers both requiring a large number of parameters and learning a different and independent latent representation of the input data per classifier . This makes such ensemble methods particularly prone to overfitting [ 20 , 1 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The leading extension of classifier chains is the recurrent classifier chain ( RCC ) , which models inter-class dependencies by using a recurrent network to iteratively predict each class , such that each class prediction is conditioned on the predictions of all previously predicted classes . The order in which classes are predicted is likewise important for RCCs [ 19 , 15 ] . To address this , the ordering is usually ad hoc with popular choices being to predict classes either frequent-to-rare or rare-to-frequent [ 15 ] , while some very recent RCCs attempt to learn an ordering for the classes [ 4 , 24 ] . However , there is no direct feedback on the optimal class ordering , and attempting to learn through reinforcement alone is a challenging task that is also prone to overfitting [ 29 ] . Additionally , RCCs inherently condition their class predictions on noisy inputs , since the predictions are conditioned on all previously-predicted classes even when most classes are independent of all other classes . The task of E-MLC is to train a model f , such that f can predict the exact label sets of previouslyunseen instances . Specifically , our goal is to use training data to obtain a classifier that can maximize the Subset Accuracy of unseen testing data , as the label set most likely to be exactly correct is the risk minimizer for Subset Accuracy . Thus , maximizing Subset Accuracy directly corresponds to optimizing for E-MLC . Subset Accuracy is a strict multi-label metric that returns 1 if the entire label set is correct , and 0 otherwise . There are four major challenges for E-MLC : • Critical label dependencies . Inter-class dependencies must be learned and modeled in order to perform E-MLC [ 7 ] . • Cascading class errors . When class predictions are made sequentially , early mistakes are likely to lead to additional errors for subsequent classes . • Strict metric . Subset Accuracy is only nonzero if the predicted label set is exactly correct , which means that a mostly-correct prediction is penalized as harshly as a completely incorrect solution during evaluation . This non-smooth metric is naturally challenging to maximize . • Exponential label set growth . The number of possible label sets grows exponentially with the number of classes . Thus , finding the label set that is exactly correct is akin to finding a needle in a haystack for large numbers of classes . We propose the Recurrent Bayesian Classifier Chain ( RBCC ) , the first recurrent classifier chain that avoids conditioning on noisy inter-class relationships , thereby learning a highly-principled ordering for class predictions . The RBCC consists of three core components . First , the RBCC explicitly learns inter-class dependencies using the Inter-class Dependency Network , which finds a Bayesian network that models the relationships between the classes . Second , this network structure is used by the Recurrent Conditional Dependency Model , which for each class produces a latent vector representation that summarizes the input features along with the classes upon which it depends . This representation is then passed on to the Bayesian-Conditioning Classifier , which produces the probability of observing each class . Importantly , the Bayesian-Conditioning Classifier ’ s predictions for a given class are only conditioned on the input features and the classes for which it is not conditionally independent ( which we refer to as the parent classes ) . Altogether , the RBCC thus produces accurate label sets , optimizing for exact multi-label classification by mitigating cascading errors . Contributions . The contributions of this work are as follows : • The RBCC is the first recurrent classifier chain method that utilizes conditional independence to perform non-noisy class conditioning . • We propose a novel training and inference algorithm for this approach . • Experimentally , we show that RBCCs significantly outperform state-of-the art methods for E-MLC on a variety of real-world datasets . 2 Methodology . 2.1 Problem Definition . Formally , we define the problem of E-MLC as follows : Let ( X , C1 , ... , CL ) ∼ PX×C1× ... ×CL , where X is the random variable of features and Ci is the ith class , such that there are L possible classes . If the observed value of Ci , ci , is 1 , then the ith class applies . Otherwise , ci = 0 and the class does not apply . Let D = { ( x , y ) i } Ni=1 be a dataset of i.i.d samples from PX×C1× ... ×CL , such that each x ∈ Rm is an observed value of X and y is a vector of length L of observed values for all classes , and thus [ y ] i ( the ith element of vector y ) is equal to ci . Our goal is to find a classifier fθ that maximizes the Subset Accuracy of unobserved test data . The Subset Accuracy ( SA ) of a prediction fθ ( x ) is defined as : SA ( fθ ( x ) , y ) : = Jy = fθ ( x ) K , where J·K is the conditional function such that JP K returns 1 if the predicate P is true , and is 0 otherwise . The risk minimizer for SA is given by : h∗ ( x ) = argmax y P ( C1 = [ y ] 1 , ... , CL = [ y ] L|X = x ) ( 1 ) Thus , our classifier fθ should aim to return the Maximum A Posteriori of the class probabilities given observations x . 2.2 Proposed method : RBCC . We propose the Recurrent Bayesian Classifier Chain ( RBCC ) , which improves upon RCCs by performing class prediction in a principled order with non-noisy class conditioning during prediction . We achieve this in part by learning a Bayesian network representation of inter-class dependencies . We note here that the focus of this work is not on a new method to construct a Bayesian network of class inter-dependencies , of which there are many existing methods [ 22 , 6 , 22 , 27 , 5 ] . Instead , the focus is on how to better incorporate a Bayesian network found through these existing methods into the recurrent classifier chain system . At a high level , RBCC aims to find to find the parameters of a classification model such that the predicted label vector is the most likely to exactly match the true label vector y . Thus , we are aiming to return the risk minimizer given in Equation 2 , and are as such optimizing for the exact goal of E-MLC . This is achieved through finding a model of P ( C1 , ... , CL|X ) parameterized by θ : θ∗ = argmax θ N∑ i=1 logP θ ( xi ; θ ) . ( 2 ) Then , the prediction for yi can be found be searching for the maximum of P θ ( xi ; θ ) . At a glance , one to model P ( C1 , ... , CL|X ) would be to factorize it into a product of per-class probabilities : P ( C1 , ... , CL|X ) = P ( C1|x ) L∏ i=2 P ( Ci|C < i , x ) , ( 3 ) as is done by [ 15 ] . However , there are two issues apparent with the above factorization . First , while in theory we can order the classes in the above product in any way if the exact true conditional probabilities are known , in practice the order of classes has an immense effect on classification performance [ 19 ] . For instance , if Cm is strongly dependent class Cn , then the prediction for cm should be conditioned on that class during training for more accurate predictions . However , depending on the ordering of the classes , cm may not be conditioned on cn in the factorization above . Second , each class Ck is conditioned on all of the preceding classes C < k , even if Ck is independent from most of these classes . Approaches that use the factorization shown in Equation 3 condition the prediction of ck on noise , risking overfitting to false correlations between classes during training . We can solve these two issues by assuming that the joint dependencies among all classes C1 , C2 , . . . , Cℓ and features X can be factorized as a directed acyclic graph ( DAG ) G. Each node in the graph corresponds to either a class Ci or the features X , and the edges encode the dependencies between nodes . As we make the standard multi-label assumption that all classes depend on the input features X , X is a parent of all other nodes in G , other than itself . Let PaG\X ( Ck ) be the parents of Ck , other than the universal parent X . We can thus simplify P ( C1 , ... , CL|X ) into the following : P ( C1 , C2 , . . . , CL|X ) = L∏ i=1 P ( Ci|PaG\X ( Ck ) , X ) . ( 4 ) The above equation does not have the flaws of the factorization in Equation 3 , as each class is now only conditioned on all of the classes upon which it depends . RBCC thus involves three primary steps to encode this key idea into one unified multi-label model : For each class we need to first identify its parent classes , summarize the information of the observations of the parent classes , and then make a prediction for that class based on the observations of its parents . In the first step , we learn the structure of the inter-class dependencies using the Inter-class Dependency Network , which fits a Bayesian network to the classes . This network does not tell us the strength of the dependencies between classes and does not alone allow us to infer which classes are associated with a given instance . Instead , the learned network identifies which classes a given class is dependent on ( the parent classes ) , and which classes it is independent of given the parent classes . This allows us to condition a down-stream classification model on only the required parent classes of each class , reducing cascading errors from incorrect class predictions . The input features and observations of the parent classes are summarized by the Recurrent Conditional Dependency Model , a recurrent neural network-based architecture , before this summarized representation is passed into the down stream classifier , which we refer to as the Bayesian-Conditioning Classifier . The Bayesian-Conditioning Classifier model makes predictions for each class Ck such that the predictions are conditioned only on the observed features x and the parent classes of Ck , according to the network structure learned from the Inter-class Dependency Network .

Summary: This paper proposes a new method, Recurrent Bayesian Classifier Chains (RBCC), for exact multi-label classification problems (maximizing subset accuracy). RBCC involves three steps: 1. Intra-class Dependency Network to learn a Bayes net to the classes; 2. Recurrent Conditional Dependency Model to summarize the information of features and the corresponding parent classes learned in the previous step; 3. Bayesian-Conditioning Classifier to predict. RBCC is shown to outperform several state-of-the-art (STOA) algorithms on some real-world multi-label data sets with respect to subset accuracy.