Recurrent Bayesian Classifier Chains for Exact Multi-Label Classification

Exact multi-label classification is the task of assigning each datapoint a set of class labels such that the assigned set exactly matches the ground truth . Optimizing for exact multi-label classification is important in domains where missing a single label can be especially costly , such as in object detection for autonomous vehicles or symptom classification for disease diagnosis . Recurrent Classifier Chains ( RCCs ) , a recurrent neural network extension of ensemble-based classifier chains , are the stateof-the-art exact multi-label classification method for maximizing subset accuracy . However , RCCs iteratively predict classes with an unprincipled ordering , and therefore indiscriminately condition class probabilities . These disadvantages make RCCs prone to predicting inaccurate label sets . In this work we propose Recurrent Bayesian Classifier Chains ( RBCCs ) , which learn a Bayesian network of class dependencies and leverage this network in order to condition the prediction of child nodes only on their parents . By conditioning predictions in this way , we perform principled and non-noisy class prediction . We demonstrate the effectiveness of our RBCC method on a variety of real-world multi-label datasets , where we routinely outperform the state of the art methods for exact multi-label classification . 1 Introduction . Exact multi-label classification ( E-MLC ) is the task of assigning multiple labels to a single instance , and is critical for a variety of of domains . For example , when detecting health conditions , one person will often experience multiple symptoms concurrently . Additionally , missing the detection of even one symptom can have drastic effects and so the entire label set must be predicted exactly correct . Missing the symptom Loss of Taste , for instance , might cause a downstream classifier to misdiagnose COVID-19 as the common cold [ 3 ] . While explicitly optimizing for exact classification is inordinately challenging , a successfully-trained model is highly rewarding as it is guaranteed to capture the joint probabilities between the features and all labels . There has been increasing interest in developing multi-label classification methods for E-MLC [ 17 , 19 , 18 , 13 , 7 , 28 , 15 , 4 , 24 ] . Most are based on classifier chains , which are ensembles of models where each classifier predicts a single class and the predictions from the i-th model are used as input into the i + 1-th model in order to learn inter-class dependencies . As the classifier chain requires an ordering of the classes a priori [ 19 ] , there are variations of classifier chains that utilize different class orderings [ 11 , 16 ] , such as Bayesian classifier chains ( BCCs ) [ 30 , 23 ] which utilize a Bayesian network to learn class orderings . A key disadvantage of these classier chain approaches is that they all require an ensemble of individual classifiers both requiring a large number of parameters and learning a different and independent latent representation of the input data per classifier . This makes such ensemble methods particularly prone to overfitting [ 20 , 1 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The leading extension of classifier chains is the recurrent classifier chain ( RCC ) , which models inter-class dependencies by using a recurrent network to iteratively predict each class , such that each class prediction is conditioned on the predictions of all previously predicted classes . The order in which classes are predicted is likewise important for RCCs [ 19 , 15 ] . To address this , the ordering is usually ad hoc with popular choices being to predict classes either frequent-to-rare or rare-to-frequent [ 15 ] , while some very recent RCCs attempt to learn an ordering for the classes [ 4 , 24 ] . However , there is no direct feedback on the optimal class ordering , and attempting to learn through reinforcement alone is a challenging task that is also prone to overfitting [ 29 ] . Additionally , RCCs inherently condition their class predictions on noisy inputs , since the predictions are conditioned on all previously-predicted classes even when most classes are independent of all other classes . The task of E-MLC is to train a model f , such that f can predict the exact label sets of previouslyunseen instances . Specifically , our goal is to use training data to obtain a classifier that can maximize the Subset Accuracy of unseen testing data , as the label set most likely to be exactly correct is the risk minimizer for Subset Accuracy . Thus , maximizing Subset Accuracy directly corresponds to optimizing for E-MLC . Subset Accuracy is a strict multi-label metric that returns 1 if the entire label set is correct , and 0 otherwise . There are four major challenges for E-MLC : • Critical label dependencies . Inter-class dependencies must be learned and modeled in order to perform E-MLC [ 7 ] . • Cascading class errors . When class predictions are made sequentially , early mistakes are likely to lead to additional errors for subsequent classes . • Strict metric . Subset Accuracy is only nonzero if the predicted label set is exactly correct , which means that a mostly-correct prediction is penalized as harshly as a completely incorrect solution during evaluation . This non-smooth metric is naturally challenging to maximize . • Exponential label set growth . The number of possible label sets grows exponentially with the number of classes . Thus , finding the label set that is exactly correct is akin to finding a needle in a haystack for large numbers of classes . We propose the Recurrent Bayesian Classifier Chain ( RBCC ) , the first recurrent classifier chain that avoids conditioning on noisy inter-class relationships , thereby learning a highly-principled ordering for class predictions . The RBCC consists of three core components . First , the RBCC explicitly learns inter-class dependencies using the Inter-class Dependency Network , which finds a Bayesian network that models the relationships between the classes . Second , this network structure is used by the Recurrent Conditional Dependency Model , which for each class produces a latent vector representation that summarizes the input features along with the classes upon which it depends . This representation is then passed on to the Bayesian-Conditioning Classifier , which produces the probability of observing each class . Importantly , the Bayesian-Conditioning Classifier ’ s predictions for a given class are only conditioned on the input features and the classes for which it is not conditionally independent ( which we refer to as the parent classes ) . Altogether , the RBCC thus produces accurate label sets , optimizing for exact multi-label classification by mitigating cascading errors . Contributions . The contributions of this work are as follows : • The RBCC is the first recurrent classifier chain method that utilizes conditional independence to perform non-noisy class conditioning . • We propose a novel training and inference algorithm for this approach . • Experimentally , we show that RBCCs significantly outperform state-of-the art methods for E-MLC on a variety of real-world datasets . 2 Methodology . 2.1 Problem Definition . Formally , we define the problem of E-MLC as follows : Let ( X , C1 , ... , CL ) ∼ PX×C1× ... ×CL , where X is the random variable of features and Ci is the ith class , such that there are L possible classes . If the observed value of Ci , ci , is 1 , then the ith class applies . Otherwise , ci = 0 and the class does not apply . Let D = { ( x , y ) i } Ni=1 be a dataset of i.i.d samples from PX×C1× ... ×CL , such that each x ∈ Rm is an observed value of X and y is a vector of length L of observed values for all classes , and thus [ y ] i ( the ith element of vector y ) is equal to ci . Our goal is to find a classifier fθ that maximizes the Subset Accuracy of unobserved test data . The Subset Accuracy ( SA ) of a prediction fθ ( x ) is defined as : SA ( fθ ( x ) , y ) : = Jy = fθ ( x ) K , where J·K is the conditional function such that JP K returns 1 if the predicate P is true , and is 0 otherwise . The risk minimizer for SA is given by : h∗ ( x ) = argmax y P ( C1 = [ y ] 1 , ... , CL = [ y ] L|X = x ) ( 1 ) Thus , our classifier fθ should aim to return the Maximum A Posteriori of the class probabilities given observations x . 2.2 Proposed method : RBCC . We propose the Recurrent Bayesian Classifier Chain ( RBCC ) , which improves upon RCCs by performing class prediction in a principled order with non-noisy class conditioning during prediction . We achieve this in part by learning a Bayesian network representation of inter-class dependencies . We note here that the focus of this work is not on a new method to construct a Bayesian network of class inter-dependencies , of which there are many existing methods [ 22 , 6 , 22 , 27 , 5 ] . Instead , the focus is on how to better incorporate a Bayesian network found through these existing methods into the recurrent classifier chain system . At a high level , RBCC aims to find to find the parameters of a classification model such that the predicted label vector is the most likely to exactly match the true label vector y . Thus , we are aiming to return the risk minimizer given in Equation 2 , and are as such optimizing for the exact goal of E-MLC . This is achieved through finding a model of P ( C1 , ... , CL|X ) parameterized by θ : θ∗ = argmax θ N∑ i=1 logP θ ( xi ; θ ) . ( 2 ) Then , the prediction for yi can be found be searching for the maximum of P θ ( xi ; θ ) . At a glance , one to model P ( C1 , ... , CL|X ) would be to factorize it into a product of per-class probabilities : P ( C1 , ... , CL|X ) = P ( C1|x ) L∏ i=2 P ( Ci|C < i , x ) , ( 3 ) as is done by [ 15 ] . However , there are two issues apparent with the above factorization . First , while in theory we can order the classes in the above product in any way if the exact true conditional probabilities are known , in practice the order of classes has an immense effect on classification performance [ 19 ] . For instance , if Cm is strongly dependent class Cn , then the prediction for cm should be conditioned on that class during training for more accurate predictions . However , depending on the ordering of the classes , cm may not be conditioned on cn in the factorization above . Second , each class Ck is conditioned on all of the preceding classes C < k , even if Ck is independent from most of these classes . Approaches that use the factorization shown in Equation 3 condition the prediction of ck on noise , risking overfitting to false correlations between classes during training . We can solve these two issues by assuming that the joint dependencies among all classes C1 , C2 , . . . , Cℓ and features X can be factorized as a directed acyclic graph ( DAG ) G. Each node in the graph corresponds to either a class Ci or the features X , and the edges encode the dependencies between nodes . As we make the standard multi-label assumption that all classes depend on the input features X , X is a parent of all other nodes in G , other than itself . Let PaG\X ( Ck ) be the parents of Ck , other than the universal parent X . We can thus simplify P ( C1 , ... , CL|X ) into the following : P ( C1 , C2 , . . . , CL|X ) = L∏ i=1 P ( Ci|PaG\X ( Ck ) , X ) . ( 4 ) The above equation does not have the flaws of the factorization in Equation 3 , as each class is now only conditioned on all of the classes upon which it depends . RBCC thus involves three primary steps to encode this key idea into one unified multi-label model : For each class we need to first identify its parent classes , summarize the information of the observations of the parent classes , and then make a prediction for that class based on the observations of its parents . In the first step , we learn the structure of the inter-class dependencies using the Inter-class Dependency Network , which fits a Bayesian network to the classes . This network does not tell us the strength of the dependencies between classes and does not alone allow us to infer which classes are associated with a given instance . Instead , the learned network identifies which classes a given class is dependent on ( the parent classes ) , and which classes it is independent of given the parent classes . This allows us to condition a down-stream classification model on only the required parent classes of each class , reducing cascading errors from incorrect class predictions . The input features and observations of the parent classes are summarized by the Recurrent Conditional Dependency Model , a recurrent neural network-based architecture , before this summarized representation is passed into the down stream classifier , which we refer to as the Bayesian-Conditioning Classifier . The Bayesian-Conditioning Classifier model makes predictions for each class Ck such that the predictions are conditioned only on the observed features x and the parent classes of Ck , according to the network structure learned from the Inter-class Dependency Network .

Summary: This paper presents a new method for optimizing the subset zero-one loss in multi-label classification. The new method is an adaptation of probabilistic classifier chains. The central idea is to replace the chain by a dependence structure learned using Bayesian networks. In theory the order of the labels in a probabilistic classifier chain does not matter, but in practice one can observe performance drops if the chosen order leads to propagation of errors. By replacing a random order with the dependency structure of a Bayesian neural network, the authors intend to solve the problem of choosing an appropriate order.   As underlying models to fit probabilities, recurrent neural networks are used. Experimental results are reported for three benchmark datasets.