Modeling Label Space Interactions in Multi-label Classification using Box Embeddings

1 INTRODUCTION . Multi-label classification is a machine learning task in which an input is associated with multiple categories . Many real-world multi-label classification datasets in modalities such as text categorization ( Lewis et al. , 2004 ) , image classification ( Lin et al. , 2015 ; Krishna et al. , 2016 ) , entity typing ( Murty et al. , 2018 ; Onoe et al. , 2021 ) , functional genomics ( Barutcuoglu et al. , 2006 ; Clare , 2003 ) , and so on , have a rich inter-dependent label structure that can be expressed using a taxonomy graph or a hierarchy . To be useful in practice , a model should produce predictions that are coherent with respect to the label taxonomy . For example , if a book is classified as drama , it should also be classified as fiction according to the label taxonomy of book genres in the left-hand side of Figure 1 . More formally , we are given a label taxonomy in the form of a directed acyclic graph G = ( L , T ) , where ( a , b ) ∈ T if and only if a is a parent of b in the taxonomy . A model will assign scores sℓ ( x ) for each label ℓ ∈ L. We say the scores are coherent with respect to the taxonomy if , for all edges ( a , b ) ∈ T , sa ( x ) ≥ sb ( x ) , and the model is consistent if this is the case for all inputs x . In the case of the book genre classification example , this implies that sfiction must be greater than or equal to sdrama , regardless of the input . The problem of producing coherent predictions for multi-label classification has garnered a lot of attention in the machine learning literature ( Wehrmann et al. , 2018c ; Giunchiglia & Lukasiewicz , 2020 ; Murty et al. , 2018 ; Cerri et al. , 2014 ) . Most methods that proposed to improve the coherence in predictions require the complete label taxonomy at inference time , and occasionally at training time as well , making these models hard to scale to large label spaces ( Giunchiglia & Lukasiewicz , 2020 ; Wehrmann et al. , 2018c ) . This brings forth a question : Can we utilize representation learning to model the label-label relationships implicitly in the embedding space ? Vilnis et al . ( 2018 ) show that probabilistic box embeddings , which represent concepts as high dimensional hyper-rectangles , can embed DAGs efficiently using the explicit information about the edges . Box embeddings represent edges by box-box containment as shown in Figure 1b . Representing the input and output labels in the same geometric space of boxes allows the multi-label taxonomy to be learned without an explicit taxonomic training signal . Moreover , there exists a large space of possible configurations that represent the same taxonomy , and if the label embeddings in the model gets close to any such configuration , then the model will always produce classifications that are coherent w.r.t the taxonomy , regardless of the input . We show through empirical evidence this is the case , and provide a formal proof for latter . In this work , we propose the multi-label box model ( MBM ) that utilizes the geometry and probabilistic semantics of box embeddings to model label-label interactions in multi-label classification . MBM represents labels as boxes using free parameters and uses a deep neural network to embed the inputs in the same space . We perform coherence analysis of the model using two measures and show that MBM not only achieves state-of-the-art predictive performance but it also significantly improves the coherence of predicted scores w.r.t latent label taxonomy . Our analysis further shows that it is possible to retrieve the latent label-label relationships solely by analysing the learnt label representations inside the MBM , endowing the model with high degree of interpretability . Finally , we also present a way to utilize the interpretability of MBM to inject partial information about label-label relationships into the model thereby improving the coherence even further . 2 RELATED WORK . Multi-label classification tasks that exhibit strong label space structure in the form of explicit label taxonomy are termed hierarchical multi-label classification ( HMLC ) in machine learning literature . Most approaches for such tasks make use of the complete hierarchy at training time . These approaches can be categorized into two buckets ( Silla & Freitas , 2010 ) : ( 1 ) Local approaches that focus on local information for each label or clusters of labels in the hierarchy and classify them independently ( Cerri et al. , 2014 ; Huang et al. , 2019 ) , and ( 2 ) Global approaches that treat the problem as a structured classification task and take global interactions into account ( Belanger & McCallum , 2016 ) . In the most general setting , however , both local and global interactions between labels exist . The recent advances in deep learning ( Wehrmann et al. , 2018a ) propose a specialized neural network architecture called Hierarchical Multi-Label Classification Network ( HMCN-R and HMCN-F ) that takes into account both local and global interactions by creating an ensemble of classifiers that can be trained using end-to-end gradient based training . However , HMCN does not try to enforce coherence strongly , focusing solely on predictive performance . In order to improve prediction coherence , recent works employ special loss functions on top of a neural network classifier to enforce coherence w.r.t the label taxonomy ( Murty et al. , 2018 ; Giunchiglia & Lukasiewicz , 2020 ) . While effective , these approaches still use the label taxonomy explicitly , making them difficult to scale to very large label spaces . Recent advances in representation learning provide various methods to embed large graphs and taxonomies parsimoniously in non-euclidean spaces ( Nickel & Kiela , 2017 ; Vilnis et al. , 2018 ; Vilnis & McCallum , 2015 ; Gu et al. , 2019 ) . The most prominent of these embedding methods include hyperbolic embeddings ( Nickel & Kiela , 2017 ; Ganea et al. , 2018a ; b ) and box embedding ( Vilnis et al. , 2018 ; Li et al. , 2019 ; Dasgupta et al. , 2020a ) . The use of representations other than Euclidean vectors for improving the coherence of multi-label classification has been limited to specific domains like text ( Chatterjee et al. , 2021 ) or specific tasks like entity typing ( Onoe et al. , 2021 ) . Moreover , while both hyperbolic and box embedding can model hierarchical relationships , it has been shown that the box embedding can also model more general graphs like DAGs much more efficiently than hyperbolic embeddings ( Patel et al. , 2020 ) . Hence , we propose a model that uses box embedding to capture general label-label relationships without the explicit use of label taxonomy to improve the coherence of model predictions . 3 OVERVIEW OF BOX EMBEDDINGS . Notations : In the problem of multi-label classification , we are given a set of labels L where L = |L| , and an instance can be labeled with an element s ∈ { 0 , 1 } L , where projection to the ith coordinate πi ( s ) = 1 means that the ith label is true . We call the set of all such labelings S , and the associated probability space ( S , P ( S ) , PS ) . We use I to denote the set of all finite closed intervals [ µ− , µ+ ] in Ω ⊂ R plus the empty set , i.e . I : = { [ µ− , µ+ ] ⊂ Ω |µ+ ≥ µ− } ∪ ∅ . We denote the smallest σ-algebra containing I as σ ( I ) and , given a valid finite measure ν , we consider the measure space ( Ω , σ ( I ) , ν ) . As a high dimensional generalization , Id will denote a d-dimensional Cartesian product of I . Definition 1 ( Box Embedding ( Vilnis et al. , 2018 ) ) . Let B : Id → S be a measurable function such that B−1 ◦ π−1i ( 1 ) = ∏d i [ µ − i , µ + i ] ∈ Id . A box embedding is defined as the function Box : L → Id which maps a label ℓ ∈ L to B−1 ◦ π−1ℓ ( { 1 } ) ∈ Id . The definition of box embeddings induces a push-forward measure Q on S such that for any R ⊆ S , Q ( S ) = ν ◦ B−1 ( R ) . The complete joint probability distribution over the labels can be modeled using Q as defined above ; however , computing B−1 ( R ) requires the use of inclusion-exclusion principle and hence is intractable for a general R. In order to avoid local identifiability issues in training , Dasgupta et al . ( 2020b ) interpret µ−i ( resp . µ+i ) as the location parameters of random variables M − i ( resp . M + i ) that are distributed according to GumbelMax ( resp . GumbelMin ) distributions , leading to a meta-probabilistic generalization of box embedding which they call Gumbel Box Process . Since GumbelMax ( resp . GumbelMin ) is a max ( resp . min ) stable distribution , it enables the computation of the location parameters of the intersection box as given in the following definition . Definition 2 ( Intersection Box ( Dasgupta et al. , 2020b ) ) . Let A = ∏d i=1 [ a − i , a + i ] and B =∏d i=1 [ b − i , b + i ] be two gumbel boxes expressed using their location parameters , then the location parameters of the intersection of these two gumbel boxes are given as A∩̃B = d∏ i=1 [ β lse ( a−i β , b−i β ) , − β lse ( −a + i β , −b + i β ) ] , ( 1 ) where lse ( x , y ) = log ( exp ( x ) + exp ( y ) ) . The expected volume of Gumbel boxes involves the Bessel Function of the Second Kind , however , as shown in Dasgupta et al . ( 2020b ) , this integral can be reasonably approximated using softplus function leading to the following definition for approximate bessel volume . Definition 3 ( Approximate Bessel Volume ( Dasgupta et al. , 2020b ) ) . For a gumbel box B =∏d i=1 [ b − i , b + i ] we define the approximate Bessel volume λ : Id → R+ as λ ( B ) : = d∏ i=1 log ( 1 + exp ( b+i − b − i β − 2γ ) ) . In the next section , we formally demonstrate the suitability of box embeddings for capturing taxonomic label relationships , and for that we first prove a couple of useful facts regarding the Gumbel intersection and Bessel approximate volume . Proposition 1 . Approximate bessel volume is monotonic with respect to set containment . That is for two Gumbel boxes A , B , a−i ≥ b − i and a + i ≤ b + i , ∀i ∈ { 1 , . . . , d } ⇐⇒ λ ( A ) ≤ λ ( B ) . ( 2 ) Proof . Follows from the monotonicity of log ( 1 + exp ( . ) ) . Proposition 2 . For any two Gumbel boxes A , B , λ ( A∩̃B ) ≤ λ ( B ) . Proof . The fact that max ( x , y ) ≤ lse ( x , y ) , and the statement of proposition 1 together imply the desired result . Since λ is neither normalized nor additive , it can not be used as a probability measure on ( Ω , σ ( Id ) ) . However , we can use proposition 1 and 2 to define a conditional probability model as follows . Corollary 1 . For two gumbel boxes A , B , let PBox ( A |B ) = λ ( A∩̃B ) λ ( B ) , then ( i ) For any two gumbel boxes A , B , we have 0 ≤ PBox ( A |B ) ≤ 1 . ( ii ) PBox ( A |C ) ≤ PBox ( B |C ) for any three gumbel boxesA , B , C , with a−i ≥ b − i , a + i ≤ b + i .

Summary: Most existing multi-label classification methods predict labels without considering natural/latent taxonomic constraints. This work introduces the multi-label box model (MBM), a method for multi-label classification that combines the encoding power of neural networks with the inductive bias and probabilistic semantics of box embeddings (trainable Venn-diagrams based on hyper-rectangles). MBM has some nice properties: they can be trained via gradient descent from data, and box embeddings can be read as calibrated conditional probabilities. Experiments are conducted on seven benchmark datasets with promising results, and results suggest that the label embeddings seem to capture the latent label taxonomy.