Modeling Label Space Interactions in Multi-label Classification using Box Embeddings
Authors: Dhruvesh Patel
Pavitra Dangati
Jay-Yoon Lee
Michael Boratko
Andrew McCallum
Venue: ICLR 2022 Poster (2022)
Keywords: Multi-label classification
Box Embeddings
Representation Learning
Embeddings

Abstract: Multi-label classification is a challenging structured prediction task in which a set of output class labels are predicted for each input. Real-world datasets often have natural or latent taxonomic relationships between labels, making it desirable for models to employ label representations capable of capturing such taxonomies. Most existing multi-label classification methods do not do so, resulting in label predictions that are inconsistent with the taxonomic constraints, thus failing to accurately represent the fundamentals of problem setting. In this work, we introduce the multi-label box model (MBM), a multi-label classification method that combines the encoding power of neural networks with the inductive bias and probabilistic semantics of box embeddings (Vilnis, et al 2018).  Box embeddings can be understood as trainable Venn-diagrams based on hyper-rectangles.  Representing labels by boxes rather than vectors, MBM is able to capture taxonomic relations among labels.  Furthermore, since box embeddings allow these relations to be learned by stochastic gradient descent from data, and to be read as calibrated conditional probabilities, our model is endowed with a high degree of interpretability. This interpretability also facilitates the injection of partial information about label-label relationships into model training, to further improve its consistency. We provide theoretical grounding for our method and show experimentally the model's ability to learn the true latent taxonomic structure from data. Through extensive empirical evaluations on both small and large-scale multi-label classification datasets, we show that BBM can significantly improve taxonomic consistency while preserving or surpassing the state-of-the-art predictive performance.

Published as a conference paper at ICLR 2022
MODELING LABEL SPACE INTERACTIONS IN MULTI- LABEL CLASSIFICATION USING BOX EMBEDDINGS
Dhruvesh Patel, Pavitra Dangati, Jay- Yoo- Lee, Michael Boratko, Andrew McCallum Manning College of Information & Computer Sciences University of Massachusetts Amherst (dhruveshpate, sdangati, jaylee, mboratko, mccallum) uus.umass.edu
ABSTRACT
Multi-label classification is challenging structured prediction task in which a set of output class labels are predicted for each input. Real-world datasets often have taxonomic relationships between labels which can be explicit, implicit, or partially observed. Most existing multi-label classification methods either ignore the label taxonomy require the complete specification of the taxonomy at training and inference time to enforce coherence their predictions. In this work we intro- duce the multi-label box model (MBM), multi-label classification method that combines the encoding power of neural networks with the inductive bias of proba- bilistic box embeddings (Vilnis et al. 2018), which can be understood trainable Venn-diagrams based on hyper-rectangles, By representing labels as boxes, MBM is able capture taxonomic relations among labels without them being provided explicitly. Furthermore, since MBM learns the label-label relationships from data and represents them as calibrated conditional probabilities. it provides high de- gree of interpretability. This interpretability also facilitates the injection of partial information about label-label relationships into model training, further improve its consistency. We provide theoretical grounding for our method and show ex- perimentally the model's ability to learn the true latent taxonomic structure from data. Through extensive empirical evaluations twelve multi-laber classification datasets, we show that MBM can significantly improve taxonomic consistency while maintaining the sfate-of-the-rr predictive performance
INTRODUCTION
Multi-label classification is a machine learning task which an input associated with multiple categories. Many real-world multi-label classification datasets in modalities such text categoriza-
tion (Lewis et al. 2004), image classification (Lin et al. 2015 Krishna et al. 2016). entity typing
(Murty al.][201 [0188] Onoe et al. (02T), functional genomics (Barutcuoglu et al. 2006 Clare (003), and so have a rich inter-dependent label structure that can be expressed using a taxonomy graph or a hierarchy. To be useful practice, model should produce predictions that are coherent with respect to the label taxonomy. For example, if a book is classified as drama, it should also be clas- sified as fiction according to the label taxonomy of book genres in the left-hand side of Figure T More formally,
given a label taxonomy in the form of a directed acyclic graph G (C, T) where (a, b) E T and only if a a parent of b the taxonomy. A model will assign scores sy (x)) for each label & L. We say the scores are coherent with respect to the taxonomy if, for all edges (a, b) E T, (x() Z sb(x), and the model is consistent if this is the case for all inputs ε. In the case of the book genre classification example, this implies that Station must be greater than or equal to 8drama- regardless of the input.
The problem of producing coherent predictions for multi-label classification has garnered lot of attention in the machine learning literature (Wehrmann et al. 2018a| Giunchiglia & Lukasiewicz) 2020 Murty et al. 2018 Cerri et al. 2014). Most methods proposed to improve the coherence in predictions require the complete label taxonomy at inference time, and occasionally at training time
'The code and implementation details are available at https: tttt:p://gitubb.... thub. www.b..commiiee com/iesl/bon /bbo-ollcc
iel 2022

---

Published as a conference paper at ICLR 2022
Drama
Fiction
Science
Erama
Fution
Uhosk
Nedionl
Soloor
X,
Classics
Backagy
Drama
Figure I: Left: A label taxonomy represented as a DAG, Center: A 2-dimensional box embedding of this taxonomy. Right: Scores assigned by the box embedding model to each label for two inputs.
as well, making these models hard to scale to large label spaces (Giunchiglia & Lukasiewicz 2020); Wehrmann al. 2018b). This brings forth question: Can we utilize representation learning to model the label- label relationships implicitly in the embedding space?
Vilnis et 2018) introduced probabilistic box embeddings, which represent concepts as high di- mensional hyper-rectangles and demonstrated that they can embed DAGs efficiently using explicit information about the edges. Box embeddings represent edges by box-box containment as shown in Figure[p. Representing the input and output labels in the same geometric space of boxes allows the multi-Tabel taxonomy to be learned without an explicit taxonomic training signal. Moreover, there exists a large space of possible configurations that represent the same taxonomy, and if the label embeddings in the model gets close to any such configuration, then the model will always pro- duce classifications that are coherent w.r.t the taxonomy, regardless of the input. We show through empirical evidence this is the case, and provide a formal proof for latter.
In this work, we propose the multi-label box model (MBM) that utilizes the geometry and proba- bilistic semantics of box embeddings to model label-label interactions multi-label classification. MBM represents labels boxes using free parameters and uses deep neural network embed the inputs in the same space. We perform coherence analysis of the model using two measures and show that MBM not only achieves state-of-the-ee predictive performance but also significantly improves the coherence of predicted scores w.r.t latent label taxonomy. Our analysis further shows that it is possible to retrieve the latent label-label relationships solely by analysing the learnt label representations inside the MBM, endowing the model with high degree of interpretability. Finally, we also present a way to utilize the interpretability MBM inject partial information about label-label relationships into the model thereby improving the coherence even further.
RELATED WORK
Multi-label classification tasks that exhibit strong label space structure in the form explicit label taxonomy are termed hierarchical multi-label classification (HMLC) in machine learning literature. Most approaches for such tasks make use of the complete hierarchy at training time. These ap- proaches be categorized into two buckets (Silla & Freitas 2010): (1) Local approaches that focus on local information for each label or clusters of labels in the hierarchy and classify them in- dependently (Cerri al.) 2014 Huang et al. 2019). and (2) Global approuches that treat the problem as a structured classification task and take global interactions into account (Belanger & McCallum 2016). In the most general setting. however, both local and global interactions between labels exist. The recent advances in deep learning (Wehrmann et al., 2018a) propose a specialized neural network architecture called Hierarchical Multi-Label Classification Network (HMCN-R and HMCN-F) that takes into account both local and global interactions by creating ensemble of classifiers that can be trained using end-to-end gradient based training. However, HMCN does not try to enforce coher- ence strongly, focusing solely on predictive performance. In order to improve prediction coherence, recent works employ special loss functions on top of neural network classifier to enforce coherence w.n. the label taxonomy (Murty al. 2018 Giunchiglia & Lukasiewicz) 2020). While effective, these approaches still use the label taxonomy explicitly, making them difficult to scale to very large label spaces.
2

---

Published as a conference paper at ICLR 2022
Recent advances in representation learning provide various methods to embed large graphs and tax- onomies parsimoniously in non-euclidean spaces, The most prominent of these embedding methods include hyperbolic embeddings (Nickel & Kiela 2017 Ganea al. 2018a b) and box embedding (Vilnis et al., 2018: Dasgupta et al. 2021a). The use of representations other than Fuclidean vec- tors for improving the coherence of multi-label classification has been limited specific domains like text (Chatterjee et al. (021) or specific tasks like entity typing (Onoe et al. (021). Moreover, while both hyperbolic and box embedding can model hierarchical relationships, it has been shown that the box embedding can also model more general graphs like DAGs much more efficiently than hyperbolic embeddings (Patel et al., 2020 Boratko et al., 2021 Dasgupta et al. 2021b). Hence, we propose model that uses box embedding to capture general label-label relationships without the explicit use of label taxonomy to improve the coherence of model predictions.
OVERVIEW OF BOX EMBEDDINGS
Notations: In the problem of multi-label classification, we are given set of labels L where = [ci and an instance can be labeled with an element s E {0,1]* where projection to the ith coordinate (s)
means that the ith label is true. We call the set of all such labelings S, and the associated probability space (S, P(S), Ps). We use T to denote the set of all finite closed inter- vals ,+=] in Ω C R plus the empty set, i.e. I == {[r .jrt] C Ωt 1ut 1 U. We denote the smallest o-algebra containing as (Il) and, given a valid finite measure v, we consider the measure space (9, (1), As a high dimensional generalization, Id will denote d-dimensional Cartesian product of I.
Definition (Box Embedding (Vilnis et al.
2018 Let :: S be measurable function such that B- 1 क. (1) II. |4 14 E Idd box embedding is defined as the function Box - Id which maps label L to B T ({13) E ld.
The definition of box embeddings induces push-forward measure Q on S such that for any R C S, Q(S) o B 1-1---118 (R). The complete joint probability distribution over the labels can be modeled using Q as defined above; however, computing B (R)( requires the use of inclusion-exclusion principle and hence intractable for a general R.
In order to avoid local identifiability issues in training, Dasgupta et al. (0200) interpret 14 (resp- wit) as the location parameters of random variables M,- (resp. Mf+) that are distributed according to GumbelMax (resp. GumbelMin) distributions, leading to meta-probabiliitic generalization of box embedding which they call Gumbel Box Process. Since GumbelMax (resp. GumbelMin) is a max (resp. min) stable distribution, it enables the computation of the location parameters of the intersection box given in the following definition.
Definition (Intersection Box (Dasgupta al. 2020)). Let A IT , and B 1b, 6,7] be two gumbel boxes expressed using their location parameters, then the location parameters of the intersection of these two gumbel boxes given
16
ag b
-Blse (- a,t
II lse (=1
3
B
where Ise(x, log(exp(2)) (z) + exp(y)).
The expected volume Gumbel boxes involves the Bessel Function the Second Kind, however, as shown in Dasgupta et al.) (2020), this integral can be reasonably approximated using softplus function leading to the following definition for approximate bessel volume.
Definition 3 (Approximate Bessel Volume (Dasgupta et al. 2020)). For gumbel box B
b, 6,*] we define the approximate Bessel volume A: Idd - R.
A(B)
II 3 log (1
(1+ exp
(1 (1+
( b b, 27) )
In the next section, formally demonstrate the suitability of box embeddings for capturing taxo- nomic label relationships, and for that we first state a couple of useful facts regarding the Gumbel intersection and Bessel approximate volume.
3

---

Published as a conference paper at ICLR 2022
Proposition 1. Approximate bessel volume is monotonic with respect containment. That is for two Gumbel boxes A, B,
0; 2 b, and a S6, ei (l,.... = X(A) S X(B..
Proof. Follows from the monotonicity of log(1 exp(.).
Proposition 2. For any two Gumbel boxes A, B, X(AΩB) S (B))
Proof. The fact that max(æ, v) Ise(x, y). and the statement of proposition together imply the desired result.
L
Since is neither normalized nor additive, it cannot be used probability measure (2, (I*).. However, we can use proposition I and 2 to define a conditional probability model as follows.
Corollary 1. For two gumbel boxes A, B, let Poxx (A B)
X(ANB)
then
A(B)
(i) For any two gumbel boxes A, B, have 0 S Pox (A (AIB) B3 S 1.
(ii) PBox (A|C) S PBox(B C*) C for any three gumbel boxes A, B, C. with a; 2 bj at Sb.
4 MULTI-LABEL. BOX MODEL
In order to perform the task of multi-label classification we need to model the conditional probabil- ities P where Y e S and X is the input. Using definition Box L - 1 as
|
we define label box embeddings
Box (fi) I Vi, log(1 exp utt,),
where xb_ +bt RLdd trainable parameters. The input instance X is encoded fixed- width element of I using a parametric instance box embedding Box, = Id o Fo X' + Id, where Fo X' + Rd is a neural network with parameters 0 and _d Rg + 1d defined
(x) [ 8, 6J,
where 5 10-5 The conditional probability for Y S given input X is computed using condi- tional probability under the Gumbel box model as
PMBM (YaM(Y)) : = II PMBM (Y, 0. II Poxx B प,." ((Y)) Box (X))
Using the definition of PBox as stated through corollary [ we get the following expression for the conditional probability of Y; under the model, where the intersection ñ is the Gumbel Intersection and measure À is Approximate Bessel Volume.
A (Box, & (4)) â Boxa(X)) X(Boxg(X))
PMBM 1/ =
4.1 MODELING LABEL-SPACE INTERACTIONS
In Section 1 alluded the fact that the inductive bias of the MBM allows it efficiently model partially specified first-order label interactions. Now we make this remark more concrete. the partial specification of label interaction is defined using taxonomy that be represented directed acyclic graph (DAG), the following proposition shows that MBM has strong inductive bias towards maintaining coherence its scores.
Proposition 3. Let G (C, T) denote a DAG defined over the labels where L is the set of all labels 1 1 1) 1} is the of edges. Then there exists some for all x A.
and T= ((e, E,) E, Eg E L. Pp(ı
1) such that P(.. ()) (y: = 1[ 2 Pi
Proof. For all (L,, { T, let v0 be such that (e,) C (oxx Note that such ? exists since for each i E (I...., L)). (6) is defined using only ę);. It follows from corollary that Poox PBoxxBoxx (f,) Box (Xx) Prax (Bxx(Box (Bxx((g)) [Bxx (X)) for any X, 0.
4

---

Published as a conference paper at ICLR 2022
4.2 LEARNING
The entire MBM is specified using parameters (vb, 0) where v & RR are the label embed- ding parameters and 0 the parameters of the instance encoder neural network Fe. Given data D {(x(1) yy"))..... (x))).,((()),,,, (x(N), y())) the model parameters are learnt by minimizing negative log- likelihood loss
D
La (v,, (,00; - ΣΣ log P(u, |xx'; 30,
(3)
using the ADAM optimizer (Kingma & Ba 2017). In order to empirically verify the intuition behind proposition 3 we also propose the use of label interaction loss
La(v) =- Σ +(t,,))
Σ (li,
(L,G)ET
(G,G)TT
that utilizes the geometry of box embeddings to inject partial information about label interactions specified using a label taxonomy G (C,T), T). When label interaction loss is applied (MBM-T), the total loss is Lr = Lull + vLGg where v a hyperparameter. For the Box model, label interaction score for pair of labels is defined as
SMBM (f, = log PBxx (f:) (xx))(1)))
(5)
5 BASELINES
Our choice of baselines reflects the focus of this work, i.e., introducing coherence prediction us- ing suitable representation spaces. To this end, baselines consist of high-performing neural network that only uses Euclidean vector representations (MVM), and another that uses hyperbolic representations (MHM). In order test the importance of the probabilistic semanties used formu- late MBM, we also include as baseline, a non-probabilistic box model as defined in Abboud et al. (2020). The base input encoder architecture Fg in all these baselines same as the one used in MBM.
Multi-label Vector Model (MVM) An input encoder neural network Fe X Rd is used to encode the inputs and label embedding matrix ub is used to represent the labels. The conditional probability of labels given the input and label interaction scores are given as
PMvv((1 1 sb. 0) o (FF(" Ma v/r), and $MVM(4,4) == o Mr,))
respectively, where a is the logistic sigmoid function, and Ga E Rdxd a matrix of trainable parameters. The parameters (8, w) are learnt through LT (Eq. 3 Note that, when label interaction not being explicitly modeled i.e., // = 0, M fixed as the identity matrix and MVM reduces to a special case of multi-layer perceptron.
Multi-label Hyperbolic Model (MHM) As discussed in Chatterjee et al. 2021 the isometry be- tween the Lorentz model and Poincare disk model for hyperbolic geometry be used produce the retraction formula used to project d-dimensional euclidean vectors to d-dimensional Poincaré ball
Bdd
{x Rd
|| < 1}. Here the projection operator II Rd + Bdd and the distance
d: Bd
Bd
R2 the hyperbolic space are given as:
|
wl
[1x)
d(u, v) arcosh 1
1+
1
(1 1|
|
Specifically, the input is first encoded using Fo and then projected into Bd. The unnormalized score for each label is computed as the negative of the distance between the hyperbolic projections of encoded input and label representation. Since the hyperbolic distance function consists of arcosh, the negative distance interpreted as log-probability score. Hence, the conditional prg-robbbyylity of the labels and the label interaction score is given, respectively, by the following expressions, which are then used to learn the parameters (1b, 8) in the same way as MVM and MBM.
log PAIHM (v1- 1 1=) -((T(((())) (vhr)) and 8MHM(4 45) 1)): ddT(())))) m(,h))))
5

---

Published as a conference paper at ICLR 2022
Non-probabilise box model (BoxE): In order to test the importance of the probabilistic semantics used to formulate the MBM, we also include as baseline, box model that does not use probabilistic semantics. For this, replace the PMBM with the won-rrobabilistic score defined in Abboud et al. (2020) and replace the log-likelihood loss (Tai) with a margin based loss marginn-
EVALUATION AND RESULTS
We evaluate the performance of MBM using real-world multi-label classification datasets (Dim- itrovski et al. 2011; Clare 2003) spanning across three domains: text (Enron), images (Imclel07a, Imclef07d, Diatoms), and functional genomics (Expr, Celleycle, Derisi, Spo). These datasets are the ideal test bed as they provide explicit label taxonomies with different connectivity: trees, forests, and more general DAGs. Moreover, all the labels of all training and test instances respect the la- bel taxonomy. The datasets include both categorical and continuous input features. We convert the categorical features into one-hot feature vectors and standardize all continuous features. The in- put encoder Fo uses common architecture for all models consisting of MLP with maximum of 3 layers. We perform a grid search over number of MLP layers, activation function, hidden di- mensions, dropout, learning rate and use the best parameters for each model The Mean Average Precision (MAP), that is the mean over the average precision values across instances in the test set, used evaluate the predictive performance of the models. Table rreports average metric values across 10 runs with different random seeds (the standard error intervals are small and are omitted for the sake of readability). As seen in Table 1 the predictive performance (MAP) of MBM is better than all other embedding based methods on at least 11 out of 12 datasets. To check the statistical significance of our results, following (emssar(0006) we first perform the Friedman test and obtain a p-value of 2 x 10 9. The critical diagram of the post-hoc Nemenyi test (Figure 2 performed after the Friedman test allows us to conclude that difference in predictive performance between MBM and all other embedding based models is statistically significant. Furthermore, from these results, one can conclude with confidence that MBM with its probabilistic formulation preforms signifi- cantly better than BoxE, which is the non-probabilistic variant of box embeddings. In the following section, we discuss another important aspect of performance, that is, the coherence of predictions.
6.1 COHERENCE ANALYSIS
MAP good metric for predictive performance, however does not take into account the in- consistencies in the predicted scores w.r.t. the label taxonomy. For instance, recalling the earlier example in Figure 1 a consistent model would always assign higher score to fiction when compared to drama, since book classified as drama should also be classified as fiction. Since MAP is inca- pable of capturing such coherence conditions, we perform further analysis to check the coherence of the predicted scores. This is done by considering two opposite perspectives towards coherence evaluation. The first one is a punitive approach, where we count the number of instances that are given inconsistent scores w.r.t the taxonomy. In the second one, more constructive approach, we try to check the amenability of the model, by applying post-hoc corrections to the predicted scores to enforce coherence. The results of these two approaches are presented by using Constraint Violation (CV) and Mean Average Precision post Coherence correction (CMAP), which are described below. Note that we do not recommend the use of either CV or CMAP as the sole metrics to evaluate the model, we present these as measures to analyse the coherence of the model predictions.
Constraint violation is a punitive measure that quantifies the extent to which the label scores gen- erated by the model violate the partial ordering of the latent label taxonomy regardless of true labels for the instances. Hence, lower value of CV implies higher taxonomic coherence in the predictions.
CV(s) = DIT Σ
Σ
1
-
(k) - 8
<0)
Mean Average Precision post Coherence correction (CMAP): Given a complete or partial tax- onomy G (C, for the labels, coherence can be imposed post-hoc by applying modification function& 8 : IR + 12_ to the label-scores produced by the model such that 8(s;) 5(s;) < 0 for
?Please see the Reproducibiliry Statement after Section for further details on the implementation of the baselines and the experiments.

---

Published as a conference paper at ICLR 2022
Table 1: Performance comparison MBM models with the baselines for the 12 multi-label classi- fication datasets. The left section compares the models that do not require explicit taxonomy, i.e., BoxE, MVM, MHM and MBM. the left section, the models with the best performance w.r.t. MAP are highlighted. The right section shows the performance when we include taxonomy information in training through LG (MVM-T, MHM-T and MBM-T), where the highlighted cells indicate an improvement performance (MAP and CV) w.r.t. the respective non-T model. All metries reported are averaged across five runs with different seeds.
Dataset
Metric
BoxE
MVM
MHM
MBM
MVM-T
MHM-T
MBM-T C-HMCNN
MAP
37.30
38.37
31.91
38.45
37.94
31.90
38.42
38.41
ExprPUN
CMAPT
37.92
37.66
32.05
38.72
37.41
32.02
38.67
38.41
CV1
4.79
1.99
1.94
2.55
1.97
1.92
1.87
0
MAP
31.96
31.68
28.76
34,20
31.61
28.74
34.61
34.35
CellcycleFUN
CMAРT
32.70
31.34
28.92
34.39
31.33
28.89
34.78
34.35
CVJ
4.02
3.42
1.78
1.77
3.45
1.78
1.35
0
MAP
26.66
23.70
26.87
28.69
24.16
24,40
28.71
28.19
DerisiFUN
CMAPt
26.96
24.13
26.98
28.86
24.35
24.52
28.88
28.19
CV1
2.27
4.79
0.95
1.67
4.01
0.85
1.43
0
MAPТ
27.97
25.18
26.58
30.10
24.21
26.57
29.62
29.18
SpoFUN
CMAPt
28.38
25,38
26.79
30.27
24,55
26.79
29.78
29.18
CVJ
2.75
4.23
1.68
1.75
4.73
1.69
1.53
0
MAPT
46.75
44.92
40.53
48.45
40.52
48.45
48.61
ExprGO
CMAPt
47.28
41.78
40.71
48.56
41.84
40.70
48.56
48.61
CV.
5.74
7.05
5.12
2.46
7.05
5.19
1.91
0
MAPТ
43.08
44.25
39.77
44.28
44.19
39.74
44.93
45.61
CellcycleGO
CMAPt
43.79
41.09
39.90
44.23
41.02
39.76
45.01
45.61
CV1
5.06
3.07
2.35
2.84
3.03
2.49
2.16
MAP
40.44
41.22
40.16
42.03
41.13
40.10
42.02
42.24
DerisiGO
CMAPt
40.73
38.21
40.28
42.14
38.21
40.20
42.12
42.24
CV1
3.16
3.43
1.98
2.37
3.46
2.02
1.13
0
MAPП
40.88
42.19
39.81
42.22
42,20
39.70
41.74
42.77
SpoGO
CMAPt
41.27
38.96
39.89
42.31
39.04
39.77
41.54
42.77
CVJ
3.89
2.81
1.93
2.68
2.77
1.90
1.80
0
MAP1
80.44
73.68
75.95
79.95
75.62
80.06
Enron
CMAPt
80.46
66.87
76
79.94
66.87
75.68
80.05
80.04
CV1
0.20
2.53
0.29
0.04
2.53
0.36
0.03
0
MAP
43.71
73.06
56.97
79.14
72.65
56.86
79.14
76.23
Diatoms
CMAРt
45.16
72.44
56.14
79.30
72.18
56.07
79.23
76.23
CV1
6.39
18.97
5.59
3.46
19.20
5.55
0.34
MAP
83.71
77.14
65.29
91.45
78.22
65.30
69.26
90.26
Imclef07a
CMAPt
84.73
76.56
66.01
91.73
77.46
66.01
69.48
90.26
CV1
12.73
23.02
4.75
5.65
22.86
4.75
2.40
0
MAPП
87.95
88.49
75.72
89.49
88.59
75.69
89.56
89.22
Imclefl77d
CMAPt
88.93
86.89
76.98
89.99
86.87
76.95
90.07
89.22
CVJ
11.93
10.72
7.52
7.16
11.02
7.56
5.66
0
MAP
4.92
5.29
6.42
2.08
5.38
7.33
2.17
Avg. Rank
CMAP
4.08
6.08
6.17
1.96
6.25 7
6.92
2.29
2.25
CV
6.67
7
3.83
4.33
4
2.17
1
all 44) E T. This can be achieved by modifying the score s; for each label I to be either the maximum (6 the scores of any of its descendants or the minimum of the scores of its ancestors (6%) in the taxonomy G. Concretely, given the scores E Rd produced by a model, for an input, the modification functions given
min (s) A
8jz
8 (s);
max (y.Dest
8j
where Alc and (1) the set of ancestors and descendants, respectively, I in the graph G. In practice, if one is given partial label taxonomy, would select whichever post-hoe modifi-

---

Published as a conference paper at ICLR 2022
cation performs better, and we refer to the maximum MAP obtained after applying one of the mod- ification functions 6G og to the scores as C Intuitively, a CMAP value close to MAP value implies that the model is more perceptive of the latent hierarchy in the label space. As seen from the critical diagram of the Nemenyi test (Figure when coherence is considered along with the predictive performance (CMAP), MBM outperforms MVM and MHM (with statistical significance) indicating that MBM is much more perceptive of the latent taxonomy compared to other embedding based models (MVM and MHM). In conclusion, while on one extreme there is MVM, which ex- hibits reasonable good predictive performance but fails to maintain coherence w.r.t the taxonomy, on the other extreme we have MHM that exhibits lowest constraint violations but gives inadeuutee pre- dictive performance. MBM, however, demonstrates good characteristics on both fronts predicvee performance as well as coherence.
Comparison with the state-of-the-art: C-HMCNN (Guunhhilia & Lukasiewicz, 2020). is the cur- rent state-of-the art method for hierarchical MLCC, and includes the modification function & (called MCM in their work) in their model directly, applying it to scores produced by multi-layer percep- tron at training as well as inference time At training time, this modification is accompanied by a clever modification of BCE loss to form novel MCLoss which works well with the modification function during training. There are two key differences between C-HMCNN and MBM that one needs to keep in mind when comparing their performance. First, C-HMCNN does not attempt to use the label representations themselves to increase coherence an idea central to MBM and this work. Second, unlike MBM, C-HMCNN requires the complete label taxonomy at training and infer- ence time to enforce the coherence. Owing to the use of complete label taxonomy at inference time, C-HMCNN always has constraint violations. However, as seen from the Table [] and the Figure
2. MBM, which does not require the label taxonomy at all, performs comparably to C-HMCNN in terms of MAP (p-value of 0.677 in two sided Wilcoxon test between MBM and C-HMCNN shows that there is no statistical difference between their performances).
Dased on MA7 ranu scraa
Ecsed CMAT resuts acrus delents
10
MMM
MMM
MMM
Figure 2: Critical diagrams of the post-hoe Nemenyi test across all 12 datasets.
ANALYSIS OF LEARNED LABEL EMBEDDINGS
In this section, we analyze the geometry of the learned label embeddings, finding that the simple geometry of box embeddings endows the MBM model with high degree of interpretability. In order to verify that label box embeddings are producing consistent scores by using inclusion in the box space, we inject into the model the taxonomy information through the additional loss term (Eq. 4). As seen from the right section in Table I injecting explicit taxonomic information into the label embeddings further reduces the constrain violation for MBM on all twelve datasets. However, the same does not aid the MVM model and even drops its predictive performance significantly. This underscores the importance of embedding geometry for inducing taxonomic coherence, and validates our intuition about the arrangement of label embeddings in MBM.
To determine the extent to which the label embeddings capture the latent label taxonomy with- out it being explicitly provided, we perform nncestor-descencee learned label embeddings. Each pair of labels (l,,4) get score (i, that is determined using their corresponding label embeddings the, viy. Since MVM and MBM have different geometri- cal interpretations, we use different scoring for each. Specifically, (NvvM( = vh /1///1 (i.)( = (4. These scores are then compared to true incestor-descende in the taxonomy to obtain respective ROC curves as shown in Figure 3 As seen in Table 3 MBM captures the true label taxonomy the best (AUC 2 0.87) for all datasets.
classification solely using the
relations
We find that o% produces lesser change the scores predicted hy all the models (MBM, MVM and MHM), and hence is used as the modification function for this analysis (see Appendix E further discussion on this) *Comparision with another competitive baseline HMCN is provided in Appendix D
8

---

Published as a conference paper at ICLR 2022
Dataset
BoxE
MVM
MHM
MBM
ExprFUN
0.59
0.61
0.40
0.89
CelleycleFUN
0.71
0.59
0..00
0.93
DerisiFUN
0.64
0.61
0.41
0.87
SpoFUN
0.65
0.61
0.39
0.87
Enron
0.66
0.49
Diatoms
0.67
0.68
0,44
0.96
MVM
Imclef07a
0.58
0.63
0.45
0.87
MHM
KBM
Figure 3: The figure the left shows the envelope of the ROC curves for the ancestor-descendant relationship classification in the label space for different embedding based models across the datasets and the table the right is the area under the ROC curves. As seen, MBM implicitly captures the label taxonomy well compared to all other models.
Furthermore, for hyperberbolic space, it is suggested that the magni- tude of embeddings relate the level of generality in taxonomy (Nickel & Kiela, 2017). We show that the same observation holds for box embeddings, with the vector embedding magnitude replaced by box embedding volume. To see this, we compute the Spearman rank correlation between the number of descendants of node in the true taxonomy and the embedding magnitude, negative embedding magnitude, and embedding volume for MVM, MHM, and MBM, respectively. The correlation values reported in Table confirm our intuition regarding box embeddings stated above It is interesting to note that as shown by the correlation between the vector magnitude and level of the label node as well as low constraint violations, nodes in the taxonomy. However, as shown by the ROC curves in Figure 3 unlike MBM, it fails to capture the exact connectivity of the nodes at different depths the taxonomy.
Table 2: Spearman rank correlation between the number of descendants in the label taxonomy with each of the follow- ing: embedding magnitude for MVM, negative embedding magnitude for MHM and box embedding volume for MBM.
Expr FUN
Celleycle FUN
Derisi
Spo
Model
FUN
FUN
Enron
Diatoms
Inclef07a
MHM MVM
-0.37
0.38
0.40
0.38
0.19
0.31
0.32
0.06
0.11
0.01
0.06
0.11
-0.02
MBM
0.47
0.49
0.50
0.48
0.47
0,23
0.43
MHM model does capture the depth of the label
8 CONCLUSION
In this work, demonstrate that box embeddings with its probabilistic formulation effectively capture taxonomic relations present between labels in multi-label classification without requiring explicit access to the taxonomy. The proposed model achieves fine balance between predictive performance and coherence. Furthermore, we find that the taxonomic relationships between labels can be easily injected via extra supervision during training, increasing the coherence of the predic- tions further. Since the model has the same computational complexity as simple neural network model, and unlike C-HMCNN it does not require the label taxonomy, the proposed model can be scaled to work with extremely large label spaces. Moreover, due to the ficxibility of using any neural network encoder this model can be easily extended different input modalities, such raw text, images, etc. We wish to pursue these two directions in our future work.
REPRODUCIBILITY STATEMENT
In this section, provide pointers to information necessary reproduce the results mentioned in this paper. The description of the datasets with various statistics, links to download them, and instructions pre-proces: them are provided in Appendix B The BoxE baseline is further described
RRfer to Appendix for scatter plots of box embedding volume vs. the number of children in the label taxonomy.

---

Published as a conference paper at ICLR 2022
with expressions for distance and the margin based loss function in Appendix A.I The hyper- parameter settings for each model-dataset combination (all 96 settings) used to produce the reported results are provided in 3 We also include discussion computational complexity of the models in Appendix A.2 Finally, the code for all the models (including the baselines) used in this paper is available at https:::///....... / h/tg://iitttbtt com eesl //wx-mxmmmmm c- --lr-102222
ACKNOWLEDGEMENTS
The authors would like thank the members of the Information and Extraction Synthesis Labora- tory (IESL) at UMass Amherst for helpful discussions. This work was partially supported by IBM Research AT through the AI Horizons Network and the Chan Zuckerberg Initiative under the project Scientific Knowledge Base Construction. Additional support was provided by the National Science Foundation (NSF) under Grant Numbers IIS- 17661888., I1S-1922090, and IIS-1955567 the Defense Advanced Research Projects Agency (DARPA) via Contract No. FA8750-17-C-0HF award No. 89341790 from the University of Southern California, and the Office of Naval Research (ONR) via Contract No. N66001 1924032 under Subaward No. 123875727 from the University of Southern California. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IBM, CZI, NSF, DARPA, ONR, or the U.S. Government.
under Sub-
REFERENCES
Ralph Abboud, Ismail Ceylan, Thomas Lukasiewicz, and Tommaso Salvatori. BoxE: A box em- bedding model for knowledge base completion. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in neural information processing systems, volume 33, pp. 9649-9661. Curran Associates, Inc., 2020. URL paper/2020/
ngs www.rnurpps.... cc/ .ddf
6dbbebabe 4a 3
Zafer Barutcuoglu, Robert E. Schapire, and Olga G. Troyanskaya. Hierarchical multi-label pre- diction of gene function. Bioinformatics. 22(7): 836, 01 2006. ISSN 1367-4803, doi: 10. 10.1093/boonfommaiccsooooo URL https:: oropi.oorg/000 1093/bioinformaticooo
bt.k 048
David Belanger and Andrew McCallum. Structured prediction energy networks. Maria Flo- rina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Leurning Research, pp. 983-992, New York, New York, USA, 20-22 Jun 2016. PMLR. URL http://proceedings
pzess/v48/belange:111 html
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https: //www. wandb. com/ Software available from wandb.com.
Michael Boratko, Dongxu Zhang, Nicholas Monath, Luke Vilnis, Kenneth L. Clarkson, and An- drew McCallum. Capacity and bias of learned geometric embeddings for directed graphs. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openzevieww wwwprrevieww.neteeeeee www.:neeee.onnieee FIqTX6Fe2Wv
Ricardo Cerri, Rodrigo C. Barros, and Andrão C.P.L.F. de Carvalho. Hierarchical multi-label classification using local neural networks. Journal of Computer and System Sciences, 80(1):39 56, 2014. ISSN 0022-0000.
doi: htt.://ceee..ttto.ooooooo.ooaaaa
10160.11161.... fcs. 2013.03..0000000 URL http:
77 www.. www.scceiiiccicccooooocccceeccee
Soumya Chatterjee, Ayush Maheshwari, Ganesh Ramakrishnan, and Saketha Nath Jagaralpudi. Joint learning of hyperbolic label embeddings for hierarchical multi-label classification. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2829 2841, Online, April 2021. Association for Computational Linguistics.
URL https: www. a cl www.w..o.......ee. wow.oooooyyyyccoo /2021 eac -ma inaain.. 247
10

---

Published as a conference paper at ICLR 2022
Tejas Chheda, Purujit Goyal, Trang Tran, Dhruvesh Patel, Michael Boratko, Shib Sankar Dasgupta, and Andrew McCallum. Box embeddings: An open- library for representation learning using geometric structures. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 203-211, Online and Punta Cana, Dominican Republie, November 2021, Association for Computational Linguistics. doi: 10.18653/ org. 2021 emnlp demo 24
v1/2021. .ew.022.mmmpddmmoo.. URL.tttss://aclattboo
Amanda Clare. Machine learning and data mining for yeast functional genomics. PhD thesis, Citeseer, 2003.
Shib Sankar Dasgupta, Michael Boratko, Dongxu Zhang, Luke Vilnis, Xiang Lorraine Li, and An- drew McCallum. Improving local identifiability in probabilistic box embeddings. In Advances in Neural Information Processing Systems, 2020. URL http:://pprooceeiigggg neurips. Paper. pdf
Shib Sankar Dasgupta, Michael Boratko, Shriya Atmakuri, Xiang Lorraine Li, Dhruvesh Patel, and Andrew McCallum. Word2box: Learning word representation using box embeddings, 2021a.
Shib Sankar Dasgupta, Xiang Lorraine Li, Michael Boratko, Dongxu Zhang, and Andrew Mc-
Callum. Box-to-box transformations for modeling joint hierarchies. In Pruceedings of the 6th Workshop on Representation Learning for NLP (RepLANLP-2022) pp. 277-288, Online, August 2021b. Association for Computational Linguistics. doi: 10.18653/v 1/2021 www.://lrrpllllll11. 1.2... URL
/ttpps//acllanthooooo..oogg
repl 4n p-
Janez Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 7:1-30, 2006.
Ivica Dimitrovski, Dragi Kocev, Suzana Loskovska, and Sašo Džeroski. Hierarchical annotation of medical images. Pattern Recognition, (4110-11) 2011.
Oetavian-Eugen Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic Entailment Cones for Learning Hierarchical Embeddings. Technical report, 2018a. URL hhttps://en.
Octavian-Eugen Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic Neural Networks. 2018b. doi: arXiv:1805.091 12v2. htLp: //axviv..riivvv/aa wwtp://////aaa.oo.ooooo 09112
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. Allennlp: deep semantic natural language processing platform. 2017.
Elconora Giunchiglia and Thomas Lukasiewicz. Coherent Hicrarchical Multi-Label Classification Networks. 2020. URL
10151
Wei Huang, Enhong Chen, Qi Liu, Yuying Chen, Zai Huang, Yang Liu, Zhou Zhao, Dan Zhang, and Shijin Wang. Hicrarchical multi-label text classification: An attention based recurrent network approach. In International Conference Information and Knowledge Management, Proceed- ings, pp. 1051-1060. ACM, 2019. ISBN 9781450369763. doi: 10.1145//353388.3337888 URL 3357888
sttps://doo. org. 10.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome: Connecting language and vision using crowdsourced dense image annotations, 2016.
David D Lewis, Yiming Yang, Tony Russell-Rose, and Fan Li. Rev1: new benchmark collection for text categorization research. Journal of machine learning research, 5(Apr):361-397 2004.
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
11

---

Published as a conference paper at ICLR 2022
Shikhar Murty, Patrick Verga, Luke Vilnis, Irena Radovanovic, and Andrew McCallum. Hierarchical losses and new resources for fine- grained entity typing and linking. In ACL 2018 56th Anmual Meeting the Association for Computational Linguistics. Proceedings of the Conference (Long Papers), volume 1, pp. 97-109, 2018. ISBN 9781948087322. doi: 10.18653/1/p118-1010 URL https: www. c web.
Maximilian Nickel and Douwe Kiela. Poincar 'e Embeddings for Learning Hierarchical Rep-
resentations. 2017. ISSN 19457871. doi: 10.1109/ICME.2203.6675555 URL http:
//arxiv.org larrrg/ab//170000 08039
Yasumasa Onoe, Michael Boratko, and Greg Durrett. Modeling fine-grained entity types with box embeddings, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vitito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. H. Wallach, H. Larochelle, Beygelzimer, F. d'Alché-Buu E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran 9015
neuriss..eppppppp... Associates, Inc., 2019. URL http: http::ppppppppps...rrppppp.aaacp
an- wttppp/aaaaaaaaaaaaaaaeeaaoooora
Dhruvesh Patel, Shib Sankar Dasgupta, Michael Boratko, Xiang Li, Luke Vilnis, and Andrew Mc- Callum. Representing joint hierarchies with box embeddings. In Automated Knowledge Base Construction, 2020. httthhttt////peeeeeeee
net. www.mmirumiioooo 4
C. N. Silla and A. Freitas. A survey of hierarchical classification across different application do- mains. Data Mining and Knowledge Discovery, 22:31-72, 2010.
Luke Vilnis, Xiang Li, Shikhar Murty, and Andrew McCallum. Probabilistic Embedding of Knowl- edge Graphs with Box Lattice Measures. 2018. URL https://ppppp cs umass. .edaasseeuu///
"luke/box-ltte ces http:ep.....ttt////a/ooorrraaaaa
06627
Jonatas Wehrmann, Ricardo Cerri, and Rodrigo Barros. Hierarchical multi-label classification net-
works. In International Conference on Machine Learning, pp. 5075-5084. PMLR, 2018a.
Jónatas Wehrmann, Ricardo Cerri, and Rodrigo C Barros. Hicrarchical Multi-Label Classification Networks. Technical report, 2018b.
12

---

Published as a conference paper at ICLR 2022
A IMPLEMENTATION DETAILS
In this section, we describe the implementation details: training procedure and hyper-paramettr search.
Frameworks used: We implement all the models described in this work using PyTorch (Paszke al. 2019). We also make use of NLP specific abstractions over PyTorch provided by AllenNLP Gaardner et al. (2017). We also make use of the abstractions provided in the Box Embedding library Chheda et al. (2021) in our implementation of MBM. For the extensive hypee-parameter search, we use the academic version of Weights & Biases library (Biewald 2020).
Data pre-processin: The datasets were pre-processed in the exact same manner as in Giunchiglia & Lukasiewiez (2020), i.e., the categorical features were converted to one-hot feature vectors and the continuous input features were standardized. Also, following the Giunchiglia & Lukasiewicz (2020); Wehrmann et al. (2018a), the labels corresponding to the root nodes were removed from training and evaluation. Further details of the original datasets provided B
Training: We used ADAM (Kingma & Ba 2017) with batch size of 4 to learn the model param- eters for all the models. Since the naive implementation of the label interaction loss described in Eq. is too expensive to compute at each mini-batch step, approximate it by randomly sampling (without replacement), at each mini-batch step, subset of edges T T. The size of the sampled (last column in Table
set is
Hyper-pun ppprmpprr search: The following hyper-parameee were searched using the performance on the validation set: optimizer learning rate, hidden dimensions of the MLP, number of hidden layers (maximum of 3), activation functions, dropout probabilities, weight decay and sampling per- centage for the labels for computing the label interaction loss(L)) for the MHM-T, MBM-T and MVM-T models. After an initial investigation, we identified that the softplus activation performed the best for all models and hence, the activation was fixed to softplus for the final grid search. Also note that in order to be comparable the previous work that uses these datasets, we had divide our experiments into two kinds of runs tuns-the search runs and the final runs. First, in the search rims, we use validation set obtain the best hyper-pane srper-pprameters for each model-dataset combination. After identifying the best set of typer-parameters the training and validation sets were combined. We call the models trained on this combined dataset as the final runs. Since one cannot use early stopping without validation set, the number of epochs train the models in the final runs was also obtained as a hyper-parameter and was set to the best epoch for the corresponding search run. All the results reported using the metrics obtained in the final runs (10 runs with different random seeds for each cataaet-model pair with best syper-parametes obtained from the corresponding search runs). Table summarizes the search ranges used and Table presents the final syper-parameters obtained.
Table 3: Summary of the hyper-paramett search ranges for each dataset and model. The best hyper- parameters for each model and dataset combination were picked using grid search using MAP on the validation set. Note that and label sample percent are only applicable the MBM-T, MVM-T and MHM-T models.
Dansets
banch size
hidlen dim
layers
activation
dcopout
App weight
label sumple percent
1o-4, Se-4 16-3, 1622 250, 500, 1000, 1750
1,2.3 sigmoid, rolu, solples
I.O, U.I. I.3. U.
1o-3, Te& 16-7, 1e-9
10,20,30
Table 4: Best super-parameter, obtained through grid search for each dataset-model combination. It should be noted that the activation function chosen for all model configurations softplus.
Dataset
Model
learning rate
hidden dim
layers
dropout
% label sample
CellcycleFUN
BoxE
0.001
500
0.3
CellcycleFUN
C-HMCNN
0.001
500
3
0.3
100
CellcycleFUN
MBM
0.0001
1750
0
CellcycleFUN
MBM-T
0.0001
500
30
1e-07
CellcycleFUN
MHM
0.0001
1000
3
CelleycleFUN
MHM-T
0.0001
1000
3
0
30
le-07
CellcyclcFUN
MVM
0.001
1750
0
CellcyclcFUN
MVM-T
0.001
1750
30
1c-07
13

---

Published as a conference paper at ICLR 2022
Table 4 continued from previous page
Dataset
Model
learning rate
hidden dim
layers
dropout
% label sample
CellcycleGO
BoxE
0.001
500
0.3
CelleycleGO
C-HMCNN
0.001
500
0.5
CelleycleGO
MBM
0.0001
1750
0
CelleycleGO
MBM-T
0.0001
1750
3
0
10
1e-07
CellcycleGO
MHM
0.001
1750
0.3
CellcycleGO
MHM-T
0.001
1750
0.3
30
1e-07
CellcycleGO
MVM
0.001
500
0.5
CelleycleGO
MVM-T
0.001
500
2
0.5
30
1c-07
DerisiFUN
BoxE
0.001
500
0.3
DerisiFUN
C-HMONN
0.0001
1000
0.3
100
DerisiFUN
MBM
0.0001
500
DerisiFUN
MBM-T
0.0001
500
30
1c-07
DerisiFUN
MHM
0.001
500
0.1
DerisiFUN
MHM-T
0.001
500
0.1
30
1c-07
DerisiFUN
MVM
0.001
1750
0
DerisiFUN
MVM-T
0.001
1750
0
30
1e-07
DerisiGO
BoxE
0.001
500
0.3
DerisiGO
C-HMCNN
0.001
1000
0.3
100
DerisiGO
MBM
0.0001
500
DerisiGO
MBM-T
0.0001
500
20
1e-07
DerisiGO
MHM
0,001
500
0
DerisiGO
MHM-T
0.001
500
0
30
1c-07
DerisiGO
MVM
0.001
500
0.3
DerisiGO
MVM-T
0.001
500
0.3
30
1c-07
Diatoms
BoxE
0.001
1750
0.3
Diatoms
C-HMCNN
0.0001
500
0.1
100
Diatoms
MBM
0.0001
1750
Diatoms
MBM-T
0.0001
1750
0
30
1c-07
Diatoms
MHM
0.0001
1750
Diatoms
MHM-T
0.0001
1750
30
1e-07
Diatoms
MVM
0.0001
1000
Diatoms
MVM-T
0.0001
1000
0
30
1e-07
Enron
BoxE
0.001
500
0.3
Enron
C-HMCNN
0.0001
500
2
0.3
100
Enron
MBM
0.0001
1750
Enron
MBM-T
0.0001
1750
30
1c-07
Enron
MHM
0.0001
1750
0.1
Enron
MHM-T T
0.0001
1750
2
0.1
30
1e-07
Enron
MVM
0.0005
500
3
0
Enron
MVM-T
0.0005
500
30
1c-07
ExprFUN
BoxE
0.001
500
0.3
ExprFUN
C-HMCNN
0.0001
1000
0.3
100
ExprFUN
MBM
0.0005
1000
ExprFUN
MBM-T
0.0005
1000
0
30
1c-07
ExprFUN
MHM
0.0001
1750
3
0
ExprFUN
MHM-T
0.0001
1750
0
30
1e-07
ExprFUN
MVM
0.0005
500
0.5
ExprFUN
MVM-T
0.0005
500
0.5
30
1c-07
ExprGO
BoxE
0.0001
1750
0.3
ExprGO
C-HMCNN
0.001
500
0.5
100
ExprGO
MBM
0.0001
1000
2
0
ExprGO
MBM-T
0.0001
1000
0
5.
le-07
ExprGO
MHM
0.0001
1750
ExprGO
MHM-T
0.0001
1750
30
1c-07
ExprGO
MVM
0.0001
1750
0.1
ExprGO
MVM-T
0.0001
1750
3
0.1
30
1c-07
Imclef07a
BoxE
0.001
1000
0.1
imclef07a
C-HMCNN
0.001
1000
0
100
14

---

Published as a conference paper at ICLR 2022
Table 4 continued from previous page
Dataset
Model
learning rate
hidden dim
layers
dropout
% label sample
V
Imelef07a
MBM
0.0001
1750
Imclef07a
MBM-T
0.0005
1750
0.25
30
1c-07
Imclef07a
MHM
0,001
1000
2
0
Imclef07a
MHM-T
0.001
1000
2
0
30
1e-07
Imclef07a
MVM
0.001
1000
Imclef07a
MVM-T
0.001
1000
30
1e-07
Imclef07d
BoxE
0.001
1750
0.3
Icclef07d
C-HMCNN
0.001
500
0.1
100
Imclef07d
MBM
0.0005
1750
Imclef07d
MBM-T
0.0005
1750
3
0
30
1e-09
Imclef07d
MHM
0.001
1750
0
Imclef07d
MHM-T
0.001
1750
30
1e-07
Imclef07d
MVM
0.001
1000
Imclef07d
MVM-T
0.001
1000
2
0
30
1c-07
SPoFUN
BoxE
0.001
500
0.3
SpoFUN
C-HMONN
0,001
500
0.5
100
SpoFUN
MBM
0.0001
1000
0
SpoFUN
MBM-T
0,0005
500
30
1c-09
SpoFUN
MHM
0.0001
1000
SpoFUN
MHM-T
0.0001
1000
0
30
1c-07
SpoFUN
MVM
0.001
1750
0
SpoFUN
MVM-T
0.001
1750
0
30
1e-07
SpoGO
BoxE
0.001
500
0.3
SpoGO
C-HMCNN
0.001
1000
0.3
100
SpoGO
MBM
0.0001
1000
SpoGO
MBM-T
0.0001
1000
10
1e-07
SpoGO
MHM
0,001
500
3
0
SpoGO
MHM-T
0.001
500
0
30
1c-07
SpoGO
MVM
0.001
500
0.5
SpoGO
MVM-T
0.001
500
0.5
30
1c-07
A.1
FURTHER DETAILS FOR THE BASELINES
BoxE: Let p E Rdd be a point and z, Z E Rdd with 2 < 7, be the lower left and upper right coordinates of non-probabilistic label box. Then non probbabbbiiiitcc compatibility score between the point and the box can be defined as shown in Abboud et al. (2020). Concretely, Abboud et al. (220)) define per-dimension distance
[ps ppl #
if S4 7
dist(p:, 24, Z;)
P: Pi-oel
K, otherwise,
where G: = (2, - /2. wi Z; are the center and width of the projection of the box (i.e., an interval) in dimension i, and k. 0.5 w; (w; factor. The score function that measures the compatibility of the point p and box (z, Z) is deñned as SpexE (p, 7) =-[lu[la- where = dist(p,, 2;, Z.). Given the trainable parameters w . E R for representing the labels, the lower left and the upper right coordinates a non-probabillistcc box for label are taken to be 2; VTT, and Z, v._, + log(1 exp vi respectively. The input x is encoded as a point p E TR. using multi-layer perceptron Fo. just like MVM. However, since the score is negative distance, the parameters are learnt using margin based loss given as:
wiFT) is the width dependent
max (0, 1 (SBoxE (Folx), Zp: Zp) SBoXE BBoxE(Fo(x)) (Fo(z), on Zn))
Lmarrin (4,,, 0; D) = Σ Σ
Pi + [W
PEP,REN
where P, and are the positive and negative labels for i-th data instance, respectively.
15

---

Published as a conference paper at ICLR 2022
A.2 COMPUTATIONAL COMPLEXITY
The computational complexity of the MBM can be divided into two parts the computation of the instance box Boxe and the computation of probability score P(Y X) given the encoding of the instance and label box parameters. Computing Box, amounts to splitting the output of beed-forward network (xx:) into min and parameters <(x)) - .(xx), and thus has the same com- plexity as that the MVM. Assuming that the operations > II. log and exp have unit complexity, computing P(Y X) also has equivalent complexity. The computation of conditional probability for a single label using the MVM model involves computing ((((z)- -yk)). an O(dL) operation. Now, with same assumptions for the MBM, computing A(Boxo n Box involves d invo- cations of log(exp(-) + exp(-)) followed by subtractions and invocations of log(1 + exp(.)), resulting (d) operations. The computation of (Boxx(ar) omits the intersection calculation, but otherwise is the same, and thus also (dd). Hence, calculating (( is (d), resulting
X(Bo(zz)
in overall complexity of (dT.), which is equal to that of MVM. We note that the complexity of our model is not dependent on the depth of the hierarchy, as in C-HMCNN, and thus can reasonably scale to arbitrarily deep hierarchies as might be present in extreme multi-label classification. These theoretical statements are supported by Table [3] which provides average epoch duration in seconds for MVM, MBM, MBM-T, and C-HMCNN with the same hidden size. Hence, we can conclude that the computational complexity of MVM = MBM < MBM-T.
Table 5: Average epoch duration in seconds for MVM, MBM, MBM-T and CHMCNN with the same hidden size
Model
ExprFUN
CelleycleFUN
SpoFUN
DerisiFUN
MVM
17.2
15.4
15.8
15.5
MBM
18.4
16.9
16.3
15.2
MBM-T
20.1
26.7
24.1
27.1
C-HMCNN
17.8
17.0
15.9
15.4
BoxE
18.9
16.6
15.9
16.1
Computational resources used: For datasets with number of labels less than 500, i.c., the 4 FUN- CAT datasets, Imclef07a, Imclef07d, Diatoms and Enron, all the models were trained TitanX GPU (memory=12GB). For the GO datasets that have number labels greater than 4000, all the models trained M40 GPU (memory=24GB).
A.3 CODE
Executable python code with detailed instructions to reproduce the results reported in i provided using at obtaining the pre-processed datasets, training new model from scratch (MBM or any baseline), evaluating pre-trained model on test set, directly downloading the pre- me-trrined models for datasets.
com/ hes///esl/// ttss//box-nt wecrrriii 2022 We also include instructions
B DATASETS
The 12 datasets used in this work wary greatly terms of domain, number of labels, number of instances, and the connectivity of label taxonomy. The characteristics of each dataset w.r.t these properties is summarized in 6 These datasets do not require a licence and are available for public usage. The links the sources for all the datasets are provided in Table 7 below. The continuous features are standardized by removing mean and scaling to unit variance, and the categorical features encoded as one-hot vectors.
16

---

Published as a conference paper at ICLR 2022
Table 6: Summary of the datasets used in experiments. The feature based multi-label datasets span
across 3 domains: functional genomics, image and text.
Input/Feature
Label
#Instances Val
Dataset
Domain
#Labels
Type
Taxonomy
Train
Test
Expr FUN
Genomics
Continuous
Forest
500
1636
849
1288
Cellcycle FUN
Genomics
Continuous
Forest
500
1628
848
1281
Derisi FUN
Genomics
Continuous
Forest
500
1608
842
1275
Spo FUN
Genomics
Continuous
Forest
500
1600
837
1266
Expr GO
Genomics
Continuous
DAG
4132
1636
849
1288
Cellcycle GO
Genomics
Continuous
DAG
4126
1625
848
1278
Derisi GO
Genomics
Continuous
DAG
4120
1605
842
1272
Spo GO
Genomics
Continuous
DAG
4120
1597
837
1263
Diatoms
Image
Continuous
Tree
399
1500
565
1054
Imclef07a
Image
Continuous
Tree
97
7000
3000
1006
Imclef07d
Image
Continuous
Tree
47
7000
3000
1006
Enron
Text
Binary
Tree
57
650
338
600
Table 7: The table provides the links to download the data from original source.
Downkod
Imclel17a, Imclefl7dd Enron, Diatoms Expr, Spo, Derisi, Calleycle (FUNGO))
betes /7kt. 33e, Zora Kwwtw//hh/////cco. http://ht//eeeeeeeeeeeeee /ocky
Fe PEtpas7 /tta1 .89. 401 euven, Be7 19/ Hn www.ooattttttt....
C SIGNIFICANCE TESTING
In section G we provide the results for Friedman test followed by post-hoc Nemenyi test, and also the result of pairwise Wilcoxon test where the former are not sufficient, i.e. for the case of MBM vs C-HMCNN w.r.t MAP. For the sake of completeness, table S provides results of pairwise Wilcoxon test comparing all models with MBM and MBM-T w.r.t all three metrics.
The table presents the results of the Wilcoxon signed-rank test. Each cell shows the p- value for the null hypothesis that two related paired samples, here MBM/MBM-T and the other model (column), come from the same distribution, with the alternative hypothesis that MBM/MBM- models are better performance compared to the other model.
Table 8:
Metric
BoxE
MVM
MHM
MBM
MVM-T
MHM-T
MBM-T
C-HMCNN
MBM
0.0005
0.0002
0.0002
0.0002
0.0002
0.4392
0.3386
MAP
MBM-T
0.0171
0.0261
0.0002
0.5608
0.0212
0.0002
0.4849
MBM
0.0007
0.0002
0.0002
0.0002
0.0002
0.361
0.1506
CMAP
MBM-T
0.0212
0.0134
0.0002
0.6389
0.0134
0.0002
0.3667
MBM MBM-T
0.0002
0.0012
0.7407
0.0012
0.6890
1.0
1.0
CV
0.0002
0.0002
0.0046
0.0002
0.0002
0.0046
1.0
D COMPARISON TO C-HMCNN AND HMCN IN TERMS OF AU(PRC)
Due to our focus on coherence and predictive performance at the same time. we use mean average precision as the metric to evaluate predictive performance. One could, however, also use AU(PRC) as the metric for predictive performance. For the sake of completeness, in Table g we present the AU(RCC) values for the final models obtained our experiments, and compare it with those provided in Giunchiglia & Lukasiewicz (2020) and Wehrmann al. (2018a). We believe that the difference between the AU(PRC) reported for C-HMCNN in Giunchighaa & Lukasiewicz (2020) and work is due to the difference in the sypee-parammttr search strategy. Specifically, the reference metric to identify the best papper-prraeterss while Giunchiglia & Lukasiewicz (2020) uses AU(PRC) itself.
MAP as
17

---

Published as a conference paper at ICLR 2022
Table 9: The table presents the area under the precision-recall curve (AU(PRC) for the models presented in this paper and HMCN-F (Wehrmann et al. 2018a). Here, the columns with are taken from their respective papers, and C-HMCNN our implementation of the corresponding model for which the best hyper-parameters also obtained, like other models in our implementation, using MAP on validation set.
Dataset
C-HMCNN
HCMN-F*
C-HMCNN
BoxE
MVM
MHM
MBM
ExprFUN
30.2
30.1
28.17
16.23
28.5
19.6
25.62
CellcycleFUN
25.5
25.2
24.46
12.25
21.19
18.19
21.57
DerisiFUN
19.5
19.3
17.61
10.19
14.45
17.42
18.74
SpoFUN
21.5
21.1
20.72
13.69
16.19
17.16
19.57
ExprGO
44.7
45.2
41.92
21.74
40.5
28.91
40.89
CellcycleGO
41.3
40
40.25
25.86
39.39
33.77
38.19
DerisiGO
37
36.9
36,35
16.22
34.63
34.59
36.42
SpoGO
38.2
37.6
37.09
23.4
36.54
34.37
36.38
Enron
75.6
72.4
75.18
72.67
69
24.76
74.53
Diatoms
75.8
53
70.39
1.86
68.3
38.29
78.21
Imclef07a
95.6
95
92.81
5.43
80.57
58.06
94.61
Imclef07d
92.7
92
89.29
7.96
89.73
64.33
90.48
E MAP AND CMAP
In MCL. problems, where predictions consistent with a taxonomy are necessary, the model designer might be ready to sacrifice some absolute predictive performance for increased coherence, where the predictive performance can be measure using Mean Average Precision (MAP). First recall that Average Precision (AP) for instance is the weighted mean of precisions achieved at each thresh- old, with the increase recall from the previous threshold used as the weight, and Mean Average
Precision (MAP) is the mean of AP across instances. Given data D, let 8; (k) P(y: 1 (x)(
denote the score generated by the model for label given the input (k), Then MAP is computed as
D AP ((())) y())
MAP(s)
D
given the scores E R produced by a model, for an input, the two post-hoc modification functions that can be applied make these score coherent are given as
As described in Section 6.1
(8):
min
8jz
8 (8):
max 2y (L,
8jz
(8)
Ly LyEAnc (1.)U
where Anco(l) and Desa(() the set of ancestors and descendants, respectively, in the graph G.
One can compute MAP after applying either one of the two modification functions. If human were to make consistent score of a label to be the minimum of its ancestors or maximum of its descendants. Once picked, the same approach has to be applied to all the labels. We argue that in terms of correlation with human judgment, both these methods are equivalent. Hence, one can pick the modification function based on empirical performance of the post-hoc correction algorithm. We observe that 82 ie.,, "min of ancestors" approach, produced higher MAP with both MVM and MBM models compared to 6M as can be seen in [O] We believe that "min of ancestors" works better because it promotes sparsity in predictions, which essential for the task of MI.C
corrections based the complete hierarchy, they would either pick the
18

---

Published as a conference paper at ICLR 2022
Table 10: CMAP represents post-hoc correction of scores by taking the "minimum of ancestors". CMAP" represents post-hoc correction of scores by taking "maximum of descendants". As seen, CMAP produces better results for both MBM and MVM compared to CMAP'.
MVM CMAP
MBM
Dataset
CMAP
MAP
CMAP
CMAP
MAP
ExprFUN
37.66
14.83
38.37
38.72
1.78
38.45
CellcycleFUN
31.34
10.62
31.68
34.39
1.79
34.2
DerisiFUN
24.13
6.28
23.7
28.86
1.79
28.69
SpoFUN
25.38
7.4
25.18
30.27
1.79
30.1
ExprGO
41.78
26.15
44.92
48.56
0.8
48.45
CellcycleGO
41.09
21.64
44.25
44.23
0.8
44.28
DerisiGO
38.21
19.68
41.22
42.14
0.8
42.03
SpoGO
38.96
19.79
42.19
42.31
0.8
42.22
Enron
66.87
55.48
73.68
79.94
9.96
79.95
Diatoms
72.44
58.36
73.06
79.3
0.49
79.14
Imclef07a
76.56
69.66
77.14
91.73
3.12
91.45
Imclef07d
86.89
81.32
88.49
89.99
6.52
89.49
F PERFORMANCE ON LONG TAIL OF LABELS
Since MVM MBM both are embedding based models, we expect the performance of the MBM model to wary with the label frequency in the same manner as MVM. This hypothesis is corroborated by the following table which shows the spearman rank correlation between the label frequency and the mean average precision (MAP) for that label. The high positive correlation for all the models suggests that the performance both MVM and MBM often degrades as the frequency of the labels drop. However, shown by the table IT both the models perform reasonably well for extremely low frequency labels. Here, tail. MAP the mean average precision for labels with frequency 0.001 % lower.
Table 11: Comparing the performance of long tails of labels for MBM, MBM-T and MVM. Here, correlation the spearman rank correlation between label frequency and MAP of the label and tail MAP the MAP for labels with frequency 0.001% or lower.
MBM
MBM-T
MVM
Dataset
Correlation
MAP
Correlation
tail MAPP
Correlation
tail_MAP
ExprFUN
0.72
48.91
0.73
48.68
0.75
49.45
CellcycleFUN
0.74
46.11
0.75
46.17
0.82
44.02
DerisiFUN
0.79
41.86
0.82
41.62
0.79
41.78
SpoFUN
0.8
42.67
0.77
42.75
0.79
42.15
Enron
0.84
79.19
0.83
74.97
0.91
67.63
Diatoms
0.29
86.3
0.29
85.46
0.28
82.2
Imclef07a
0.73
59.59
0.74
60.2
0.82
58.78
G EFFECT OF EMBEDDING DIMENSION
To study the effect of embedding dimensions the performance, plot the MAP and CV vs the hidden dimension size for a subset of the datasets. As shown figure 4 at extremely low dimen- sions, the MBM model consistently outperforms the the most competitive embedding based model MVM. demonstrating the usefulness of the favorable inductive bias of the box space in inducing parsimonious representations.
19

---

Published as a conference paper at ICLR 2022
Pe formancee var adinn Wii wer.tedbbdinng dimeniiion
Pufforancs var varation wil onocoddd dinoosion
2.5
8
203
400
031
300
9353
Entescring dimeraton
(a) MAP
(b) CV
Figure 4: Above figure shows the variation of performance w.r. the embedding dimensions for model MBM and MVM on FUNCAT datasets. Note that the training and evaluation performed for this analysis includes the root node.
20

---

Published as a conference paper at ICLR 2022
H
EFFECT OF HIERARCHY ON THE MARGINAL SCORE
402
200
150
125
(a) Expr FUN
(b) Cellcycle FUN
2675
109204 9000
2425
122
2975
110
2025
Chidien
numberr crardre
(c) Derisi FUN
(d) Spo FUN
440
366.0
405
362.5
300.0
426
667.6
120
365.0
410
560.0
347.5
405
2 TUPDRR 2himm
2
2
2'
2
(e) Enron
(f) Diatoms
Figure 5: As discussed in Section 7 the scatter plots further show that the magnitude of box em- beddings(box volume) relate the level of generality taxonomy.Jn In the figure, the marginal score refers to box volume of a label and the number of children in the label space taxonomy refers to the level of generality.Non Note Note that the training and evaluation performed for this analysis includes the root node.
21


Reviews:
[{"review_id": "j6mpJdsVn3t", "reviewer": "Reviewer_X5cP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I really like the idea at the base of this paper, however I think some revisions are needed. \n\n1. The authors should change the term “consistent” with “coherent” throughout the paper. Indeed, the authors use the word “consistent” to express the fact that the model is coherent with the taxonomy. This however, has nothing to do with the Fisher consistency and can mislead the readers. \n\n2. At page 4 the authors first write $Y \\in S$ and then $Y \\in \\mathcal{S}$, did you mean in both cases $Y \\in \\mathcal{P}(S)$?\n\n3. At page 4, why do the authors use the function $f$? The equation can easily rewritten by substituting $f$ with its value. Also, why was $\\delta$ chosen to be equal to $10^{-5}$?  \n\n4. The functional genomics datasets come in two versions, the Funcat version with a tree taxonomy, and the GO version with a forest taxonomy. I see that the authors have used the GO version for expr, cellcycle, derisi and spo, but they have used the Funcat version for diatoms. It would be interesting to have both the GO and the Funcat version for all the datasets. Further, the authors have just used Imclef07a, while Imclef07d is also available. I would recommend to test the model also on Imclef07d. \n\n5. In order to check the statistical significance of the obtained results the authors should perform the pairwise Wilcoxon test (at least to compare the performance of MVM vs MBM and MHM vs MBM).\n\n6. C-HMCNN does not apply a post-hoc modification, as the max layer is embedded in the network itself. Notice that post-hoc means something that is applied just at inference time (and not at training time).\n\n7. Would it be possible to also do MVM-T? \n\n8.  In the analysis of the learned embeddings, why weren’t the ROC curves for MHM shown? \n\n9. In the data pre-processing, what do the authors mean by “The datasets were pre-processed to remove noisy characters, fix encoding issues”?\n\n10. Can you also report in the appendix the final hyperparameters for each model?\n\n11. Why is there such a big difference between the performances of C-HMCNN (as implemented by the authors) and your implementation?\n \n12. Why aren’t the results in terms of AU(PRC) of MHM reported? \n\n\nMinor comments: \n- It should be C-HMCNN and not CHMCNN\n- Add upward arrows in Table 2 next to MAP and CMAP", "summary_of_the_paper": "In this work the authors propose a new model for multi-label classification problems. In their model, the authors represent the labels as boxes and they use neural networks to embed the input datapoints in the same space. Thanks to the box representation, the model achieves higher interpretability, and the authors can inject background knowledge (in the form of taxonomies) into the model to improve the coherence of the learnt labels representation with the taxonomy. \nThe model has been tested on 7 publicly available datasets, and evaluated using three different metrics, namely, CMAP, CV and and MAP. \n", "summary_of_the_review": "The paper shows an interesting idea, however to have full acceptance some revisions are needed (see full review). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SQ4qv9azxJk", "reviewer": "Reviewer_HdXC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe idea to use box embeddings for hierarchical multi-label classification seems good and promising.\n\nThe paper has mostly been written clearly.\n\nWeaknesses:\n\nThe proposed method is a relatively straight-forward application of box embeddings as developed in earlier literature (including e.g. Dasgupta et al., 2020a) into hierarchical multi-label classification.\n\nI am not sure I understand why one would need 'a model that uses box embedding to capture general label-label relationships without the explicit use of label taxonomy'. Would it not be beneficial to explicitly use the label taxonomy? The paper writes that requiring a 'complete label taxonomy either at inference time or both at training as well as inference time, making these models hard to scale to large label spaces'. I would like to see concrete examples of when this really happens. To me it seems that it is relatively easy to learn a mostly complete label taxonomy as the first task and then use e.g. CHMCNN.\n\nNo clear enough definition has been provided for the MAP measure. Is the mean taken over the labels and for each label there is an area under the precision-recall curve calculated?\n\nIt has not been explained sufficiently why MAP is a good measure to look at. As the paper correctly states, 'it does not take into account inconsistencies in the predicted scores w.r.t. the label taxonomy'. Instead of MAP, one could consider different score thresholds and for each threshold calculate how many labels would get correct predictions. However, indeed, Giunchiglia & Lukasiewicz, 2020 also seems to use the same MAP measure.\n\nThe baselines in the experiments could have included HMCN-R and HMCN-F from Giunchiglia & Lukasiewicz, 2020. While it is true that 'HMCN does not try to enforce consistency strongly and focuses solely on predictive performance', it does not mean that HMCN could not be a strong competitor in the experiments.\n\nThe origin of the MVM (Multi-label Vector Model) as a baseline has not been explained or referenced. What is the motivation behind this method?\n\nStatistical significance of the differences of MBM over MVM has not been demonstrated through a statistical test. Currently the text about Table 2 states that 'We observe that the predictive performance of MBM measured using MAP is better than that of MVM and MHM on 5 out of the 7 datasets.'\n\nIt is not clear whether the label interaction loss has been used as a regularizer on top of the main NLL-loss or in some other way. If as regularizer, then is there a regularization parameter lambda defining how much each loss contributes to the overall loss? How is this parameter lambda tuned?\n\nSupplementary figure 3 shows the trajectories of MAP, but what would be the trajectories of CV and CMAP?\n\nThe interpretability claims seem too bold in my opinion. It is still hard to interpret high-dimensional boxes and their overlaps. The experiments would benefit from more baselines.\n\npage 9: 'hyperberbolic' -> 'hyperbolic'\n", "summary_of_the_paper": "The paper proposes a method using box embeddings to perform hierarchical multi-label classification.\n", "summary_of_the_review": "The paper is interesting and results seem to be quite good, but the justification of why availability of the label taxonomy should not be available is a bit lacking. The proposed method is quite a simple step from existing literature, and there are several shortcomings about the text highlighted above. The experiments would benefit form more baselines.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NHAj0hKyoTw", "reviewer": "Reviewer_F1ga", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Most existing multi-label classification methods predict labels that are inconsistent with natural/latent taxonomic constraints.\n- This work introduces the multi-label box model (MBM), a method for multi-label classification that combines the encoding power of neural networks with the inductive bias and probabilistic semantics of box embeddings (trainable Venn-diagrams based on hyper-rectangles). This enables MBMs to capture taxonomic relations among labels.\n- MBMs can be trained via gradient descent from data.\n- Box embeddings can be read as calibrated conditional probabilities.\n- The model is naturally more interpretable than other methods for multi-label classification.\n- Partial information about label-label relationships can be injected into the model training, improving the model's consistency. This is done via the \"label interaction loss\".\n-  MBM is theoretically grounded, and it can improve taxonomic consistency while preserving or surpassing state-of-the-art predictive performance.\n- Multi-label classification is an important problem that has real world applications and is relevant to the ML/AI community. Enforcing taxonomic constraints in multi-label classification is an important subproblem.\n- The approach seems quite novel.\n- The method is well-framed in the existing literature.\n- The method is very-well motivated in theory.\n- The baseline models for comparison seem reasonable.\n- Evaluations are conducted over 7 benchmark datasets, which provide label space taxonomies which are additionally useful for evaluating the learned taxonomy.\n- The evaluation metrics seem reasonable.\n- The evaluation results are promising.\n- Experimental results suggest that the label embeddings seem to capture the latent label taxonomy without it being explicitly provided.\n- the approach can reasonably scale to arbitrarily deep hierarchies as might be present in extreme multilabel classification.\n\n\nWeaknesses:\n- Multi-label classification is a well-studied problem, and there has been work on enforcing taxonomic constraints for multi-label classification in the past. This somewhat limits novelty; however, the approach seems novel, so this is not a major concern.\n- The encoder neural network tested is relatively simple: a three-layer MLP; it would be interesting to see whether this method generalizes to more complex neural networks (i.e., can good box representations still be learned with deeper networks on more challenging problems).\n- It would be nice to see which results are statistically significant in Table 2.\n- Figure 2 is a bit hard to interpret due to the large number of settings.\n- Evaluations w.r.t. the taxonomy quality assume the ground-truth taxonomies are perfect, but in practice, there might be a human-bias in how labels are grouped. It would be interesting to see if you could reverse engineer the taxonomy from the embeddings and see if humans agree that the taxonomy makes logical sense and doesn't have many atypical/meaningless relationships detected.\n\nAdditional Questions/Comments:\n- Is there any disadvantage to using hyper-rectangles over different geometric assumptions (e.g., hyperspheres)?\n- Since the method naturally outputs calibrated probabilities and utuilizes taxonomic constraints, I wonder if it is more robust to imbalance problems that tend to plague multi-label classification. Do you have any insights into whether this is true or not?\n- The improvements in the metrics seem relatively small when inroducing taxonomy information in the training. Do you think this is because the base model is already capturing model of the latent taxonomic structure and doesn't need the additional guidance.", "summary_of_the_paper": "Most existing multi-label classification methods predict labels without considering natural/latent taxonomic constraints. This work introduces the multi-label box model (MBM), a method for multi-label classification that combines the encoding power of neural networks with the inductive bias and probabilistic semantics of box embeddings (trainable Venn-diagrams based on hyper-rectangles). MBM has some nice properties: they can be trained via gradient descent from data, and box embeddings can be read as calibrated conditional probabilities. Experiments are conducted on seven benchmark datasets with promising results, and results suggest that the label embeddings seem to capture the latent label taxonomy.", "summary_of_the_review": "Multi-label classification is an important problem that has real world applications and is relevant to the ML/AI community. Enforcing taxonomic constraints in multi-label classification is an important subproblem. The proposed approach is a novel solution to this subproblem, and is well-framed in the literature and theoretically-grounded. the paper is relatively well-structured. Experiments and analysis seem sound, and experimental results are proming: MBM is improve taxonomic consistency while preserving or surpassing state-of-the-art predictive performance on several benchmark datasets compared to reasonable baselines.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BDjPqT77la", "reviewer": "Reviewer_Eo7g", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The paper tackles an interesting problem and it also applies when we have no access to the taxonomy.\n- The presentation is relatively clear and self contained.\n- The technical details appear sound.\n\nWeaknesses:\n\n- The novelty/originality is limited: box embeddings, probabilistic semantics of their intersections, gumbel boxes, bessel volume, etc are all adopted from existing literature. Propositions and the corollary appear more like observations rather than results. \n\n- The overall setup is not well-motivated. While I see why box embeddings can be useful for encoding class interactions and their hierarchies (which are in the form of directed acyclic graphs), it is not very clear to me whether one needs a probabilistic semantics as is used in this work. \n\n- The BoxE paper (Abboud et al, BoxE: A Box Embedding Model for Knowledge Base Completion, NeurIPS, 2020), which is also a box embedding model (proposed for link prediction), is very relevant for this work: It shows that boxes can capture arbitrary relational hierarchies in the space. This is shown for binary relations and therefore it is more general, but it clearly applies to class hierarchies which are only a special case. This suggests that one can model hierarchies using box embeddings in an even simpler way, where box containment implies a subclass relationship (without the probabilistic interpretation). If we already know the class taxonomy, we can even inject this information to the space, i.e., if C is a subclass of D, one can enforce the corresponding C-box to be contained in the corresponding D-box in the space, etc. That is, if the class taxonomy is known, one can provably enforce these using the ideas presented in (Abboud et al). If the taxonomy is not known, then it can still be learned.\n\n- The probabilistic interpretation of boxes may bring in some value, but it is not clear to me whether this is the case, after reading the paper. In fact, the above-outlined approach would achieve the similar goals in a simpler way in my understanding - This may possibly result in an even stronger baseline than the given baselines. Notice that when a C-box is contained in the D-box after training, any prediction for class C will be consistent, since it will also be a D by the space configuration (and there are many possible configurations that can achieve the same thing). \n\n\n", "summary_of_the_paper": "The problem tackled in this paper is that of multi-label classification, where class labels form a taxonomy (i.e., a hierarchy), and the goal is to enforce the predictions to comply with the label taxonomy.  This is a well-known problem in machine learning and it is important as it imposes a more structured learning space, by implicitly enforcing predictions to be consistent with the label taxonomy. The approach taken in this paper is based on box embeddings, where the idea is to view classes as boxes in the space, and apply a certain probabilistic semantics of box embeddings to eventually model taxonomic relations between classes.\n", "summary_of_the_review": "The paper is well-written and interesting but limited in originality and novelty given works of Vilnis et al., Dasgupta el al. It is an interesting application of earlier findings to the domain of multi-label classification, but it is lacking in motivation for the precise model choices. The comparison with earlier works is limited without which the significance of the work is unclear, which led me to suggest a weak reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies multi label classification problem. Particularly, they introduce multi-label box model, which uses probabilistic semantics of box embeddings, representing labels as boxes instead of vectors. Their model is evaluated extensively on 12 datasets, and reviewers agreed the paper was well written and well motivated. While it is pretty straightforward application of box embeddings to multi label problem, it is well motivated and the paper adds to the existing literature on box embeddings. 

Reviewer Eo7g had a concern with experimental setting, including missing a baseline Abboud et al. (2020). Even after the baseline was added, the reviewer was not convinced about the model’s performance, as the baseline was not extensively tuned. The authors responded with the heavy computational costs of associated with tuning the margin. Reviewer X5cP also pointed the problem with HMC dataset, for which the experimental results should be updated. Given the issues, the paper could benefit from another round of revisions.
Decision: Accept (Poster)