Hyperbolic Embedding Inference for Structured Multi-Label Prediction
Authors: Bo Xiong
Michael Cochez
Mojtaba Nayyeri
Steffen Staab
Venue: NeurIPS 2022 Accept (2022)
Keywords: Structured multi-Label prediction
hyperbolic geometry

Abstract: We consider a structured multi-label prediction problem where the labels are organized under implication and mutual exclusion constraints. A major concern is to produce predictions that are logically consistent with these constraints. To do so, we formulate this problem as an embedding inference problem where the constraints are imposed onto the embeddings of labels by geometric construction. Particularly, we consider a hyperbolic Poincaré ball model in which we encode labels as Poincaré hyperplanes that work as linear decision boundaries. The hyperplanes are interpreted as convex regions such that the logical relationships (implication and exclusion) are geometrically encoded using the insideness and disjointedness of these regions, respectively. We show theoretical groundings of the method for preserving logical relationships in the embedding space. Extensive experiments on 12 datasets show 1) significant improvements in mean average precision; 2) lower number of constraint violations;  3) an order of magnitude fewer dimensions than baselines.

Hyperbolic Embedding Inference for Structured Multi-Label Prediction
Bo Xiong
Michael Cochez Vrije Universiteit Amsterdam Discovery Lab, Elsevier Amsterdam, The Netherlands
Mojtaba Nayyeri University of Stuttgart Stuttgart, Germany
University of Stuttgart Stuttgart, Germany
Steffen Staab University of Stuttgart University of Southampton Stuttgart, Germany
Abstract
We consider structured multi-label prediction problem where the labels are orga- nized under implication and mutual exclusion constraints. A major concern is to produce predictions that are logically consistent with these constraints. To do so, formulate this problem as an embedding inference problem where the constraints are imposed onto the embeddings of labels by geometric construction. Particu- larly, we consider hyperbolic Poincaré ball model which we encode labels as Poincaré hyperplanes that work as linear decision boundaries. The hyperplanes are interpreted as convex regions such that the logical relationships (implication and exclusion) are geometrically encoded using insideness and disjointedness of these regions, respectively. We show theoretical groundings of the method for preserving logical relationships the embedding space. Extensive experiments show 1) significant improvements in mean average precision; 2) lower number of constraint violations: 3) an order of magnitude fewer dimensions than baselines.
12 datasets
Introduction
Structured multi-label prediction a task aiming to associate every object with multiple labels that are semantically constrained in structured manner (e.g., by implication and exclusion constraints). This task is of growing importance in many applications such as image annotation 2 text categorization 3 ! and functional genomics [5] d One of the central concerns of the task is produce predictions that are logically consistent with the constraints of the labels. For example, a protein must be labeled to have the function nucleic acid binding if it is already labeled to have the function RNA binding (i.e., implication) and must not have the function drug binding (i.e., mutual exclusion).
Various works have been proposed to improve the prediction consistency 7
g
9
10
One line A key
of work called label embedding aims to represent labels as low-dimensional vectors disadvantage of the vector-based representations is that they only capture weak forms of correlation or "similarity" between labels, but do not strongly enforce the logical relationships. Another line of work 7 14 & imposes these logical constraints directly the losses of neural networks. However, they do not explicitly learn the representations of labels and typically require a complete label taxonomy, which is not always available in and scalable rerl-world settings [|]
Correspondence bo.xiong @ deiiss.uni wui-stuttgart.co
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Implication
Exclusion:
Girl
0.8
Worten
0.7
Plant
Person
Person
Mother
0.5
Paren
0.4
ther
Women
Parent
0.2
0.1
Tree
Plant
Tree
Mother
Father
0.0
person women Sirl
parent
mother
plant
tree
father
(b)
Figure I: (a) A HEX graph describing the logical relationships (implication and exclusion) between different labels; (b) The learned label embeddings (linear decision boundaries) in the Poincaré ball, where all constraints the HEX graph are respected; (c) The prediction scores of given instance of mother respect all constraints in the HEX graph, where each score is calculated as the confidence of instance embedding being member of the convex region of the corresponding label embedding.
Simbedding-based inference [5]. which imposes logical constraints directly to the label embeddings, able inductively infer the underlying label relationships from incomplete labelings [16]. Once all embeddings are adhering to the constraints, each label can be predicted independently without access- ing the label relationships, which significantly reduces the computation cost during inference [15]] The key idea, which is inspired by the Venn diagram [1 17 or set-theoretic semantics [18], is to represent each label as a convex region [ A prominent example is the multi-label box model (MBM) [1] that models label implications as box containments. However, MBM learns box-like decision boundaries, which are typically not compatible with standard classifiers (i.e., hyperplane margin-based models such as logistic regression [19). Besides, box models suffer from theoretical limitation, i.e., lower-way intersections enforce higher-way interactions 20) Finally, current methods ignore the importance of constraining mutual exclusion, which is essential as otherwise, model could trivially obtain zero implication violation by assigning the same score to all labels.
In this paper, we consider structured multi-label prediction problem with implication and mutual exclusion constraints that are jointly described by hierarchy and exclusion (HEX) graph (see aa for an example). The key idea of our method is to transform the logical constraints into soft geometric constraints in the embedding space. In particular, we consider a hyperbolic Poincaré ball model that has demonstrated ad udvantages in representing hierarchical data [2] 22 and assign each label Poincaré hyperplane that has several favorable theoretical properties in classification. Each Poincaré hyperplane can be interpreted as a convex region such that the implication and mutual exclusion are modeled by geometric insideness and disjointness between the corresponding regions, respectively. In this way, multi-label classifier can be defined by measuring the confidence of an instance having a label as geometric membership. Unlike other hyperbolic region-based models such as hyperbolic cones [2]] and hyperbolic disks 24], Poincaré hyperplane works as a linear decision boundary and can seamlessly integrated into existing margin-based classifiers such as hyperbolic logistic regression [1 Figure bb) shows an example of the learned label representations that respect all the constraints given in Figure a)) We show theoretical groundings of the proposed method on modeling implication and mutual exclusion. Extensive experiments on 12 multi-label classification tasks show the model's 's capability to improve the mean average precision significantly while keeping the number of constraint violations low and requiring order of magnitude fewer dimensions.
2 Preliminaries
Poincaré ball model The Poincaré ball (D"", _) is one of the models of hyperbolic geometry that is very suitable for representing hierarchies due to its exponentially growing volume 255 The Poincaré ball is defined as an open n-ball n {x E IR" ||x|| < 1} equipped with a Riemannian metric df E5 where Ax T- xxp= _ I, is the Euclidean metric tensor, Xy is the conformal
2
factor, and ||- 112 denotes the I norm in Fuclidean space. The distance between two points x, y E m can be defined by (x, y) = cosh 1 (1 (1+ 2
x |
x 10 1vl)

---

Structured multi-label prediction Let X' S R" denote an n nd-immnsiinnll instance space and C = (2.12..... }. C 2 2 denote the finite set of possible labels. Given a set of N training examples D
((r,, T.) 1 Sis N, 34 E X I, C f)), multi-label prediction aims to learn a labeling function f:X 2 mapping from the instance space to the powersel of the label space, fxx C.
Structured multi-label prediction additionally imposes a set of prior-known logical constraints over the labels, namely, the predictions must be logically consistent with these constraints. Analogous to Mirzazadeh et al. [I5], we consider two forms of logical constraints between labels: implication and mutual exclusion. Specifically, an implication of the form le = 4o imposes the constraint that whenever an instance is labeled as la then it must also be labeled as 4,, i.e., la + lo is a shorthand notation for Vz E X. la fax + i fz)). Mutual exclusions are constraints of the form The V -4g, implying that an instance cannot be simultancously labeled as L, and 4, i.c., -l, V -ls shorthand notation for Va E X11a & f(x) V l & f(x). We can concisely represent a set of implication and exclusion constraints with hierarchy and exclusion (HEX) graph (2).
Definition 1 (HEX graph 2 A HEX graph G = (V, En, E.) is graph consisting of set of nodes V {vi,.... ). directed (hierarchy) edges Ey, C V V, and undirected (exclusion) edges E. - V x V, such that the subgraph G₁, = (V. Ex,) is a DAG and the subgraph Ge = (V, Ee) has no self loop. Each node v, E V represents the label 1, A directed edge (U,, E E,,, represents the implication 1 lj. and an undirected edge 0y) E Ee represents the exclusion -l, V -lj-
Note that an arbitrary HEX graph may contain redundant edges. A hierarchy edge (1z, U2)) is redundant when there is a path in GR from et to Vi which does not contain the edge (vz, Uy)) Similarly, an exclusion edge (vs, U3) is redundant when there is another exclusion edge connecting their ancestors (or connecting one node's ancestor the other node). We can transform HEX graph into an equivalent HEX graph by adding or removing redundant edges. In this paper, we only consider HEX graphs that have a minimal number of edges, we call such HEX graph a minimal sparse HEX graph (see Fig. ITa) () for an example). Given a minimal sparse HEX graph, we define the HEX- -Eo-prrpertt as
Definition 2 HEX-property)) A labeling function f has the HEX property with respect to HEX graph G for all X, fx) respects all constraints represented by G.
We also call such function f logically consistent w.r.t G. Given the HEX graph and the HEX-propert structured multi-label prediction is formally defined as a constrained optimization problem.
Definition 3 (Structured multi-label prediction). The structured multi- prediction task with respect to a training set D = {(x;, L.) | < S N, E X, Li C C), minimal HEX graph G (V, En, Ee), and multi-label prediction function f, is the task of learning I such that the function f minimizes LiDiLE) loss(] (=;), Li), with loss a predefined function, while attempting to maintain the HEX-property with respect to G.
Note that this definition allows for a soft interpretation of the constraints, meaning that the goal is to adhere to all of them, but we do allow for loosening some if necessary. For example, a mutual exclusion constraint is allowed to loosen when an instance (e.g., image), though rarely happens, is simultaneously labeled as two mutual exclusive labels (e.g., dog and car).
3 Hyperbolic embedding inference
We consider learning a real-valued ranking function h : X L + [0, 1]. where the output is interpreted as the confidence of an instance x X' having label E L. Afterward, binary multi-label classifier f X >> 2 can simply obtained by thresholding the ranking function with a threshold t, i.c., f(r) {l h (z, () > vl E C}. The objective of h is to assign higher scores to positive instance-label pairs than that of negative instance-label pairs.
3.1 Geometric construction
Given an n-dimensional Poincaré ball D", we associate each instance x, X with a point in the Poincaré ball and associate each label 4 e L with a Poincaré hyperplane, such that its corresponding positive and negative instances are correctly separated by the hyperplane.
? Deng et al.
2 use subsumption, which is the inverse relation of implication that here.
3

---

Poincaré hyperplanes Let IB" denote the set of n-balls in R" whose boundaries OB" intersect the Poincaré ball ID" perpendicularly. Poincaré hyperplanes are defined by OB" n (see Fig. a()) plus all linear subspaces going through the origin. For the former cases, Poincaré hyperplane can be uniquely defined by its center point that has a minimal distance to the origin.
Definition 4. Given a (center) point с E D where с + 0, the Poincaré hyperplane is defined as He {p E D :g (loge (p) - 0}
where с the center point and č E T.Da is the normal vector passing through the origin O.
Intuitively, this corresponds to the union of all geodesics passing through с while orthogonal to the normal vector č E T.D In the case where is the center of the hyperplane, č must simultancously pass through с and the origin. Hence, can be simply taken as с without loss of generality. For the special case where passing through the origin. In this paper, we exclude these special cases by assuming + O.
0, the Poincaré hyperplanes are all linear subspaces (Euclidean planes)
Geometric intuition Essentially, the Poincaré hyperplane works as a linear decision boundary that separates the embedding space into two regions where the smaller region (i.e., convex hull) interpreted as the space of positive samples while the other one is interpreted as the space of negative samples. Two reasons motivate us to model labels as Poincaré hyperplanes: 1) Modeling labels as hyperplanes has several desired theoretical advantages in margin-based classifiers. Our model shares the same philosophy as existing learning frameworks such as hyperbolic logistic regression [99] and hyperbolic SVM 26:: 2) More importantly, unlike Euclidean space that is flat, hyperbolic Poincaré ball a curved space in which there are infinitely many non-patt man-palllle hyperplanes which do not intersect, implying that linear decision boundaries in hyperbolic space capture more complicated set-theoretic interactions, such as implication and mutual exclusion.
Enclosing balls Given Poincaré hyperplane He, we call the corresponding n-ball B. that en- closes He its enclosing n-ball. Formally, an enclosing n-ball IB (o, r) is defined by B" (o,r) = {p: |p oll S r), where E IR and r are the center point and the radius, respectively. Given He. we have the following closed-form representation B
Proposition 1. Given Poincaré hyperplane He where с # O. there exists an n-ball B (e),Te)
such that He C Be (e). i.e., He is a subset of DB (Oe: re). B is uniquely given by
(1+
||c|2) C, 2||c|
1 - 1-||c|
B = B"
2||c||
Proof sketch. The key idea is to solve a quadratic equation given by the fact that the radius of B the radius of D and the distance from the center of D" to the center of BB must satisfy the Pythagorean theorem 22. Full proof is in the supplementary material.
3.2 Geometric interpretation
Our main idea is to transform the logical relationships between labels into geometric relationships between their corresponding enclosing n-balls. In particular, the implication modeled by the geometric insideness while the mutual exclusion is modeled by the geometric disjointness.
Implication
The logical implication between two labels is interpreted as geometric relations between n-balls, i.e., n-ball insideness illustrated Fig. (b). In particular, an n-ball B (Ow rw)) contains B,, (ou,Tu) Fu) if and only if [lo,, Ow + Tu < Tw: and thus we can create an insideness loss defined by
Cinside (BBi (IB Bw) = max lou - Tu Tw].
(3)
Clearly, the insideness loss term satisfies the properties correctness and transitivity Lemma (Correctness). B, is inside of B if and only if Linsite (Bu, Bw) = O.
Lemma 2 (Transitivity). If Linside (Bu, Bww 0 and Liside (B BB) O, we have Linside (Bu. BB) 0.
Linside (Ba Bw) + Linside (Bw,Byy S Lisidee WI
IIn this paper, we distinguish normal vectors from regular points by adding an arrow on top its letters. "Note that by using the metric in the Poincaré ball, each region has infinite (exponentially growing) volume.

---

[8
Poincaré Ball
Fc
Poincaré Ball
Poincaré Ball
Oc
IB, (0
Bu
B
(a)
(b)
(c)
Figure (a) Poincaré hyperplane defined the intersection between the Poincaré ball I and the boundary of an n-ball Be. The Poincaré hyperplane is uniquely parameterized by a center point c, and the corresponding n-ball (its radius and center) can be uniquely determined by Proposition[] | (h) Label implication is interpreted n-ball insideness.
(c) Mutual exclusion is interpreted as n-ball disjointedness.
Mutual exclusion Similarly, we interpret mutual exclusion as geometric disconnectedness between n-balls illustrated in Fig. cc. Bu disconnecting from 1w can be measured by subtracting the distance between their center points from the sum of their radii. Inversely, the corresponding loss is
Laisjoint (B, (w) max (0, Fw + Tu l|o,, Ow
Again, the disjointedness loss term satisfies the correctness property Lemma (Correctness). B, disconnects from B if and only if Laiejoini (Bu,, Bww =
3.3 Classification and learning
Given the embeddings of instances and labels, instance can be classified by measuring the geometric membership, i... the confidence of point E ID' being inside the enclosing ball B.
Membership and non-membership Formally, given instance embedding D and label embedding associated with an enclosing n-ball Be. The confidence of an instance p being inside the enclosing n-ball Be can be measured by subtracting the distance between the center point of B. and from the radius of B.. The corresponding loss defined as the inverse of the measure, given by
membership (p, Be (Oc; re)) max (0, 0. p re).
Symmetrically, for negative instance-label relations, the loss of mon-membesshi can be defined as
L membership (p, Be (Oe: re)) max re lo. pl]}-
(6)
Clearly, we have the following properties that follow directly from the definitions.
Lemma 4. A point p is member of Be if and only if Emembership (p, Be) = 0. Lemma 5. A point p is not member of Be if and only if Lnon-membershhp (p, BB) = O.
Lemma 1-2, Lemma 3, Lemma 4-5 immediately follow the definitions of geometric insideness, disjointedness, and membership, respectively.
We aim to learn an encoder Eg (i.e., hyperbolic neural network whose designs depend on the datasets), where is the trainable parameter, and a function C which maps labels the center points of the corresponding Poincaré hyperplanes in the Poincaré ball.
Now, define h( 1)
(Lnon membership Egg (a:), C(I) Lmmmbershhp (F)())) C(I))) our rank- ing function, where d is the sigmoid function. The final classification function is then defined by f(x) h (a,, 1) > 0.5). We call classifier hyperbolic multi-label embedding inference (HMI). Given a HEX graph, HMI has the following guarantee.
Proposition (HEX-property). The classification function of HMI has the HEX property with respect to G if for every constraint in G, the corresponding loss term is 0.
Learning with soft constraints Let De (( (2,, L.) D,,, In E L2} be set of positive instance-label pairs and D ((,,, L.) E D. h E L. La} be the set of negative
instance-label pairs. By combining the loss functions of membership, non-membership. insideness and disjointedness, the learning objective can be formulated
5

---

min 0,0
Σ
memberrsii (Es (Ii) Be(r)) +
Σ
mnnn-membership
(Ea (2i) (BC())))
(Rr DDD
(Mida)ED-
+t
Lineide (Be(L), BBu,) + (B.
Σ
Σ
Ldissoina (Bc,,)), Be(,))))))))))))))))))))))))BB
(7)
(Hs,y))EE
The first two terms are losses for positive and negative samples while the last two terms are implication and exclusion constraints, respectively, with À being the penalty weight of the constraints.
The following corollary shows that model has strong inductive bias for preserving consistency. Corollary 1. Given HEX graph G of labels, if the loss terms Linside and ddisjoint are O, then the learned prediction function f
logically consistent.
Classification via hyperbolic logistic regression A key advantage of our method is that the losses of constraints are compatible with other (margin- hyperbolic classifiers such as hyperbolic logistic regression (HLR) 19 and hyperbolic support vector machine (HSVM) 26 In our experi- ment we explore HLR, which formulates the logits as the distances from an instance to Poincaré hyperplane of label. That is, ((x, I) d (Fa (x:) Ho (4)) d(p, H&) has the following closed form:
( 2 c) + P. c)
d(p. He) sinh
(8)
(-c) + pl |
where @ the Möbius addition [9]. The classifier is defined by [(x) = (1 a (h (x, 2 0.5, 4/ We dub such classifier combined with HMI as HMI+HLR.
[}} where is the sigmoid function.
4 Evaluation
4.1 Experiment setup
Datasets We consider 12 datasets that have been used for evaluating multi-label prediction meth- ods & [[]. These consist of 8 functional genomic datasets (28]. 3 image annotation datasets 29 30 and text classification dataset 3T. All input features are pre- pre-processed in the same way as described by Patel et al. [1I] For all datasets, the implication constraints (label taxonomy) are given. Following Mirzazadeh al. 16 we add exclusion constraints between sibling nodes whenever this does not create a contradiction (i.e., they share common descendant nodes). We also explore other strategies for deriving exclusions, but no significant difference was observed (see the supplement for analysis). Similar to MBM [I]] and its baselines, we sample 30% of the implications and exclusions constraints for training the model.
Hyperbolic encoder We adopt simple hyperbolic linear layer the instance encoder for all datasets. A single-layer hyperbolic fully-forward linear layer defined by [W=[W.b](C = tanh @ [W 6 4 b), with & being Möbius matrix-vector multiplication defined by M & x
tanh
[Mx] x
(anh ([xx))([I())))
Mx where W E Raxd is a trainable matrix and x is point x E |M*]
D Mx # 0. denotes Möbius addition given by x + y (1+2(x,y)+1ss x x) 11 tanh
1+2(x,y) [x| ylF
denotes Möbius version of pointwise non-linearity given by tanh @ (x) expo (lanh (x))).
with expo and logo being the exponential and logarithmic maps, see for more details.
Baselines We compare our approach with both classical vector-based and state-of-the-arr region- based embedding methods. In particular, we consider two vector-based models: 1) The multi-label vector model (MVM) 33.. which encodes both inputs and labels as Euclidean vectors; 2) the multi- label hyperbolic model (MHM) used by Chen et al. [3] which represents inputs and labels as hyperbolic points; and two box models: 3) the proppobbaillittcc box model (BoxE) [+]] and 4) the probabilistic multi-label box model (MBM) [I]] that encodes both instances and labels as oxis-parallel hyper-rectangles. Besides, we compare with 5) hyperbolic logistic regression (HLR) 19 since also
encodes labels Poincaré hyperplanes (but does not geometric constraints). Furthermore,

---

Table 1: Comparison of performance and consistency 12 datasets, where underline indicates the hest results over sebeddinngbbscee methods, and boldface indicates the best results over all methods. implemented HMI, HLR and HMI+HLR. Other results are taken from Patel ct al. [T]] All metrics are averaged across 10 runs with
random seeds (standard deviations are relatively small (in range x 10-1. 2.3 10 "J) and are hence omitted).
Ours
Embeddings
Non-embeddingg C-HMCNN
Dataset
Metric
HMI
HMI+HLR
MVM
MHM
BoxE
MBM
HLR
mAP
38.53
38.50
37.94
31.90
37.30
38.42
37.98
38.41
ExprFUN
CmAP
38.72
38.62
37.41
32.02
37.92
38.67
37.44
38.41
HCV
0.92
1.07
1.97
1.92
4.79
1.87
2.17
mAP
34.82
34,84
31.61
28.74
31.96
34.61
34.05
34.35
CellcycleFUN
CmAP
34.90
35.00
31.33
28.89
32.70
34.78
34.11
34.35
HCV
1,30
1.32
3.45
1.78
4.02
1.35
2.30
MAP
36.71
36.71
24.16
24.40
26.66
28.71
26.65
28.19
DerisiFUN
CmAP
36.94
36.89
24.35
24.52
26.96
28.88
26.83
28.19
HCV
0.73
0.87
4.01
0.85
2.27
1.43
2.30
mAPT
36.47
36.44
24.21
26.57
27.97
29.62
28.29
29.18
SpoFUN
CmAP
36.43
36.54
24,55
26.79
28.38
29.78
28.31
29. 18
HCV
0.92
1.05
4.73
1.69
2.75
1.53
1.98
MAPT
48.63
48.50
44,97
40.32
46.75
48.45
48.65
48.61
ExprGO
CmAP
48.68
48.61
41.84
40.70
47.28
48.56
48.65
48.61
HCV
1.37
1.45
7.05
5.19
5.74
1.91
1.35
MAPT
45.58
45,51
44.19
39.74
43.08
44.93
40.28
45.61
CellcycleGO
CmAP
45.58
45.53
41.02
39.76
43.79
45.01
40.30
45.61
HCV 4
1.19
1.12
3.03
2.49
2.16
3.26
mAP f
42.31
42.12
41.13
40.10
40.44
42.02
40.33
42.24
DerisiGO
CmAP
42.38
42.28
38.21
40.20
40.73
42.12
40.35
42.24
HCV
0.86
0.99
3,46
2.02
3.16
1.13
mAP
42.70
42.74
42.20
39.70
40.88
41.74
39.22
42.77
SpoGO
CmAP
42.76
42.77
39.04
39.77
41.27
41.54
42.77
HCV
0.95
1.20
2.77
1.90
3.89
1.80
mAP
80.43
80.43
73.68
75.62
80.44
80.06
78.87
80.04
Enron
CmAP
80,50
80.47
66.87
75.68
80.46
80.05
78.94
80.04
HCV
0
2.53
0.36
0.20
0.03
0.04
mAP
79.19
79.10
72.65
56.86
43.71
79.14
77.90
76.23
Diatoms
CmAP
79.40
79.36
72.18
56.07
45.16
79.23
78.07
76.23
HCV
0.17
0.18
19.20
5.55
6.39
0.34
6.36
mAP
90.67
89.60
78.22
65.30
83.71
69.26
88.33
90.26
Imclet07a
CmAP
90.89
89.71
77.46
66.01
84.73
69.48
88.45
90.26
HCV
0,20
0.19
22.86
4.75
12.73
2.40
1.77
mAP
89.19
89.20
88.59
75.69
87.95
89.56
88.91
89,22
Imclef07d
CmAP
90.00
90.02
86.87
76.95
88.93
90.07
87.38
89.22
HCV
0.37
0.36
11.02
7.56
11.93
5.66
6.88
mAP
1.75
2.42
6.33
7.58
5.75
3.5
5.25
3.08
Avg. Rank
CmAP
1.58
2.08
7.16
7.41
5.25
3.58
5.41
3.33
HCV
2.25
2.75
7.42
5.25
7.25
4.25
5.58
1.00
compare with 6) C-HMCNN, a state-of-the-art non-embedding based method that injects hierarchy constraints directly into the loss function without embedding labels. A notable difference is that C-HMCNN needs the full hierarchy constraints as its input. Finally, we also implement HMI+HLR, combination of our proposed constraints with HLR for ablation study.
Implementation details We implement HMI, HLR and HMC-HLR using PyTorch 34 and train the models on NVIDIA A100 with 40GB memory. We train HMI, HLR and HMI+HLR using optimizer implemented by the Geoopt library RG with a batch size of 4. We also explore some larger batch sizes but it does not yield better results, which is also observed in Wehrmann et al. 14] We set the dropout rate to 0.6 suggested by [14] to avoid the case that the model overfits the small training sets. We employ early-stopping strategy with patience 20 save training time. The results of other baselines are as reported by Patel et al. [I]] that we closely follow.
Riemannian Adam 35
The learning rate is searched from (le 4, 5e 4, le 3,, 5e le 2). The penalty weight of the
violation is searched from (1e - 5, 5e 4, 1e ----444 4, 5e 3, 1e - 2) and we also show its impact in an
ablation. The best dimension per dataset searched from {32, 64, 128, 256). which one order of magnitude lower than that used by Patel et al. MI ((250, 500, 1000, 1750}). All methods have been run 10 times with random seeds and the average results are reported. We omit the standard deviations since they are in a very small range (22 x 10 2.3 x 10 10-3)). Our code is open available at
https//github.com//oong/oo//////

---

CD
CD
2
5
HM4
MHMM
HMI HMMCNNN
NHM
CHMMCNN
MVW
HMI-HLR C HMMNN MBM
HMI
BoxE
Boxt HUR
HUR
HM +IIL
HLRR
MBM
BoxE
MBM
MHM
(a) mAP
(b) CmAP
(c) HCV
Figure 3: Critical diagrams of the post-hoc Nemenyi test across all 12 datascts.
Evaluation protocols In line with Patel et al.
consider Mean Average Precision (mAP)
which summarizes the information of precisions and recalls with varied thresholds. We also report two metrics that additionally take the constraints into account: 1) Constrained mAP (CmAP) is variant of mAP that replaces the score of each label with the maximum scores of its descendant labels in the hierarchy [T]. Hierarchy Constraint Violation (HCV) [ measures the extent which the label scores violate the implication constraints regardless of true labels for the instances. HCV computed as HCV
TeiTEi Dn E(.,)EEF, 1 (ht h > 0), where h, means the prediction score label 1, Clearly, a lower value of HCV implies higher consistency in the predictions.
4.2 Main results
As Table shows, our method HMI either achieves the best (7-8/12 datasets) or competitive (4-5/12 datasets) performance (mAP and CmAP) over all compared methods. HMI outperforms all methods w.r.t the average ranking of mAP/CmAP, showcasing the advantages of HMI. We observed that the CmAP is close to mAP, indicating that the model is adhering to the label constraints [1]] In terms of predictive consistency (HCV), HMI consistently achieves the best the second- results. Note that C-HMCNN always gets zero HCV because it exploits the complete hierarchy. HMI achieves competitive HCV, despite only using 30% of the hierarchy.
Statistical significance Follow- ing Patel al. [ and Giunchiglia and Lukasiewicz g. we test the
Table 2: Results of Wilcoxon test over HMI against baselines.
statistical significance of the perfor- mance across all datasets. First, we perform the Friedman test [37 and show that there exists significant difference w.r.t. all metries with
Method
mAP
CmAP
CV
HMI vs C-HMCNN
5.8
4.4
x 10
HMI vs MBM
3.3 10
24 10-4
4.9 x 10-4
HMI vs HMI+HLR
2.3 x 10
388 10-2
9.7 10-1
p-values < 0.05. Next, we conduct the post-hoc Nemenyi test to verify the statistical differences of the average ranking. The critical diagram w.r.t the average ranking of mAP/CmAP is shown in Fig. 5 in which the methods that have significant differences (significance level 0.05) are connected by a horizontal line. As shown in the diagrams, it is clear to conclude that there is statistically significant difference w.r.t mAPs/CmAPs of HMI and HMI+HLR against MVM, BoxE, MHM, and HLR but not the two strong baselines (MBM and C-HMCNN), We further perform Wilcoxon test that considers not only the differences in rankings but also the numerical differences in the performance. The Wilcoxon test results show that there a statistically significant difference between the mAPs/CmAPs of HMI and the two strong baselines with p-value < 0.05. In terms of HCV, our statistical significance test in Figure 3 and Table 2 shows that HMI and HMI+HLR significantly outperform MVM, BoxE, MHM, HLR, and MBM but not C-HMCNN since it has HCV. However, we observed that the predictive performance (mAP, CmAP) is not fully proportional to the HCV, e.g.. HMI outperforms C-HMCNN w.r.t. mAP/CmAP many of the datasets even though C-HMCNN has zero CV.

---

Classification via hyperbolic logistic regression To validate whether our proposed geometric constraints are able to improve hyperbolic logistic regression (HLR) [9]] we implement HMI+HLR.
combination of our proposed constraints with HIR as described in Section 3.3 Table show that HMI+HLR outperforms HI.R with statistical confidence, showcasing that HMI is able to improve the predictive performance and consistency of HLR. However, there is significant difference (with between the two variants of our method (HMI and HMI+HLR).
p-value larger than 0.05 in Table 2
4.3 Ablation studies & parameter sensitivity.
For further ablation, we introduce one additional metric. Exclusion Constraint Violation (ECV) measures, analogous to HCV, the fraction of the exclusion constraints violated by the predictions ie., ECV P E. _ Eu.d)EEE (J+ A f We introduce this because HCV can be made zero trivially by associating all labels with the same score. Hence, in the ablation study, we will show how the exclusion constraints (the results of ECV) complement HCV and influence the overall performance.
Impact of penalty weight Ta-
ble 3 shows the results of HMI on "CellcycleFUN" and "Cellcy- cleGO" dataset. We observed that with different penalty weights, the obtained results are slightly differ- ent. Even without penalty (A = 0). the model already achieves ac- ceptable results, in particular, it outperforms MVM, MHM, and BoxE, indicating that our hyper- bolic model, to some extent, is ca-
Table 3: Impact of violation penalty weight À on CellcycleFUN and CelleyoleGO dataset.
À
0.001
mAP
33.87
34.78
34,82
34.76
32.28
CelleydleFUN
34.03
34.83
34.85
IICV
2.33
1.87
1.30
1.04
0.75
4.33
3.77
1.35
MAP
40.26
41.47
45.58
45.36
41.28
CalleycleGO
42.05
45.60
40.75
HCV
2.28
1.57
1.19
0.99
0.86
3.27
1.71
1.34
pable of capturing label hierarchies without any explicit constraints. However, as Table 3 shows, a proper 0.001,
0.005 and À 0.01 indeed improves the performance and consistency. Finally, we observed that increasing À to 0.1, though further improves consistency (HCV and ECV)), does not further improve mAP and CmAP. We conjecture that this is because large À would encourage the model to "overfit" the given constraints while "underfittin" the classification loss.
Impact of implication & exclu- sion To study the roles of impli- cation and exclusion. We imple- mented three variants of HMI by removing either implication clusion, or removing both of them. MableAdepice the results of these variants. It is clear that both impli- cation and exclusion constraints improve the base model that has no constraints. When implica- tion and exclusion are jointly con-
Table 4: Impact of implication and exclusion constraints Cellcycle- FUN and CelleycleGO dataset.
Datasel
Metric
IJMI
w/o implication
w/o exclusion
BUn constraints
CelleycleFUN
mAP
34.82
$4,701
$4,74
13,87
CmAP
34.90
34.75
34,82
34,03
HCV
1.30
2.34
1.45
2,35
ECV
2.40
2.67
3.63
4.35
CellcycleGO
mAP
45,58
42.56
44.50
40.26
CmAP
45.58
42.56
45.31
39.87
HCV
1.19
2.16
1.73
2.28
BCV
2.17
3.68
3.07
3.98
strained, the performance is significantly improved again. We also observed that implication and exclusion constraints, to some extent, do complement each other, e.g., by only using implication (resp. exclusion), the model archives lower ECV (resp. HCV). Finally, we observed that even without exclusion, our model still slightly outperforms MBM, showcasing the advantages of hyperbolic space for modeling hierarchies.
Impact of sampling ratio To study whether our method is able to preserve logical constraints from incomplete label constraints we compare the performance of HMI with different ratios for sampling the training constraints. As Figure aa) depicts, with zero sampling ratio, our method already achieves acceptable results. We conjecture that this is because some constraints can be learned from the data. However, Figure clearly shows that including constraints indeed helps to improve the performance. Making the sampling ratio larger than 30-40% does not lead to significant

---

performance gain. We conjecture that this is because certain ratio of training constraints is sufficient for inferring the full set of constraints.
Impact of embedding dimensionality We study
how the choice of dimen-
46
sionality affects performance. As Figure depicts, HMI achieves acceptable results even in a very low dimension (n 100). When increasing the dimension an order of magnitude (n 1000). the performance grows only slightly. Note that all reported baselines achieved acceptable results with dimensions in [500, 1000, 1750] (see hyperparameter settings the Appendix Patel et al. MI). We conjecture that the reason we can achieve good performance with fewer dimensions is that the hyperbolic hyperplane is more suitable for representing hierarchical decision boundaries.
(CnAP)
44
44
42
mAP (CellcycIeFUN)
42
MAP (CellcycleFUNN
CmAP (CellcycleFUN)
CmAP (CellcycleFUN)
40
(CellcycleGO)
(CellcycleGO)
38
CmAP (CelicycleGO)
38
CMAA (CelicycleGO)
E
36
36
34
40
100
101
10a
10a
Sampling ratio
Dimensionality
(a)
Figure 4: (a) The variation of performance w.r. the sampling ratio. (b) The variation performance the embedding dimensions.
Comparison with MBM with only implication or without any constraint To faithfully study the advantages of hyperbolic hyperplane on modeling label relations than that of the box model (MBM), we also implement two versions of HMI by considering only (30%) implication constraints and without any constraint (sampling ratio= 0), respectively. Our Wilcoxon test Table shows that HMI with only implication and HMI without any constraint still outperform their corresponding counterparts of MBM on CmAP and HCV (with p-value < 0.05) while achieving comparable results on mAP (i.e., with better average ranks but without statistical significance, we believe this is because mAP is less sensitive the constraints than CmAP).
Table 5: Results of Wilcoxon test on HMT against MBM in the settings where only implications available and without any constraint. mcans statistical difference between the compared methods.
Method
mAP
CmAP
CV
HMI (impl.) vs MBM (impl.)
2.4 x 10
1.2 x 10
HMI (no conts.) vs MBM (no conts.)
1.3 x 10-2
6.1 10-3
Conclusion
In this paper, we focus on structured multi-label prediction task whose output is supposed to respect the implication and exclusion constraints. We show that such problem can be formulated hyperbolic Poincaré ball space whose linear decision boundaries (Poincaré hyperplanes) can be interpreted convex regions. The implication and exclusion constraints are geometrically interpreted as insideness and disjointedness, respectively. Experiments on 12 datasets show significant improvements in mean average precision and lower constraint violations, even with an order of magnitude fewer dimensions than baselines.
Acknowledgments
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS for supporting Bo Xiong. Bo Xiong is funded by the European Union's 's Horizon 2020 research and innovation programme under the Marie Sklodowska-Cuun Nayyeri is funded by the German Federal Ministry for Economic Affairs and Climate Action under Grant Agreement Number 20002F08 Meistee-Meister).. We thank Prof. Martin Keller-Ressel and all anonymous reviewers for providing valuable suggestions that improved the paper.
grant agreement No: 860801. Mojtaba
10

---

References
[1] Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. Large-scale object classification using label relation graphs. In ECCV (1), volume 8689 of Lecture Notes in Computer Science, pages 48 48-64. Springer, 2014.
[2] Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome. Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. Large-scale object classification using label relation graphs. In European conference on computer vision, pages 48 64-4..366 Springer, 2014.
[3] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vis., 123(1):32-73, 2017.
[4] Federico López, Benjamin Heinzerling, and Michael Strube. Fine-grained entity typing in hyper- bolic space. In RepLANN.P@ACI, pages 169-180. Association for Computational Linguisties, 2019.
[5] Thi Thuy Duong Vu and Jachee Jung. Protein function prediction with gene ontology: from traditional to deep learning models. PeerJ, 9:e 12019, 2021.
[6] Maxat Kulmanov and Robert Hoehndorf. Deepgoplus: improved protein function prediction from sequence. Bioinformatics, 36(2):422-422 -292 2020.
[7] Ricardo Cerri, Rodrigo C. Barros, and André Carlos Ponce de Leon Ferreira Carvalho. Hierarchical multi-label classification using local neural networks. J. Comput. Syst. Sci., 80(1):39-555 56, 2014.
[8] Eleonora Giunchiglia and Thomas Lukasiewicz. Coherent hierarchical multi-label classification networks. In NeurlPS, 2020.
[9] Shikhar Murty, Patrick Verga, Luke Vilnis, Irena Radovanovic, and Andrew McCallum. Hierar- chical losses and new resources for fine-grained entity typing and linking. In ACL (1), pages 97-109. Association for Computational Linguisties, 2018.
[10] Jonatas Wehrmann, Ricardo Cerri, and Rodrigo C. Barros. Hierarchical multi-label classification networks. In /CML, volume 80 of Proceedings of Machine Learning Research, pages 5225-5234. PMLR, 2018.
[11) Dhruvesh Patel, Pavitra Dangati, Jay- Yoon Lee, Michael Boratko, and Andrew McCallum. Modeling label space interactions in multi-label classification using box embeddings. In International Conference Learning Representations, 2021.
[12] Wei Bi and James T. Kwok. Multilabel classification tree- and dag-structured hierarchies. In ICML., pages 17-24. Omnipress, 2011.
[13] Boli Chen, Xin Huang, Lin Xiao, Zixin Cai, and Liping Jing. Hyperbolic interaction model for hierarchical multi-label classification. AAAI, pages 7496-7503. AAAI Press, 2020.
[14] Jonatas Wehrmann, Ricardo Cerri, and Rodrigo Barros. Hierarchical multi-label classification networks. In International Conference Machine Learning, pages 5075-5084. PMLR, 2018.
[15] Farzaneh Mirzazadeh, Siamak Ravanbakhsh, Nan Ding, and Dale Schuurmans. Embedding inference for structured multilabel prediction. In NIPS, pages 3555 3363,,63,300 2015.
[16] Farzanch Mirzazadeh. Solving association problems with convex co-embedding. 2017.
[17] John Venn. I. on the diagrammatic and mechanical representation of propositions and reasonings. The London, Edinburgh, and Dublin philosophical magazine and journal of science, 10(59):1- 18, 1880.
[18] Jean Van Heijenoort. Set-theoretic semantics. In Studies Logic and the Foundations of Mathematics, volume 87, pages 183-190. Elsevier, 1977.

---

[19] Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. Advances in neural information processing systems, 31, 2018.
[20] Luke Vilnis. Geometric representation learning. Doctoral dissertations. nr. 2146 University of Massachusetts Amberst, pages 1 116-119, 2021.
[21] Menglin Yang. Zhihao Li, Min Zhou, Jiahong Liu, and Irwin King. Hicf: Hyperbolic informative collaborative filtering. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2212-2221, 2022.
[22] Bo Xiong, Shichao Zhu, Mojtaba Nayyeri, Chengjin Xu, Shirui Pan, Chuan Zhou, and Steffen Staab. Ultrahyperoolic knowledge graph embeddings. Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, 2022.
[23] Octavian-Eugen Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic entailment cones for learning hierarchical embeddings. In ICML,, volume 80 of Proceedings of Machine Learning Research, pages 1632-1641. PMLR, 2018.
[24] Ryota Suzuki, Ryusuke Takahama, and Shun Onoda. Hyperbolic disk embeddings for directed acyclic graphs. In JCML, volume 97 of Proceedings of Machine Learning Research, pages 6066-6075. PMLR, 2019.
[25] Ben Andrews and Christopher Hopper. The Ricci flow in Riemannian geometry: a complete proof of the differentiable 1/4-pinching sphere theorem. springer, 2010.
[26] Hyunghoon Cho, Benjamin DeMeo, Jian Peng, and Bonnie Berger. Large- margin classification in hyperbolic space. In The 22nd international conference on artificial intelligence and statistics, pages 1842. PMLR, 2019.
[27] Richard V Kadison. The pythagorean theorem: the finite case. Proceedings of the National Academy of Sciences, 99(7):4178 4184, 2002.
[28] Amanda Clare. Machine learning and data mining for yeast yeast functional genomics. PhD thesis, University of Wales, Aherystwyth, 2003.
[29] Ivica Dimitrovski, Dragi Kocev, Suzana Loskovska, and Saso Dzeroski. Hierarchical annotation of medical images. Pattern Recognit., 44(10-11):245) 2449, 2011.
[30] Ivica Dimitrovski, Dragi Kocev, Suzana Loskovska, and Saso Dzeroski. Hicrarchical classi- fication of diatom images using ensembles of predictive clustering trees. Ecol. Informatics. 7(1):19-29, 2012.
[31] Bryan Klimt and Yiming Yang. The enron corpus: A new dataset for email classification research. In ECML, volume 3201 of Lecture Notes in Computer Science, pages 217-226. Springer, 2004.
[32] Guoyin Wang, Chunyuan Li, Wenlin Wang, Yizhe Zhang, Dinghan Shen, Xinyuan Zhang, Ricardo Henao, and Lawrence Carin. Joint embedding of words and labels for text classification. In ACL (1), pages 2321-2331. Association for Computational Linguistics, 2018.
[33] Ralph Abboud, ismail ilkan Ceylan, Thomas Lukasiewicz, and Tommaso Salvatori. Boxe: box embedding model for knowledge base completion. In NeurlPS, 2020.
[34] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, figh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[35] Gary Bécigneul and Setavian-Fugen Ganea. Riemannian adaptive optimization methods. In ICLR (Poster). OpenReview.net, 2019.
[36] Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian optimization in pytorch. CoRR, abs/2005.028) 2020.
[37] Janez Demsar. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res., 7:1-30, 2006.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Some proofs have been included in the supplementary material only. Some Lemmas which follow directly from definition do not have an explicitly stated proof.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes] All code, data and instructions to reproduce the results attached supplementary material.
(b) Did you specify all the training details (e.g.. g., data splits, hyperparameters, how they were chosen)? [Yes] All training details, hyperparameters are described in the supplementary material.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We run all experiments 10 times and reported the average results. The standard deviations are very small (in range [2 x 10 4,2.3 x 10 10= and hence omitted the concrete numbers.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] The computing resources are described in the supplementary material.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]]
(b) Did you mention the license of the assets? [Yes] We mention the links of the datasets, where the respective licences can found.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "rHUExJsuCRB", "reviewer": "Reviewer_zjFa", "summary": "This paper introduces a new model for performing multi-label classification which is particularly well-suited to situations where there are elements of structure (hierarchy, mutual exclusion) in the label space. The fundamental idea is to use hyperplanes in hyperbolic space as the decision boundaries for label assignment. The main benefit of hyperbolic space over Euclidean space is that there are infinitely many hyperplanes in hyperbolic space which are non-parallel and do not intersect, thus allowing for the regions corresponding to label assignment to capture hierarchical and exclusionary patterns. The previous SOTA model used axis-aligned hyperrectangles, or \"boxes\", and the authors argue that such a constrained geometric region is not in alignment with the natural decision boundaries provided by neural network architectures, whereas using the hyperplanes in hyperbolic space are more amenable to the output of such encoders. A thorough analysis on 12 datasets, along with additional ablation studies are provided in support of these claims.", "questions": "1. On lines 39-40 the authors write that current methods ignore the importance of constraining mutual exclusion, and while they do highlight an important point (that zero implication violation could be attained by assigning the same score to all labels) it should also be noted that other models (in particular, those based on box regions) could also very easily incorporate a similar mutual exclusion loss. For example, MBM could just include a loss which measures the volume of intersection between boxes associated with mutually exclusive labels, thus encouraging these boxes to be disjoint. With this awareness, do the authors feel that it is worth noting that HMI and HMI+HLR columns in Table 1 are making use of this additional information? Alternatively, it might be relatively easy to implement and train such a method for MBM, or to rerun HMI and HMI+HLR with just the implication information to provide two additional columns in Table 1.\n\n2. Patel et al. 2021 also included an evaluation of the models without any constraint injection, as this is the setting which may be most common in practice. Would it be possible to include such an evaluation on the 12 datasets, at least in the Appendix?\n\n3. I did not understand the section on \"Classification via hyperbolic logistic regression\" (lines 188-195). Why is it desirable to consider the distances from an instance to a Poincaré hyperplane of a label as the logit? Wouldn't we want a higher score for points points which are further \"inside\" the n-Ball, rather than on the boundary?\n\n### Suggestions\n1. Line 114-115: The sentence here reads as though a (single) Poincaré hyperplane is the union of a hypersurface $\\partial \\mathbb B^n \\cap \\mathbb D^n$ and all linear subspaces going through the origin, which is not what is meant. Instead, consider saying that \"Poincaré hyperplanes are defined by $\\partial \\mathbb B^n(\\mathbf z) \\cap \\mathbb D^n$ where $\\mathbb B^n(]mathbf z)$ is a ball centered at $z$ whose boundary $\\partial \\mathbb B^n(\\mathbf z)$ intersects the Poincaré ball $\\mathbb D^n$ perpendicularly. In the limit, as $|\\mathbf z| \\to \\infty$, we obtain linear subspaces going through the origin, which are also considered that Poincaré hyperplanes, however we exclude this limiting case from consideration in practice.\"\n2. As suggested in 1, it seems far simpler to parameterize the hyperplanes using the center of the ball rather than the \"center point $\\mathbf c$\". I don't believe you actually need the notion of the center point, and as such you will be able to remove quite a lot of unrelated technical details (Definition 4 - which is good, because the log map hadn't been defined previously - much of the discussion in lines 119-141). Instead of the function $\\mathcal C$ mapping to a center point, we can either think of it mapping to the center of the ball or simply think of it mapping to a Poincaré hyperplane itself, without specifying a particular parameterization. Many additional subscripts can be removed as well.\n3. Lines 130-131: It is mentioned that Euclidean hyperplanes do not naturally allow for modeling set-theoretic semantics, but this is a bit untrue - they certainly can capture set-theoretic relationships in the same way that the hyperbolic space allows for, but they are limited in the complexity of the patterns of such relationships that they can capture. I believe the fact you wish to highlight is that the existence of infinitely many non-parallel hyperplanes in hyperbolic space which do not intersect implies that decision boundaries in hyperbolic space can capture more complicated set-theoretic interactions, such as mutual exclusivity.\n4. Lines 147-148: I could not understand the sentence starting \"In particular\". I think what is meant is that \"In particular, an $n$-ball $\\mathbb B_\\mathbf w$ contains $\\mathbb B_\\mathbf u$ if and only if $\\lVert \\mathbf o_\\mathbf u - \\mathbf o_\\mathbf w \\rVert + r_\\mathbf u < r_\\mathbf w$, and thus we can create an insideness loss by defining...\"\n5. Line 186: \"loss of the embeddings\" is not defined, perhaps what is meant is $\\mathcal L_\\text{inside}$ and $\\mathcal L_\\text{disjoint}$.\n\n### Typos\nLine 26: \"works\" -> \"work\"  \nLine 70: \"Analog\" -> \"Analogous\"  \nLine 186: \"labels and if\" -> \"labels, if\"  \nLine 224: \"follows\" -> \"follow\"  \nLine 238: \"methods\" -> \"method\"  \nLine 313: \"constraints. We\" -> \"constraints we\"", "limitations": "The authors mention that a limitation of the work is that it currently only considers implication and exclusion, and they suggest logical equivalence ($l_a \\wedge l_b \\iff l_c$) as another potential useful constraint to include, however I think that more can be said on this point. The fact that the current model does not include such a constraint is not merely that it has yet to be implemented, there is a representational limitation in that the intersection of two balls is not, generally speaking, another ball, and therefore the existing model does not lend itself to easily encoding this constraint. On the other hand, models which use box regions can very easily incorporate such a constraint, as the intersection of two boxes is also a box.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: The method is novel, and the reasons for this approach are well-motivated.\n\nQuality: The submission is technically sound. The experimental results, while only a minor improvement quantitatively, are statistically significant. One caveat, which will be brought up in the questions, is that it is not necessarily clear that the full benefit comes from the use of hyperbolic hyperplanes, but rather the injection of the mutual exclusion constraints, which other models could also potentially easily exploit. Still, even if the model is only on-par with SOTA results when taking this into account, the underlying idea is sound.\n\nClarity: The submission is mostly clear, but there are a number of improvements to the presentation I would recommend in the Questions section. Even so, I feel I would be able to reproduce the model from the description provided in the paper.\n\nSignificance: The task is of interest to a number of researchers, and more broadly the idea of the incorporation of structure in Hyperbolic space would be of interest to the community of researchers working on such representations. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kJR1SrDkpHj", "reviewer": "Reviewer_v6kK", "summary": "This paper proposes to model hierarchical structure as an embedding inference using Poincare balls. Hierarchical inclusion and exclusion are used to construct training losses in Poincare space and experiments show the proposed model generally outperforms box-based baselines across multiple datasets.", "questions": "1. The theoretical comparison with box representation needs elaboration. For instance, intersection of boxes can represent new concepts since it remains a box shape. But intersections of Poincare plane seems not have this nice attribute. While when modeling clean structures (i.e. membership and nonmembership) Poincare has good potential, but fuzzy hierarchies seems a lack of capability.\n\n2. This paper is not citing the NeurIPS 17 on Poincare representation of hierarchy (Nickel and Kiela). What is new/old in this paper needs to be stressed.", "limitations": "The amount of novelty over the NeurIPS 17 paper is unclear. This paper has a substantial amount of analysis which is a bonus to have. But the technical crux is on the model.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The proposed Poincare hyperplane has some nice attributes on inclusion and exclusion. The construction of the training loss is intuitive and sense-making.\n\n2. Experiments on hierarchical multilabel classification suggest promising results over box-based baselines.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SQGrn1UuxHE", "reviewer": "Reviewer_mG3n", "summary": "This study explores a structured multi-label prediction problem. To this end, the authors propose to convert logical constraints into\nsoft geometric constraints in the hyperbolic embedding space, where the hyperplanes are viewed as convex areas, with insideness and disjointness of these regions representing logical linkages (implication and exclusion). Extensive tests on 12 multi-label classification problems demonstrate the model's capacity to boost performance.", "questions": "Is this method only applicable to the Poincaré disk model, and is it applicable to other hyperbolic models as well?\n\nIs there another relationship in the data set and how to solve it?", "limitations": "As mentioned by the author, other logical constraints can exist in these datasets and the authors do not currently consider these relationships.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: This study presents a novel translation that converts logical constraints into soft geometric constraints in the hyperbolic embedding space. Besides there is clear geometric intution where the implication is modeled by the geometric insideness while the mutual exclusion is modeled by the geometric disjointness.\n\nWeakness: \nThe experiment lacks some case studies that they can faithfully show that the obtained results well reflect the initial motivation.\n\n\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Hm-e2REfBZN", "reviewer": "Reviewer_UtNv", "summary": "This paper introduces a method for multi-label classification when the class labels have known dependency structures, namely implication and mutual exclusion. Hyperbolic geometry is employed to jointly learn parameters for an encoder that embeds points in a Poincare ball and Poincare hyperplanes for class labels. If the embedded point lies on one side (\"inside\") of the Poincare hyperplane, it is predicted to have that class label. Due to the curvature of the space, the Poincare hyperplanes can be contained completely inside one another, overlap somewhat, or non-overlapping. The method presented in this work presents a joint training objective to not only correctly classify the examples, but also encourage known constraints to be satisfied by the learned Poincare hyperplanes for each class: implication <-> complete containment and mutual exclusion <-> non-overlapping. The method is compared on standard benchmark datasets against reasonable baselines.", "questions": "- Are there reasonable probabilistic semantics for Poincare hyperplanes representing probabilities of events? One appealing quality of box embeddings is their natural probabilistic interpretation and closure under intersection, etc. Does your approach have an analogous interpretation?\n- Do the volumes of the \"membership regions\" correlate with the frequency of the respective class label?\n- Is there a way to represent (un)certainty of the given constraints? Say from a not always 100% reliable taxonomy.\n- If you train your model with only a fraction of the available constraints, does the resulting set of Poincare hyperplanes for each class recover any of the constraints that weren't used for training? Furthermore, are there any constraints that the resulting Poincare hyperplanes imply that are incorrect? \n- Are the constraints provided for training always satisfied by the learned Poincare hyperplanes?\n- It seems like the method presented performs much worse when no constraints are provided as compared to MBM. Is there a reason why HMI performs so much worse than MBM when both models are not provided with any constraints? (Context: Table 4 from the submission and Table 1 from Patel et al.)\n- Why is it that the method presented can get away with using substantially fewer dimensions? Is this due to the geometry of the Poincare ball? Also, I'm not sure that the dimensionality claim is valid or sound (See Appendix G of Patel et al. --> This shows similar results to Figure 4b in the submission).\n", "limitations": "The authors have adequately addressed the limitations of their work.\nThe potential negative societal impact of their work is not addressed, but their work has no more potential negative societal impact than any other paper submitted to NeurIPS.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Paper is overall well-written\n- Motivation is clear\n- Figures 1 & 2 make the method very clear and easy to understand!\n- Experimental evaluation is adequate and convincing with several reasonable ablations\n\nWeaknesses:\n- It seems like the method is narrowly applicable. It only seems to beat the best baseline methods when constraints are available. How often are these constraints actually available in practice?\n- As presented, only two types of logical constraints can be (softly) enforced by the objective. These constraints are arguably the most common types of constraints for these problems though.\n- The claim that their method uses less dimensionality than baselines is somewhat misleading (See Appendix G of Patel et al.)\n- The authors don't compare their method to the best performing baseline MBM without explicit constraint modeling.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept