Language Models as Hierarchy Encoders
Authors: Yuan He
Moy Yuan
Jiaoyan Chen
Ian Horrocks
Venue: NeurIPS 2024 poster (2024)
Keywords: Language Models
Transformer Encoders
Hierarchy Encoders
Hyperbolic Embedding
Primary area: natural_language_processing
TL;DR: This work introduces a novel approach to re-train transformer encoder-based language models as explicit hierarchy encoders, leveraging the expansive nature of hyperbolic geometry.

Abstract: Interpreting hierarchical structures latent in language is a key limitation of current language models (LMs). While previous research has implicitly leveraged these hierarchies to enhance LMs, approaches for their explicit encoding are yet to be explored. To address this, we introduce a novel approach to re-train transformer encoder-based LMs as Hierarchy Transformer encoders (HiTs), harnessing the expansive nature of hyperbolic space. Our method situates the output embedding space of pre-trained LMs within a Poincaré ball with a curvature that adapts to the embedding dimension, followed by re-training on hyperbolic clustering and centripetal losses. These losses are designed to effectively cluster related entities (input as texts) and organise them hierarchically. We evaluate HiTs against pre-trained LMs, standard fine-tuned LMs, and several hyperbolic embedding baselines, focusing on their capabilities in simulating transitive inference, predicting subsumptions, and transferring knowledge across hierarchies. The results demonstrate that HiTs consistently outperform all baselines in these tasks, underscoring the effectiveness and transferability of our re-trained hierarchy encoders.

Language Models as Hierarchy Encoders
Yuan He University of Oxford yuan hwwcs.eccc.o..ooo........... ox.. ac. uk
Zhangdie Yuan University Cambridge zy3178cam. ac uk
Jiaoyan Chen
Ian Horrocks
The University of Manchester
University Oxford
j jiaoyan. chen@mancheser. ac.
i an horrocks@cs ox. ac
Abstract
Interpreting hierarchical structures latent in language is key limitation of current language models (LMs). While previous research has implicitly leveraged these hierarchies enhance LMs, approaches for their explicit encoding yet be explored. To address this, we introduce a novel approach to re-train transformer encoder-basee based LMs Hierarchy Transformer encoders (HITs), harnessing the expansive nature of hyperbolic space. Our method situates the output embedding space of pre-trained LMs within Poincaré ball with curvature that adapts the embedding dimension, followed by training on hyperbolic clustering and centripetal losses. These losses designed effectively cluster related entities (input texts) and organise them hierarchically. We evaluate HITs against pre-trained LMs, standard fine-tuned LMs, and several hyperbolic embedding baselines, focusing on their capabilities in simulating transitive inference, predicting subsumptions, and transferring knowledge across hierarchies. The results demonstrate that HiTs consistently outperform all baselines in these tasks, underscoring the effectiveness and transferability of our re-trained hierarchy encoders)
Introduction
In the field of Natural Language Processing (NLP) and related areas, the emergence of transformer- based language models (LMs) such as BERT (encoder-based) [I] GPT (decoder-based) (2], and the more recent large language models (LLMs) like GPT-4 33 and Llama 4 has marked significant progression. Nonetheless, these models face a notable challenge in effectively encoding and interpreting hicrarchical structures latent in language. This limitation has been highlighted by several studies, including those by [5] and (6) which employed prompt-based probes to reveal the limited hierarchical knowledge in pre-trained LMs, and the work by (7). which demonstrated these models' struggles with capturing the transitivity of hierarchical relationships.
Prior research has explored various methods to infuse hierarchical information into LM training. Common approaches include classification-bass fine-tuning using sentence head embedding with classification layer [8] or few-shot prompting with an answer mapping to classification labels [6.
To further pre-train, or re-train LMs on a corpus constructed from hierarchical data, converted
Seee GitHub repository: https://githu: httppp://iittu...c/a/ttttttreree www.://RRROoooooee http:///RReeooottttteeeearraraaa
Datasets
on Zenodo: https: / //huggingface.e.mm co. -
wrgg/oi//10 5281 zenodo. 1051 1042 or the Huggingface Hub: https
and HiT models also the Huggingface Hub.
In this work, the term re-train refers to train LMs on a new corpus without modifying its architecture; is distinguished from standard fine-tuning that involves adding task task-specific layers which lead to additional learnable parameters.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

structural representations into textual formats to align the masked language modeling objective. Others, like [TO] and [T]. have focused on extracting analogous and contrasting examples from hierarchical structures for similarity-basee -basid contrastive learning objective. The aforementioned studies leveraged hierarchical information as implicit signals to augment LMs, yet no existing works specifically targeted the explicit encoding of hierarchies with LMs.
To bridge this gap, we introduce a novel ap- proach to re-train transformer encoder-based LMs as Hierarchy Transformer encoders (HiTs). Inspired by the efficacy of hyperbolic geometry in representing hierarchical structures [12] 13,, we propose the hyperbolic cluster- ing and centripetal losses tailored for LM re- training. As illustrated in [ transformer encoder-based LMs typically use tanh acti- vation function in the last layer, which maps each embedding dimension to the range [-1, 1]. Consequently, the output embeddings of LMs are confined within a unit d-dimensional hyper- cube. Leveraging this characteristic, utilise Poincaré ball of radius vd, whose boundary circumscribes the output embedding space of LMs. The metrics for distance and norm used in our hyperbolic losses are defined w.r.t this specific manifold. After re-training, entities are not only clustered according to their relatedness but also hierarchically organised.
bieberry
berry
respberry
extity Trst
dedkca competer
e-device
phone
liphag
p
Pre nre-trained
Hierarchy Re-trained
Figure I: Illustration of how hierarchies are explic- itly encoded in HiTs. The square (d-dimensional hyper-cube) refers to the output embedding space of transformer encoder-basee LMs whose final ac- tivation function is typically tanh, and the circum- scribed circle (d-dimensional hyper-sphere) refers to the Poincaré ball of radius vd. The distance and norm metrics involved in our hyperbolic losses are defined w.r.t. this manifold.
In evaluating HiTs, we compare their performance against pre-trained LMs, standard fine-tuned LMs, and previous hyperbolic embedding models in the Multi-hop Inference and Mixed-hop Prediction tasks. The Multi-hop Inference task, following the setting in [13], involves training models on all asserted (i.e., one-hop) subsumptions and assessing their ability to infer transitive (i.e., multi-hop) subsumptions. The Mixed-hop Prediction task is designed to mirror real-world scenarios, where models trained on incomplete hierarchy are applied predict unknown subsumption relationships between arbitrary entity pairs. Additionally, we introduce transfer learning setting, where models trained one hierarchy are tested on another. Our experiments utilise datasets derived from WordNet 14] and SNOMED CT 15] and transfer evaluation datasets from Schema.org [16], Food Ontology (FoodOn) [17], and Disease Ontology (DOID) 18] The results show that HIT's significantly surpass all baselines in these tasks, demonstrating their robustness to generalise from asserted to inferred and unseen subsumptions, and promising potential in hierarchy-basee semantic search.
2 Preliminaries
2.1 Language Models
Transformer encoder- brased LMs excel in providing fine-gran fineggriieed contextual word embeddings for enhanced language understanding. A key component these models is the self-attention mechanism, which dynamically assigns importance to different segments of the input text, thereby capturing nuanced contextual semantics more effectively. Notable examples of such models include BERT ① and RoBERTa [19], both of which utilise the masked language modelling objective during pre- training. This approach involves partially masking input sentences and prompting the model to predict the masked tokens, using the unmasked surrounding text context. For acquiring sentence- embeddings, these models can be augmented with an additional pooling layer, applied over the token embeddings (20 21.. Pooling strategies such mean, max, and sentence head pooling are employed, with their effectiveness varying across different applications. A contrastive learning objective is often applied for refining sentence-level semantics [21 22). Despite the rise of generative LLMs,
We ignore the vertices of hyper-cube, as they are on the boundary and thus are undefined the open Poincaré ball.
Results on WordNet are presented in Section 4 while results SNOMED CT are presented in Appendix

---

transformer encoder- based LMs maintain their importance, offering versatility and efficiency in tasks like text classification and semantic search.
2.2 Hyperbolic Geometry
Hyperbolic geometry, form of non-Euclidean geometry, is featured by its constant negative Gaussian curvature, a fundamental aspect that differentiates it from the flat, zero curvature of Euclidean geometry. In hyperbolic space, distances between points increase exponentially the boundary, making it inherently suitable for embedding hierarchical structures. This intuition aligns with the tree embedding theorem based on 8-hyperbolicity, as discussed in 22 and 13
moves towards
Among the various model hyperbolic geometry that are isometrid to each other, the Poincaré ball is chosen for its capacity to contain the the output embedding space of LMs directly, as explained the second last paragraph of Section | The d-dimensional Poincaré ball with negative curvature -e (where 0) is defined by the open ball I = {x E IR ||x||² < =). The distance function in this model, dependent on the curvature value c, is given by:
2 Vc
de v)
tanh - (vol- -u 0r vl)
In this equation, u, Bg ||-|| denotes the Euclidean norm, and @D. denotes the Möbius addition 24 defined as:
Dr V (1 20(u, v) c uu+ (1 c|lu()av
20(u, v) + |
Here, (.))) denotes the inner product in Euclidean space. Note that for flat curvature 0, B will be R and @r will be the Fuclidean addition.
2.3 Hierarchy
We define hierarchy as a directed acyclic graph G(V.E) where V represents vertices that symbolise entilies, and & represents edges that indicate the direct subsumption relationships asserted in the hierarchy. We can then derive indirect subsumption relationships T based on direct ones through transitive reasoning. We borrow the notation from description logic to denote the subsumption relationship e₁ E eg. meaning that e1 a sub-class of ez. Under the closed-wond assumption, we consider an edge (2),
negative sample if (e1,e2) E CUT T. Particularly, a hard negative is identified when é and e2 are also siblings that share the same parent.
Explicit hierarchies can often be derived from structured data sources such as taxonomies, ontologies, and knowledge graphs. taxonomy and the Terminology Box (TBox) of an ontology intrinsically define subsumptions, whereas in knowledge graphs, hierarchical relationships are defined in more customised manner. For instance, WordNet, the hypernym relationship corresponds to subsumption.
Hierarchy Transformer Encoder
We intend to propose a general and effective strategy to re-train transformer encoder-based LMs Hierarchy Transformer encoders (HITs). To deal with arbitrary input lengths of entity names, we employ an architecture similar to sentence transformers (2 incorporating a mean pooling layer over token embeddings to produce sentence embeddings for entities. Note that some of the sentence transformer models have normalisation layer after pooling; we exclude this layer because its presence will constrain the embeddings' Euclidean norms to one, thus hindering hierarchical organisation of entity embeddings. It is also worth mentioning that these changes do not add in learnable parameters besides the ones already in LMs, thus retaining the original architectures of LMs as encoders. As aforementioned, the output embedding space of these LMs is typically
$ FF.g, Poincaré ball model, Poincaré half plane model, and hyperholoid model.
"nn isometry is bijective iistance-preservee transformation.
3

---

d-dimensional hyper-cube because of the tanh activation function in the last layer. Thus, we can construct Poincaré ball of radius vd (or equivalently, curvature value = 1/) whose boundary cireumscribes the hyper-cube] ? Unlike previous hyperbolic embedding methods that utilise the entire hyperbolic space and often require a projection layer to manage out-of-manite method ensures that embeddings are contained within a specific subset of this manifold. Empirical evidence supports that this subset sufficiently accommodates entities in nigh-dimensional space (see Section 4.5p. J. Based on this curvature-adaplee adapted manifold, we propose the following two losses for hierarchy re- te-training.
embeddings, our
Hyperbolic Clustering Loss This loss aims clustering related entities and distancing unrelated ones in the Poincaré ball. We formulate it in the form of triplet loss because related entities are not equivalent but their semantic distances should be smaller than those between unrelated entities. Formally, the loss is defined as:
Leluster =
Σ max(dc(e,et) - do(e, + (e,et.e-)ED
(3)
Here, inputs are presented in the form of triplet (e, et, )_), where et is a parent entity of e, and e is negative parent entity of e; 'D denotes the set of these triplets; de(-.-) refers to the hyperbolic distance function defined in Equation ( and the hyperbolic distance margin. The bold letters denote the embeddings of the corresponding entities.
Hyperbolic Centripetal Loss This loss ensures parent entities are positioned closer to the Poincaré ball's 'ssei origin than their child counterparts, reflecting the natural expansion of hierarchies from the origin to the boundary of the manifold. The term "centripetal" is used to imply that the manifold's origin represents an imaginary root entity for everything. Formally, the hyperbolic centripetal loss is defined as:
Ceentri
Σ (ecet,e))ED
max(Ile* lacl|/eellllle lle ||e/le 3,0)
Again, inputs are the triplets in D, but only the child and parent entities (the positive subsumptions) are used to calculate the loss; |1-Ilc == de 0) refers to the hyperbolic norm; 3 is the hyperbolic norm margin.
The overall hierarchy re-training loss, denoted as CHIT, is the linear combination of these two hyperbolic losses, defined as:
LHIT Celuster + Leentrs
(5)
In Figure 2, we demonstrate the impact of LHIT on entity embeddings. The entity "e-device", " being most general, is nearest to the origin. Sib- ling entities, such as "phone" " and "computer", "laptop" and "pc", are closer to their common parent than to each other, illustrating the effect of re-training cluster related entities while maintain hierarchical relationships.
pull tet push sway
phone
computer
laptop
PC
Figure 2: Illustration of the impact of LHiT during training. In Euclidean space, it seems contradic- tory that both "phone" and "computer" are pulled towards "e-device" but also pushed away from each other. However, in principle this is not a prob- lem in hyperbolic space, where distances increase exponentially relative to Euclidean distances as one moves from the origin to the boundary of the manifold.
As the HIT model functions as encoder, it does not inherently support direct predictions of subsumption relationships. To address this, we devise a probing function that leverages the hierarchy re-trained entity embeddings. This function aims to predict the subsumption rela- tionship for any given pair of entities (e1, e2), incorporating both the clustering and centripetal heuristics:
(6) Here, À 0 represents weighting factor applied to the centripetal heuristic component. The subsumption score is structured to increase as the hyperbolic distance between e and e decreases,
s(ei -dd e2) + A(I|e2ll. lle))
We have also considered scaling down each dimension of the LM embeddings by Vd to confine them within unit Poincaré ball, but found that losses are harder converge in this construction.

---

Table 1: Statisties of WordNet (Noun), Schema.org, and FoodOn, including the numbers of entities (#Entity), direct subsumptions (#DirectSub), indirect subsumptions (#IndirectSub), and the dataset splittings (#Dataset) for Multi-hop Inference and Mixed-ho hop Prediction tasks. Note that the numbers in #Dataset are counts of entity pairs rather than entity triplets.
Source
#Entity
#DirectSub
#IndirectSub
#Dataset )
multi: 834K/323K/922
WordNet
74,401
75,850
587,658
mixed: 75IK/365K/300
Schema.org
903
950
1,978
mixed: -/15K/15K
FoodOn
30,963
36,486
438,266
mixed: 361K/261K/2611
DOID
11,157
11,180
45,383
mixed: 11/K/3IK/3IK
and/or the relative difference in their hyperbolic norms (lle |le |e |Il)) increases. Essentially, for the model to predict e1 E e2, it is expected that e₁ and e₂ are relatively closer in the Poincaré ball. with e positioned further from the manifold's ' origin compared to e2. The value for À and the overall scoring threshold are to be ascertained through hyperparameter tuning on the validation set.
Evaluation
4.1 Task Definition
Multi-hop Inference This task, following the setting in [3] aims evaluate the model's ability in deducing indirect, multi-hop subsumptions T from direct, one-hop subsumptions E, so as to simulate transitive inference. We split T for validation and testing, denoted Toat and Treat- respectively. For each positive subsumption et) involved, we sampled 10 negative parents e for e, leading to 10 training triplets Following the criteria in Section 2.3 (e,eT) ) a valid negative if (e, (). & & T.. We further split the task into two settings: one with random negatives and another with hard negatives, the latter mainly comprising sibling entities. Since not every entity has enough siblings, we supplemented with random negatives that have been sampled to maintain a consistent positive-to-negativ ratio of 10.
This task aims to evaluate the model's capability in determining the existence of subsumption relationships between arbitrary entity pairs, where the entities are not necessarily seen during training. We propose a challenging setting where models trained on incomplete direct subsumptions and examined on a mix of hold-out, unseen direct and indirect (mixed-hop) subsumptions. We split & into training, validation, and test sets, denoted as Eiraiu. Eval, and Exest- respectively. The final training, validataion, and test sets for this task are Etrains Eval U Teat. and Exest Trest respectively, where Trat and Trest are re-used from the previous task. Again, each positive subsumption in these sets is paired with 10 negative samples, either randomly chosen or from sibling entities. Furthermore, an important factor that reflects the model's generalisability is to examine the transfer ability across hierarchies. To this end, we extend the mixed- prediction task with transfer setting where models trained on asserted training edges of one hierarchy are tested on arbitrary entity pairs of another.
Mixed-hov Prediction
Evaluation Metrics For both Multi-hop Inference and Mixed-hop Prediction tasks, we utilise Precision, Recall, and Fl score (abbreviated as F-score in latter discussion) of evaluation. We have opted not to include Accuracy, as preliminary testing indicated a potential bias in this metric, with misleadingly high score resulting from the much larger volume of negative compared to positive samples. important to note that, although the training phase uses entity triplets, the evaluation only involves entity pairs.
primary metrics
4.2 Dataset Construction
We constructed the primary dataset from the noun hierarchy of WordNet 11 due to its comprehensive and structured representation of linguistic hierarchies. To assess the transferability and robustness
10 training triplets constructed from 11 entity pairs.
5

---

across different domains. we additionally constructed datasets from ontologies that represent varied semantic granularities and domains, namely Schema.org [16], Food Ontology (FoodOn) [7]. and Disease Ontology (DOID) [8] We retrieved WordNet from NLTK [25 and adopted pre-processing steps similar to [2], utilising the hypernym relations between noun synsets to construct the hierarchy. For Schema.org, FoodOn, and DOID, our pre-procsssing paralleled that in [6], transforming these ontologies into hierarchies of named entities (details in Appendix AA). To accommodate the textual input requirements of LMs, we constructed entity lexicon using the name attribute in WordNet and the rdfs: label property in the ontologies ?
On WordNet (Noun), FoodOn, and DOID, we adopt consistent splitting ratio for the validation and testing sets. Specifically, we allocate two separate 5% portions of the indirect subsumptions to form Trat and Trest. respectively. Similarly, two distinct 5% portions of the direct subsumptions E are used as Eval and Etest- As Schema.org is significantly smaller than the other hierarchies and only used for transfer evaluation, we split its entire & and T sets into halves for validation and testing, respectively. T presents the extracted hierarchies' statistics and the resulting datasets for the Multi-hop Inference and Mixed-hop Prediction tasks.
In addition our main evaluation, we constructed dataset from the wedely-recognised biomedical ontology SNOMED CT [15] and conducted futher evaluation. The relevant details are presented in
Appendix D
4.3 Baselines
Naive Prior We first introduce a naive baseline (NaivePrior) that utilises the prior probability of positive subsumptions in the training set for prediction. Given that each positive sample is paired with 10 negatives, the prior probability of a positive prediction stands at 1 Consequently, Precision, Recall, and F-score on the test set are all II
i1-
Pre-trained LMs We consider pre-trained LMs as baselines to illustrate their limitations in capturing hierarchical structure semantics. As outlined in Section 3 our focus is on LMs based on the sentence transformer architecture (21). Since these LMs are optimised for cosine similarities between sentences, we devise the following probe for evaluation: for each entity pair (E1,e2), we compute the cosine similarity between the masked reference sentence "e1 is (mask)." and the sample sentence "e1 a é2.". These similarity scores serve as the subsumption scores, with thresholds identified via grid search the validation set. Note that although these LMs originate from masked language models, they cannot be easily probed via mask filling logits or perplexities as in [26] and [27 because their mask filling layers are not preserved in the released versions. We select three top-performing pre-trained LMs from the sentence transformer library of different sizes, including all- LI-MMiLLLM----22 (22.7M), MLLM1-2 (33.4M), and all-mpnet-base-22 (109M).
Fine-tuned LMs Fine-tuned LMs are used baselines to demonstrate that despite the efficacy in various tasks, standard fine-tuning struggles to address this specific challenge. Following the BERTSubs approach outlined in [8) we employ pre-trained LMs with an added linear layer for binary classification, and optimising on the Softmax loss. [ have shown that this method outperforms
various structure-based embeddings such as TransE (28] and DistMult (29), and also surpasses OWL2V (30), which integrates both structural and textual embeddings, in subsumption prediction.
Previous static hyperbolic embedding models are typically evaluated using the Multi-hop Inference task. In our study, we select the Poincaré Embedding (PoincaréEmbed) [12 and the Hyperbolic Entailment Cone (HyperbolicCone) [13 as baselines on this task. However, their lack of inductive prediction capabilities prevents their evaluation on the Mixed-hop Prediction task and its transfer setting. Additionally, we include the hyperbolic GloVe embedding (PoincaréGloVe) as a baseline our transfer evaluation. We select the best pessfoerfonnnn PoincaréGlove Ve (50x2D with an initial trick) pre-trained on a 1.4B token English Wikipedia dump. While PoincaréGloVe supports inductive prediction, its effectiveness is limited by word-level tokenisation, rendering it less effective at handling unknown words. To address this, we employ NaivePrior as a fallback method for entities that involve unknown words and cannot be predicted by PoincaréGlove.
Hyperbolic Baselines
More details of our code implementation and experiment settings are presented in Appendix B
We selected the first name (in English) if multiple names for entity were available.

---

Table 2: Multi-hop Inference and Mixed-hop Prediction test results WordNet.
Random Negatives
Hard Negatives
Model
Precision
Recall
F-score
Precision
Recall
F-score
NaivePrior
0.091
0.091
0.091
0.091
0.091
0.091
Multi-hop Inference (WordNet)
PoincaréEmbed
0.862
0.866
0.864
0.797
0.867
0,830
HyperbolicCone
0.817
0.996
0.898
0.243
0.902
0,383
all-MiniLM--L--22
0.442
0.235
0.132
0.507
0.209
fine-tune
0.800
0,513
0,625
0.764
0.597
0,670
IIT
0.864
0.879
0.871
0.905
0.908
0,907
all-MiniLM-L12--22
0.127
0.585
0.209
0.108
0.740
0.188
fine-tune
0.811
0,515
0.630
0.819
0.530
0.643
HIT
0.880
0.927
0.903
0.910
0.906
0.908
all-mppet-base-v2
0.281
0.428
0.339
0.183
0.359
0.242
fine-tune
0.796
0.501
0.615
0.758
0.628
0.687
HrT
0.897
0.936
0.916
0.886
0.912
0.899
Mixed-hop Prediction (WordNet)
all-MiniLM-L6-v2
0.160
0.438
0.235
0.131
0.504
0.208
fine-tune
0.747
0.575
0.650
0.769
0.578
0.660
HrT
0.835
0.877
0.856
0.882
0.843
0.862
all all-MiniLM-LL2--22
0.127
0.583
0.209
0.111
0.625
0.188
fine-tune
0.794
0.517
0.627
0.859
0.515
0.644
HrT
0.875
0.895
0.885
0.886
0.857
0.871
all al-mmpeet-base-22
0.287
0.439
0.347
0.197
0.344
0.250
fine-tune
0.828
0.536
0.651
0.723
0.622
0.669
HIT
0.892
0.910
0.900
0.869
0.858
0.863
4.4 Results
The effectiveness of our hierarchy re-training approach is evident from the results of both the Multi- hop Inference and Mixed-hop Prediction tasks on WordNet (see Table as well as the Transfer Mixed-hop Prediction task Schema.org FoodOn, and DOID for pre-trained LMs and models trained on WordNet (see Table 3. In the following, we present several pivotal findings based on these results.
Performance of HITs The HIT models, re-trained from LMs of various sizes, consistently out- perform their pre-trained and standard fine-tuned counterparts across evaluation tasks. In the Multi-hop Inference task, HITs exhibit exceptional performance with F-scores ranging from 0.871 to 0.916. This indicates a strong capability generalising from asserted to transitively inferred entity subsumptions. In the Mixed-hop Prediction task, F-scores ranging from 0.856 to 0.900 highlight the effectiveness HITs generalising from asserted to arbitrary entity subsumptions. For the Trans- fer Mixed-hov hop Prediction tasks, selected all- all-MiniLM-LL M--miLL--11-VV M---MnnnLM-L12----1
the pre-trained model because attains comparable performance to all-mppet-basee TTp while is more computationally efficient owing to smaller parameter size. Notably, allMMiniLM-L12-22HHH performs better than pre-trained and fine-tuned all-MiniLM-L12--2 on these transfer tasks by at least 0.150 and 0.101 in F-scores, respectively.
Limited Hierarchical Knowledge in Pre-trained LMs For the tasks on WordNet, all-mpnet-base- v2 achieves the highest F-scores among all the pre-trained models, yet these scores (e.g.. 0.347 and 0.250 on the Mixed-hop hhp Prediction task with random negatives and hard negatives, respectively) are considerably lower compared to their fine-tuned (lagging by 0.304 and 0.419) and hierarchy re-trained (lagging by 0.553 and 0.613) counterparts. This disparity confirms findings from LM probing studies such as those by and 6), demonstrating the limited hierarchical knowledge in pre-trained LMs..

---

Table 3: Transfer Mixed-hop Prediction test results Schema.org, FoodOn, and DOID.
Random Negatives
Hard Negatives
Model
Precision
Recall
F-score
Precision
Recall
F-score
NaivePrior
0.091
0.091
0.091
0.091
0.091
0.091
Transfer Mixed-hop Prediction (WordNet Schema.org)
PoincaréGloVe
0.485
0.403
0,441
0.436
0.415
0.425
all-MiniLM-L12-- 1-2-222
0.312
0,524
0.391
0.248
0.494
0,330
fine-tune
0.391
0.411
0.597
0.248
0,351
IIT
0.503
0.613
0.553
0.408
0.583
0.480
Transfer Mixed-hop Prediction (WordNet FoodOn)
PoincaréGloVe
0.192
0.224
0.207
0.189
0.200
0.195
all-MiniLMM-L1-11
0.135
0.656
0.224
0.099
0.833
0.176
fine-tune
0.407
0.690
0.177
0.282
IIIT
0.690
0.463
0.554
0.741
0.385
0.507
Transfer Mixed-hop Prediction (WordNet - DOID)
PoincaréGloVe
0.265
0.314
0.287
0.283
0.318
0.299
all-MiniLM-LL1--V
0.342
0.451
0,389
0.159
0.455
0.235
fine-tune
0.585
0.621
0.603
0.868
0.179
0.297
HIT
0.696
0.711
0.704
0.810
0.435
0.566
Limited Generalisation in Fine-tuned LMs The research by g illustrates that fine-tuned LMs perform well on single-hop subsumptions. Our observations concur, showing that fine-tuned LMs achieve comparable performance as HiTs when assessed on just single-hop test samples. However, their effectiveness wanes when applied to arbitrary entity subsumptions. For the tasks on WordNet, fine-tuned LMs underperform HrTs by 194 to 0.301 in F-scores. In the transfer task from WordNet to Schema.org, the fine-tuned 1Il-MinniLM-L1---1 model only marginally outperforms its initial state, with an increase of around 0.02 in F-scores across both negative settings.
Performance of Hyperbolic Baselines The Multi-hop Inference task with random negatives follows the evaluation in [13]. In this setup, both PoincaréEmbed and HyperbolicCone significantly outperform the pre-trained and standard fine-tuned LMs, and perform comparably to the HIT models. However, HyperbolicCone exhibits substantially worse performance in the hard negative setting; its low precision and high recall suggest difficulties in differentiating sibling entities that are closely positioned in the embedding space. In the transfer evaluation, PoincaréGloVe shows improved performance over pre-trained and standard fine-tuned models on Schema.or However, it does not demonstrate similar advantage on FoodOn and DOID, primarily due its limited vocabulary, which allows to predict almost all test samples on Schema.org org but substantially fewer on the others.
Comparison of Random and Hard Negatives For the tasks on WordNet, hard negative settings present greater challenges compared random negative settings for all pre-traines LMs. This increased difficulty, however, is not as pronounced in fine-tuned LMs and HiTs. A plausible explanation that while hard negatives pose challenges, they simultaneously act as high-quality adversarial examples, potentially leading to more robust training outcomes. In the Transfer Mixed Prediction task, hard negative settings are generally more challenging than random negative settings. For instance, in the WordNet-to-DOII transfer task, both fine-tuned and hierarchy re- eratrineed all- MiniLM-L12-v2 models exhibit significantly higher F-scores the random negative setting, with differences of 0.306 and 0.8 respectively, compared to the hard negative setting.
Case Analysis WordNet-to-DOnn Transfer In the WordNet-to-Dor tOOD transfer task, the disparity in the "disease" " category is notable: WordNet contains only 605 entities that are descendants of "disease". compared to 10K in DOID. Despite this significant difference, HIT models effectively transfer knowledge, achieving F-scores of 0.704 (random negatives) and 0.566 (hard negatives).
More discussion loss functions and an ablation study of loss margins are presented in Appendix

---

4.5 Analysis of HIT Embeddings
Distribution Figure illustrates how WordNet entity embeddings generated by all-MiniLM-L12- v2+HIT distribute w.r.t. their hyperbolic norms.
These norms effectively capture the natural pansion of the hierarchical structure, evidenced by an exponential rise in the number of child entities. A notable observation is the sharp de- cline in the number of entities when hyperbolic norms exceed 23, suggesting that few entities reside these higher levels. Additionally, the range of entity hyperbolic norms, approximately from 8 to 24, indicates that relatively small re- gion of the high-dimensional manifold suffices to accommodate all entities in WordNet.
10000
B000
6000
4000
2000
Hyperbolic
Figure 3: Distribution of WordNet entity embed- dings generated by HIT w.r.t their hyperbolic norms.
Correlation In Table 4 we compare the Pear- son correlation coefficients across different hy- perbolic models to measure the linear relation- ship between entities' hyperbolic norms and their depths in WordNet. Our analysis shows that all hyperbolic models lead to positive cor- relation between norms and depths, as expected. However, HIT demonstrates stronger correla- tion than both PoincaréEmbed and Hyperbol- icCone.
Table 4: Statistical correlations between WordNet entities' depths and their hyperbolic norms across different hyperbolic models.
HIT
PoincaréEmbed HyperbolicCone
0.346
0.130
0.245
Table 5: Hyperbolic distances between the em- beddings of selected entities ( ("computer", "pc", "fruit", "berry"), along with their individual hyper- bolic norms (h-norm) and depths in WordNet.
Case Study In Table 5 we showcase the effec- tiveness of HIT using selected entities: "com- puter". "pe" "fruit" , and "berry". The table presents the hyperbolic distances between these entities' embeddings, their individual hyperbolic norms, and their depth? in the WordNet hier- archy. We can observe that: (i) closely related entities, such as "fruit" and "berry", are signifi- cantly nearer to each other compared to more distant pairs; (ii) more specific entities like "pc" and "berry" are positioned further from the origin of the manifold than their ancestor entities; (iii the disparity in hyperbolic norms between "pc" and "computer" is greater compared to that between "fruit" and "berry,, reflecting the hierarchical depth where "pe" is grandchild of "computer", while "berry" is a direct child "fruit".
computer
pc
berry
computer
24.9
pe fruit
5.9
0.0
25.2
27.2
22.5
25.2
0.0
6,72
berry
24.9
27.2
6.72
0.0
h-norm
depth
5 Related Work
Prompt-based probing is widely used for extracting knowledge from LMs. Studies like [5] utilised cloze-style prompts for hypernym detection, while (6) approached subsumption prediction similar to Natural Language Inference. 7 examined if LMs. when correctly predicting "A is a B" and "B is C", can consistently infer the transitive relationship "A is a C". These studies collectively highlight the limited capacity of pre-trained LMs in understanding hierarchical structures. Other research efforts, such as those by [10] and [T. have aimed to incorporate structural semantics into LMs for entity encoding. However, these largely focus on entity equivalence or similarity, with less emphasis on hierarchical organisation.
Regarding hyperbolic embeddings, methods like the Poincaré embedding [12 and the hyperbolic entailment cone [13] have effectively represented hierarchical structures. Despite their efficacy, these techniques are inherently static, constrained by a fixed vocabulary of entities, and do not support
10hhe full name "personal computer" used for embedding. DDepth of an entity is the minimum number of hops to the root node. For hierarchies that do not have root node, we set up imaginary root node when calculating the depth.

---

inductive predictions about unseen data. Further explorations include learning word embeddings in hyperbolic space [T 32). These methods, however, are limited word-level tokenisation and yield non-contextual word representations. These shortcomings can be mitigated by integrating hyperbolic embeddings with transformer-basel LMs.. [33 has explored this direction, applying learnable layers to project LM embeddings into hyperbolic space for syntax parsing and sentiment analysis. Our approach diverges from theirs by focusing on training LMs as general hierarchy encoders without the need for additional learnable parameters.
6 Conclusion
This paper tackles the challenge of enabling language models to interpret and encode hierarchies. We devise the hierarchy re- re-training approach that involves joint optimisation both the hyperbolic clustering and hyperbolic centripetal losses, aiming to cluster and organise entities according to their hierarchical relationships. The resulting HIT models demonstrate proficiency in simulating transitive inference and predicting subsumptions within and across hierarchies. Additionally, our analysis of HiT embeddings highlights their geometric interpretability, further validating the effectiveness of our approach.
7 Limitations and Future Work
This work does not address the potential loss pre-trained language understanding resulted from hierarchy re-training. Also, the issue of entity naming ambiguity inherent in the dataset sources is not tackled, which could introduce noise into the training process.
For future work, several promising directions can be pursued: (i) investigating methods to measure and mitigate catastrophic forgetting, (ii) training a HIT model across multiple hierarchies, either for general or domain-specific applications, (iii) extending HIT to accommodate multiple hierarchical relationships within single model, and (iv) developing hierarchy- -baeed semantic search that contrasts with traditional similarity-basee approaches.
Acknowledgments and Disclosure of Funding
This work was supported by Samsung Research UK (SRUK), and EPSRC projects OASIS (EP/S032347/1), UK FIRES (EP/S011111/1), and ConCur (EP/V050869/1). Special thanks to Zifeng Ding for his valuable feedback during the rebuttal process.
References
[1] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep hidirectional transformers for language understanding. In Proceedings of naacI-HI.T, volume 1, page 2, 2019.
[2] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. OpenAI, 2018
[3] OpenAI. Gpt-4 technical report. ArXiv, abs/2303.087744 2023.
[4] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint urXiv:2307.0990 09988888. 2023.
[5] Michael Hanna and David Mareček. Analyzing bert's knowledge of hypernymy via prompting. In Proceedings of the Fourth BlackboxNi.P Workshop Analyzing and Interpreting Neural Networks for NLP, pages 275-282, 2021.
[6] Yuan He, Jiaoyan Chen, Ernesto Jimenez-Rui Ruiz, Hang Dong, and Ian Horrocks. Language model analysis for ontology subsumption inference. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACI 2023, pages 1439-3453 Toronto, Canada, July 2023. Association for Computational Linguistics.
10

---

[7] Ruixi Lin and Hwee Tou Ng. Does bert know that the is-a relation is transitive? In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers), pages 94-99, 2022.
[8] Jiaoyan Chen, Yuan He, Yuxia Geng, Ernesto Jiménez-Ruiz, Hang Dong, and Ian Horrocks. Contextual semantic embeddings for ontology subsumption prediction. World Wide Web, pages 1-23, 2023.
[9] Hao Liu, Yehoshua Perl, and James Geller. Concept placement using bert trained by trans- forming and summarizing biomedical ontology structure. Journal of Biomedical Informatics. 112:103607, 2020.
[10] Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier. Self- alignment pretraining for biomedical entity representations. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4228 4238, 2021.
[11] Francis Gosselin and Amal Zouaq. Sorbet: A siamese network for ontology embeddings using distance-based regression loss and bert. In International Semanfic Web Conference, pages 561-578. Springer, 2023.
[12] Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical represen- tations. Advances in neural information processing systems, 30, 2017.
[13] Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic entailment cones for learning hierarchical embeddings. In International Conference on Machine Learning, pages (646-1655, PMLR, 2018.
[14] George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11):39944) 41, 1995.
[15] Michael Q Stearns, Colin Price, Kent A Spackman, and Amy Y Wang. Snomed clinical terms: overview of the development process and project status. In Proceedings of the AMIA Symposium, page 662. American Medical Informatics Association, 2001.
[16] Ramanathan V Guha, Dan Brickley, and Steve Macbeth. Schema. org: evolution of structured data on the web. Communications of the ACM, 59(2):44-51, 2016.
[17] Damion M Dooley, Emma J Griffiths, Gurinder S Gosal, Pier L Buttigieg, Robert Hochndorf, Matthew C Lange, Lynn M Schriml, Fiona SL Brinkman, and William WL Hsiao. Foodon: harmonized food ontology to increase global food traceability, quality control and data integration. npj Science of Food, 2(1):23, 2018
[18] Lynn Marie Schriml, Cesar Arze, Suvarna Nadendla, Yu-Wei Wayne Chang, Mark Mazaitis, Victor Felix, Gang Feng, and Warren Alden Kibbe. Disease ontology: backbone for disease semantic integration. Nucleic acids research, 40(D1):D900-D946 2012.
[19] Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 19Xiv:1907.11692. 2019.
[20] Han Xiao.
http:/gtwww..cca//aaaaaa//arrrte service 2018.
[21] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNEP), pages 3982-3992 3992, 2019.
[22] Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 6894-6910, 2021.
[23] M. Gromov. Hyperbolic groups In Essays in group theory, pages 75-263. Springer, 1987.
11

---

[24] Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. Advances in neural information processing systems, 31, 2018.
[25] Edward Loper and Steven Bird. Nitk: The natural language toolkit. Proceedings of the ACL-
02 Workshop Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, pages 63 30-70, 2002.
[26] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-ICCNLP), pages 2463-2473, 2019.
[27] Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699-277 27122 2020.
[28] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26, 2013.
[29] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:14/2.6575, 2014
[30] Jiaoyan Chen, Pan Hu, Ernesto Jimenez-Ruiz, Ole Magnus Holter, Denvar Antonyrajah, and Ian Horrocks. Owl2vec*: Embedding of owl ontologies. Machine Learning. 110(7):1813-1845, 2021.
[31] Bhuwan Dhingra, Christopher Shallue, Mohammad Norouzi, Andrew Dai, and George Dahl. Embedding text in hyperbolic spaces. In Proceedings of the Twelfth Workshop on Graph-Based Methods for Natural Language Processing (TexIGraphs-12) 12), pages 59-69, 2018
[32] Alexandru Tifrea, Gary Becigneul, and Octavian- Eugann Ganea. Poincare glove: Hyperbolic word embeddings. In International Conference Learning Representations, 2018
[33] Boli Chen, Yao Fu, Guangwei Xu, Pengjun Xie, Chuanqi Tan, Mosha Chen, and Liping Jing. Probing bert in hyperbolic spaces. In International Conference on Learning Representations, 2020.
[34] Yuan He, Jiaoyan Chen, Hang Dong, Ian Horrocks, Carlo Allocca, Tachun Kim, and Brah- mananda Sapkota. Deeponto: A python package for ontology engineering with deep learning. arXiv preprint arXiv:2307.03067, 2023.
[35] Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian optimization in pytorch. arXiv preprint arXiv:2005.02819 2020.
[36] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue. Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 conference empirical methods in natural language processing: system demonstrations, pages 38 38-45, 2020.
[37] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018
[38] Gary Becigneul and Octavian- -ugeen Ganea. Riemannian adaptive optimization methods. In International Conference on Learning Representations, 2018.
12

---

A Hierarchy Construction from Ontologies
The Terminology Box (TBox) in an OWL (Web Ontology Language ontology defines entity relationships using subsumption axioms (C D) and equivalence axioms (C D), where C' and D are atomic or complex concept expressions defined in the description logic SROIQ. In this work. used atomic concepts nodes of the hierarchy. We employed an ontology reasoner to deduce direct subsumptions and included these as edges in the hierarchy. This approach fully considers both subsumption and equivalence axioms for all potential edges. Considering an atomic concept Beef, which is defined by Beef = Meat IT EderivesFrom. Cattle, as an example, the reasoning will lead to edge between Beef and Meat. Without reasoning, Beef will be misplaced under the root node, if no other subsumptions about Beef are asserted in the ontology. Tools like Protége I4 also demonstrate similar considerations in presenting ontology concept taxonomies.
also important to note the variability in naming schemes across different ontologics, which sometimes necessitates pre- pre-processing of entity names. [6] provided detailed pre-processing steps for Schema.org, FoodOn, and DOID. In this study, we used the pre-processed versions of FoodOn and DOID and applied the same pre-processing methodology to the latest version of Schema.org.
B Experiment Settings
The code implementation of this work primarily depends on DeepOnto 14 for processing hierar- chies and constructing datasets, Geoopt (35 for Poincaré ball, Sentence- Huggingfase ace -rrans-rrf ormers [36] for training and evaluation of LMs. All our experiments were conducted on a single Quadro RTX 8000 GPU.
Ceaeteenn--Trasttee Transformers (1 and
In the hierarchy re-training of our HIT models, we configured the hyperbolic clustering loss margin (a in Equation 3) at 5.0 and the hyperbolic centripetal loss margin (6 Equation 4 0.1. An exception was made for all-mmpnet-base-v2 with hard negatives, where was adjusted to 3.0, based on validation. The models were trained for 20 epochs, with training batch size of 256, 500 warm-up steps and an initial learning rate of 10 using the AdamW optimiser [37]. Model selection was conducted after each epoch, guided by performance on the validation set.
For standard fine-tuning, we largely adhered to the default settings of the Huggingface Traine maintained the same training batch size used the hierarchy re-training. Notably, our preliminary lesting revealed that fine-tuning is more prone to overfitting. Consequently, we adopted a more frequent model selection interval, performing this assessment every 500 training steps rather than epoch-wise.
B5 but
For our hyperbolic baselines, trained PoincaréEmbed with embedding dimension of 200 for 200 epochs, a training batch size of 256, 10 warm-up epochs, and constant learning rate of 0.01, using the Riemannian Adam optimiser (38 According 13]. HyperbolicCone benefits from a robust initialisation such as the one provided by a pre-trained PoincaréEmbed. Consequently, utilised the same hyperparameters the weights from our pre-trained PoincaréEmbed. As outlined in the main paper, we selected the optimal pre-trained PoincaréGloVe model reported by (32) (50x2D with initial trick) as baseline for our transfer evaluation.
train HyperbolicCone except that initialised it with
C Further Discussion on LHIT
Loss Variants As discussed in the main paper, we opted for the triplet contrastive loss format for hierarchy re-training because hierarchically related entities (e.g., subsumptions) should be closer together, yet not equivalent. We tested other forms of loss, including standard contrastive loss, which minimises absolute hyperbolic distances between related entities, and softmax contrastive loss, which
Hnttps://www. ww. org/ /ww/
"Refer to the defimition of DirectSubClassOf at hwter//cihssiiiarrr//////eeeessa
/eaarrttiee///ogn////////eeeeeae html
htttpe//pooteg.sssaffootdddt
13

---

was adopted in training PoincareEmbed [1]]. Our trial experiments demonstrated that the triplet form converges more efficiently and effectively compared to these alternatives.
Loss Margins We provide an ablation study on loss margins a and / defined in Equation (3) and Equation 4 respectively.
Table Ablation results (F-score) of allMiniLM-L12--2-HH on WordNet's Mixed-hop Prediction.
5.0, 0.1
3.0, 3 0.1
1.0, = 0.1
5.0, 3 0.5
0.885
0.865
0.867
0.899
The results from Table 6 indicate that although loss margins impact performance, the HIT model exhibits robustness their variations. Notably, higher F-score for a 5.0, 3 0.5 is observed, surpassing the results presented in the main paper. Despite this, we chose not to overly optimise a and B to avoid overfitting and to maintain the generalisability of our findings.
D Results on SNOMED CT
In the main paper, we primarily focused on models trained the WordNet (Noun). This section broadens our study to encompass models trained on SNOMED CT [3], a structured, comprehensive, and widely-used vocabulary for electronic health records. Using the tool provided by the SNOMED CT team To we converted the latest version of SNOMED CT (released in December 2023) into the OWL ontology format. We then constructed the hierarchy according to the procedure detailed in Appendix A
For entity name pre-processing in SNOMED CT, we addressed potential information leakage during testing. Typically, an SNOMED CT entity is named in the format of " (entity name) ( (branch name)) )", e.g., "virus (organism)" " The branch name denotes the top ancestor and is propagated through all its descendants. To prevent this information from biasing the model, we removed the branch name from each entity.
Table 7: Statistics of SNOMED-CT, including the numbers of entities (#Entity), direct subsumptions (#DirectSub), indirect subsumptions (#IndirectSub), and the dataset splittings (#Dataset) for Multi- hop Inference and Mixed-hop Prediction tasks.
Source
#Entity
#DirectSub
#IndirectSub
#Dataset (Train/Val/Test)
SNOMED
364,352
420,193
2,775,696
mixed: 4,160K /00KK/1,75K, 1,758K/1,7558
In Table 7 we present relevant statistics of the SNOMED CT hierarchy and its corresponding dataset, which was constructed using the method outlined in Section 4.2
details the Mixed-hop Prediction results on SNOMED CT. along with the Transfer Mixed-hoo Prediction results on Schema.org, FoodOn, and DOID for pre-trained LMs and models trained on SNOMED CT. Building on the demonstrated effectiveness of HIT in simulating transitive inference from the main body of this paper, we present only the results for inductive subsumption prediction. The transfer evaluation incorporates the same set of hierarchies as those used in the evaluation on WordNet (Noun), facilitating a meaningful comparison of model performance across different training hierarchies.
In the Mixed-hop Prediction task on SNOMED CT. all pooeels-pre-trained, fine-tuned, and hierarchy re-trained-oud outterroo their counterparts on WordNet (Noun) in terms of F-scores. This improved performance likely to be attributed SNOMED CT's more precise entity naming and better organised concept hierarchy, which reduces both textual and structural ambiguity.
In the transfer results, models trained SNOMED CT exhibit notably better F-scores DOID, which aligns with expectations given DOID's ' focus on diseases and SNOMED CT's comprehensive biomedical scope. Conversely, their performance on Schema.org, common-sense ontology, is comparatively worse. Notably, the fine-tuned all-MiniLM-L12--22 performs even worse than its
14

---

Table Mixed-hop Prediction test results on SNOMED and Transfer Mixed-hop Prediction results on Schema.org, FoodOn, and DOID.
Random Negatives
Hard Negatives
Model
Precision
Recall
F-score
Precision
Recall
F-score
MajorityPrior
0.091
0.091
0.091
0.091
0.091
0.091
Mixed-hop Prediction (SNOMED)
all M---LL------222
0.224
0.443
0.297
0.145
0.398
0.213
fine-tune
0.919
0.859
0.888
0.894
0.635
0.743
HIT
0.941
0.967
0.954
0.905
0.894
0.899
Transfer Mixed-hop Prediction (SNOMED - Schema.org)
all Min2LM-L12-v2
0.312
0.524
0.391
0.248
0.494
0.330
tine-tune
0.198
0.864
0.322
0.431
0.288
0.345
HIT
0,432
0.580
0.495
0.274
0.608
0.378
Transfer Mixed-hop Prediction (SNOMED FoodOn)
Al--MiniLM-L1--v2
0.135
0.656
0.224
0.099
0.833
0.176
fine-tune
0.378
0.540
0.445
0.638
0.371
0.469
0.700
0.500
0.583
0.594
0.442
0.506
Transfer Mixed-hop Prediction (SNOMED - DOID)
all-MiniLM-L12-v2
0.342
0.451
0.389
0.159
0.455
0.235
fine-tune
0.547
0.912
0.684
0.831
0.795
0.812
0,836
0.864
0.850
0.739
0.748
0.744
pre-trained version on Schema.org. suggesting an overfitting to biomedical domain knowledge in SNOMED CT. In contrast, HIT demonstrates greater robustness against such overfitting.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the abstract and introduction, we mentioned that our research addresses the gap of explicit hierarchical understanding current transformer encoder-based language models (LMs). We introduce hierarchy re-training approach utilising a survature-adaplee Poincaré ball, enabling these models as hierarchy encoders that can generalise from asserted subsumptions to inferred and unseen ones with notable performance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed two limitations at the end of the main paper.
Guidelines:
The answer NA means that paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This work is based on the theoretical guarantee of embedding tree-like struc- tures hyperbolic space, which has been proven in other works cited the main paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In the main paper, we have detailed our methodology in Section and the baseline models Section 4 Also, we have listed all necessary experiment settings in Appendix B We believe that these details are sufficient for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
17

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
See footnote I on the first page.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines lattps://nips.co
Sttiiide//Codeeubbiissses ssi
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttpccccpppppuuu..../ee.e Ctteeee//ooSSSSeesS..S...
wwreiisssoopoooioooooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We followed the evaluation framework of existing literature in hyperbolic embedding.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix B we have mentioned what GPUs we were using for our experi- ments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp////eerippp.cpppppbtttttaa
cc
Answer: [Yes]
Justification: This work only utilises fully open-source datasets and models, ensuring there are no ethical, privacy, or copyright concerns associated with their use.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have included citations of existing assets where appropriate.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. can/dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification:
The use of the dataset is clearly presented in the main paper. We have included
an anonymised link to our dataset and the relevant code included in the supplimentary material.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "wqXkJCOxjx", "reviewer": "Reviewer_fC9f", "summary": "This paper presents a novel approach called Hierarchy Transformer encoders (HITs) to retrain transformer-based language models to better encode hierarchical structures in language. The method involves situating the output embedding space of pre-trained language models within a Poincaré ball and training on hyperbolic clustering and centripetal losses. The authors evaluate HITs against pre-trained LMs, fine-tuned LMs, and hyperbolic embedding baselines on tasks including multi-hop inference and mixed-hop prediction. Results show that HITs consistently outperform baselines, demonstrating improved ability to capture hierarchical relationships and generalize across hierarchies.", "questions": "Could you show more examples on how hierarchies are encoded through the proposed approach?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a novel method to explicitly encode hierarchies in language models without adding new parameters.\n\n2. The authors conduct extensive experiments across multiple tasks and datasets, and show the effectiveness of their approach.", "weaknesses": "1. The proposed method requires predefined hierarchies which may limit its generalization, especially when new entity arrives while no relation is presented for the new entity.\n\n2. Applications of the learnt entity embedding are desired to show the effectiveness of the embedding. For example, how can the embedding be used in downstream tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback. We address your comments and questions below:\n\n--------\n\n**Regarding Weakness 1**:  \n\nOur HiT models **can deal with new entities** and the capability of predicting subsumptions between arbitrary entity pairs is one of the key highlights. Our Mixed-hop prediction task specifically examines a model's ability to make **inductive predictions involving unseen, new entities**. During evaluation, we exclude 10% of the subsumptions, which can include these unseen entities, to test this capability. Additionally, we conduct transfer evaluations across three different hierarchies. In all these settings, the HiT models significantly surpass the baselines, demonstrating their ability to generalise and handle new entities effectively.\n\n--------\n\n**Regarding Weakness 2**: \n\nThe primary focus of this work is to explore **explicit hierarchy encoding** within **transformer encoder-based language models**. Our evaluation tasks are tailored for this purpose and align with evaluation in prior works like [1] and [2], which aim to construct **geometrically interpretable** hierarchy embeddings, while extending to real-world scenarios where entities and relationships can be new and unseen. Predicting subsumptions and other hierarchical relationships between arbitrary entities is a crucial task for completing missing knowledge or enriching new knowledge in taxonomies and hierarchies, as well as querying hierarhical contexts from them. We will explore more specific use-cases and downstream tasks of HiTs in future work.\n\n--------\n\n**Regarding Question 1**: \n\nBelow shows two example hierarchical paths in WordNet:\n\n```\nperson -> actor -> comedian\nfluid -> liquid -> water\n```\n\nand their embeddings have the following statistics:\n\n|          |   person |   actor |   comedian |   fluid |   liquid |   water |\n|:---------|---------:|--------:|-----------:|--------:|---------:|--------:|\n| **person**   |      0   |     5.3 |       12   |    15   |     15.5 |    20   |\n| **actor**    |      5.3 |     0   |       10.9 |    18.2 |     18.7 |    22.9 |\n| **comedian** |     12   |    10.9 |        0   |    21.5 |     21.3 |    25.5 |\n| **fluid**    |     15   |    18.2 |       21.5 |     0   |      5.1 |    11.1 |\n| **liquid**   |     15.5 |    18.7 |       21.3 |     5.1 |      0   |     9   |\n| **water**    |     20   |    22.9 |       25.5 |    11.1 |      9   |     0   |\n| *h-norms*  |     13.6 |    15.5 |       19.2 |    17.3 |     17.4 |    19.8 |\n> Table: Hyperbolic distances between the embeddings of entities in above examples, along with their individual hyperbolic norms.\n\nWe can see that the hyperbolic norms of entity embeddings generally follow the hierarchical paths and related entities are closer than non-related ones.\n\n--------\n\n- [1] Nickel, Maximillian, and Douwe Kiela. \"Poincaré embeddings for learning hierarchical representations.\" NeurIPS (2017).\n- [2] Ganea, Octavian, Gary Bécigneul, and Thomas Hofmann. \"Hyperbolic entailment cones for learning hierarchical embeddings.\" ICML (2018).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JEYdBYR6lg", "reviewer": "Reviewer_9j2x", "summary": "This paper proposes a new way to retrain encoder-based language models into hierarchy encoders. Specifically, they propose to recast the output embedding space onto a Poincaré ball and retraining with the designed loss functions for organizing entities into hierarchy. The experiments on real world datasets like WordNet demonstrates the advantages of the proposed approach.", "questions": "- see. weakness", "limitations": "The authors discuss their limitation in the sec 6.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written, scoped, and nicely organized.\n- The topic of addressing hierarchy with language models is interesting.\n- The proposed approach is novel and the corresponding experimental results validate the effectiveness of the proposed method.", "weaknesses": "- Can you explain if the proposed method can be used to decoder-only models?\n- Will frequency of entities in the data impact the hierarchy captured in the representations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback. We address your comments and questions below:\n\n-------\n\n**Regarding Weakness 1**: \n\nOur current method is specifically designed for encoder-based language models due to their ability to produce embeddings with **more straightforward semantic meanings**. Decoder-based models, which generate tokens based on previously generated ones, present a **challenge in directly editing their embeddings to assign geometric interpretability**. Retrieving meaningful semantic embeddings from decoder-based LLMs is itself a challenging research question. This is an area we recognise as valuable for future research. Recent works like [1] attempt to enable typical encoder training on decoder-based LLMs by modifying attention mechanisms and specifically masking the next token. We believe our approach **can be adapted to the decoder-based framework when a more systematic way of embedding extraction from decoder-only models is developed**. Despite this, our encoder-based models can still support decoder-based models by providing hierarchy-aware context, which can enhance the generation process.\n\n-------\n\n**Regarding Weakness 2**: \n\nFrequent entities receive more exposure during training, resulting in more detailed and fine-grained hierarchical representations. This can lead to a similar challenge faced in many machine learning works: handling long-tail entities that appear less frequently. However, as our models are re-trained from pre-trained LMs, the distribution of entities in the pre-training data also has an impact. Addressing this aspect is beyond the current scope of our work.\n\n-------\n\n- [1] BehnamGhader, Parishad, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. \"LLM2vec: Large language models are secretly powerful text encoders.\" arXiv (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZboTVlyvEF", "reviewer": "Reviewer_c8m8", "summary": "The paper introduces a method to re-train transformer-based language models as Hierarchy Transformer encoders (HITs), using the properties of hyperbolic space to enhance their ability to encode hierarchical structures in language.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The utilization of hyperbolic space to encode hierarchical structures in language models is a creative and theoretically sound approach, as hyperbolic space naturally lends itself to representing hierarchies.\n2. The paper provides extensive experimental results, showing that HITs consistently outperform traditional pre-trained and fine-tuned models across multiple datasets and tasks.\n3. The methodology, experimental setup, and results are clear, making the paper accessible to readers.", "weaknesses": "1. The motivation is a bit overstated since it is heavily on the claim that current language models are significantly deficient in encoding hierarchical information. Thus, improvements can certainly be made.\n2. The application of hyperbolic spaces in language models, while interesting, isn't entirely quite inspiring as previous works have explored similar ideas. The distinction between this and prior approaches isn't as significant as suggested.\n3. The downstream tasks selected by this study have deviated from those for the LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback. We address your comments and questions below:\n\n------\n\n**Regarding Weakness 1**: \n\nIn Introduction, we acknowledge that hierarchical information has been considered in existing language model studies. Our claim emphasises the lack of **explicit geometric interpretability in hierarchical encoding**. For instance, as highlighted in [1], pre-trained language models often fail to capture the transitivity property inherent in hierarchical relationships. Our experiments demonstrate that standard fine-tuning, despite its strengths, cannot effectively capture this transitivity (and does not have suffcient interpretability) compared to our proposed HiT method. We will revise our contribution statement to make it clearer.\n\n------\n\n**Regarding Weakness 2**: \n\nThe major differences of our work compared to existing related works are discussed in Sections 1 & 5. In particular, we mentioned that: \n\n- Previous works on pre-trained language models considering hierarchies did not focus on explicit geometric interpretability (line 338); while our work seeks to construct a language model-based hierarchy encoder with such interpretability.\n- Previous hyperbolic embeddings did not support inductive prediction (line 343) or did not handle unseen entities effectively (line 344-345); while our models can naturally support inductive prediction on unseen data.\n- [2] investigated adding a downstream hyperbolic layer to pre-trained LMs for classification tasks, whereas our work focuses on explicitly encoding hierarchies without requiring additional trainable parameters (line 346-349). \n\n------\n\n**Regarding Weakness 3**: \n\nOur evaluation tasks are designed to assess whether **transformer encoder-based language models** can be trained to capture hierarchical structures by their ability to generalise from asserted subsumptions to inferred and unseen subsumptions. Our mixed-hop prediction task, which involves predicting missing subsumptions between arbitrary entities (potentially unseen), is a critical knowledge-intensive task for completing or enriching knowledge in hierarchies and taxonomies. \n\n------\n\n- [1] Ruixi Lin and Hwee Tou Ng. “Does bert know that the is-a relation is transitive?” ACL (2022).\n- [2] Boli Chen, Yao Fu, Guangwei Xu, Pengjun Xie, Chuanqi Tan, Mosha Chen, and Liping Jing. “Probing bert in hyperbolic spaces.” ICLR (2020).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RC6dXrkEw1", "reviewer": "Reviewer_DpLC", "summary": "This paper proposes a method that leverages a non-Euclidean representation space to better encode hierarchical relationships between entities. Specifically, a hyperbolic embedding space is defined, wherein items closer to the origin are higher-level concepts, and items further from the origin are lower-level concepts; items closer to each other typically have closer semantic relationships, such that it is then straightforward to translate between geometric distances and inheritance relationships between concepts—even when the relationship was not explicitly encoded in the training set.\n\nIn a series of experiments, the authors leverage sentence-Transformer models as bases to predict whether two entities have taxonomic relationships in a zero-shot manner. Comparisons are performed between the base sentence-Transformer models, the same models with task-specific fine-tuning, and the proposed hyperbolic encoding method. It is observed that the proposed method outperforms more naive methods.", "questions": "1. Why sample to a consistent 1:10 ratio of positive:negative examples? Was this decision made empirically?\n2. How long did it take to run your proposed method compared to standard fine-tuning? Is there a better trade-off between performance and compute?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed embedding space works well as a way to categorize hierarchical relationships between inputs. It also makes it easy to tell what the relationship is between two concepts in latent space, as distances between concepts and distances to the origin both have well-defined meanings in this space.\n* The method can be run with modest compute resources, making it accessible and easily scalable as sentence-Transformer models scale and improve.\n* The idea is interesting and, to my knowledge, novel.", "weaknesses": "1. As the authors acknowledge, it is not clear what other knowledge is lost when moving to this new latent space. While hierarchical relationships between entities are now easy to understand, does the new space also preserve properties of each independent entity? Evaluations on a wider variety of downstream tasks would be helpful for establishing what kinds of semantic information are preserved.\n2. The analysis in Table 4 could have been cherry-picked. A more systematic version of the analysis in Table 4 is possible, and would be nice to see: specifically, one could search over a larger subset of WordNet, quantifying whether more deeply nested concepts correlate with higher h-norms, and whether more closely related concepts have significantly lower hyperbolic distances than unrelated but semantically similar concepts.\n3. When we classify whether two entities are taxonomically related, we usually care more about whether we can leverage these relationships to improve performance on some other metric. For example, better representational quality could lead to better deductive reasoning. Consider, for example, a counterfactual reasoning task where we say “Birds have fur. Is it therefore true that sparrows have fur?” A model that better captures taxonomic relationships should be better at this task. What I’m getting at is that it would be nice to evaluate on downstream tasks that require entity classification as a key part, rather than directly evaluating entity classification (a rare task). This would demonstrate the utility of this method, and increase interest and impact among a broader audience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback. We address your comments and questions below:\n\n------\n\n**Regarding Weakness 1**: \n\nOur current focus in this paper is on enabling transformer encoder-based language models to explicitly encode hierarchies. We recognise the importance of evaluating the preservation of other semantic information. In our future work, we plan to explore how our method impacts semantic properties unrelated to hierarchies, to ensure a balanced representation that retains existing language understanding.\n\n------\n\n**Regarding Weakness 2**:\n\nWe compare the Pearson correlation coefficients across different hyperbolic models to measure the linear relationship between entities' hyperbolic norms and their depths in WordNet. Our analysis shows that all hyperbolic models lead to a positive correlation between norms and depths, as expected. However, our HiT model demonstrates a stronger correlation than both PoincaréEmbed and HyperbolicCone.\n\n|HiT | PoincaréEmbed | HyperbolicCone |\n|:-------------------------:|:-----------------------------:|:----------------------------:|\n| 0.346                   | 0.130                       | 0.245                      |\n\n>Table: Statistical correlations between WordNet entities' depths and their hyperbolic norms in HiT, Poincaré Embed, and HyperbolicCone, respectively.\n\nWe will add this analysis in the revised manuscript.\n\nThe **Hard Negative** setting for each task is designed to determine if our models can distinguish unrelated but semantically similar concepts (e.g., sibling entities), as you suggested. Our results demonstrate that HiT models consistently outperform other baselines in this setting. The hyperbolic distances between closely related concepts will not be significantly lower than unrelated concepts because our loss functions are optimised for relative differences, but they are sufficiently distinct to enable good predictions.\n\n------\n\n**Regarding Weakness 3**: \n\nOur current evaluation tasks aim to understand if models can generalise from asserted subsumptions to inferred (transitive) and unseen (inductive) subsumptions, which is a critical capability for completing missing knowledge or enriching new knowledge in taxonomies and hierarchies. We acknowledge the importance of further downstream tasks to demonstrate more real-world utilities. Taking your counterfactual reasoning example, it can be formulated as predicting if “sparrow” is subsumed by “something that has fur”, which can be seen as an existential restriction (a kind of complex concept) in the context of ontology. We will extend our settings to handle complex, non-atomic entities in future work. \n\n\n------\n\n**Regarding Question 1**: \n\nWe followed the evaluation setting in [1] to maintain consistency with existing hyperbolic baselines.\n\n------ \n\n**Regarding Question 2**: \n\nUsing our GPU resources (see Appendix), taking all-MiniLM-L6-v2 as the base model, hierarchy re-training takes approximately 65 minutes, while standard fine-tuning takes about 17 minutes. Standard fine-tuning requires only a few epochs for convergence, but its performance is capped and cannot be further improved. Early stopping is a possible optimisation for hierarchy re-training to save time, but it was not considered in this work. We will explore this and other optimisation strategies in future work.\n\n------\n\n- [1] Ganea, Octavian, Gary Bécigneul, and Thomas Hofmann. \"Hyperbolic entailment cones for learning hierarchical embeddings.\" ICML (2018).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel method that retrains transformer encoders to better encode structures in language. The output embedding space is mapped onto a Poincaré ball, and the model is trained with specially designed objectives. Experiments show that the proposed approach outperforms baselines in tasks requiring the encoding of hierarchical structures.

Most reviews are ambivalent. The novelty and clarity of the work are appreciated by all reviewers. With the authors’ additional clarifications and experiments during the discussion period, most of the reviewers’ concerns have been addressed. However, a major concern that remains is its broad applicability, including other tasks and decoder architectures. Such efforts will help the paper appeal to a broader audience, but perhaps out of the scope. Overall, I recommend that this paper be accepted, and encourage the authors to incorporate the discussion into the revision.

At least one review is discounted due to lack of participation in the discussion period.
Decision: Accept (poster)