End-to-End Ontology Learning with Large Language Models
Authors: Andy Lo
Albert Q. Jiang
Wenda Li
Mateja Jamnik
Venue: NeurIPS 2024 poster (2024)
Keywords: Ontology Learning
Large Language Models
Knowledge Representation
Primary area: other
TL;DR: Building ontologies with Large Language Models

Abstract: Ontologies are useful for automatic machine processing of domain knowledge as they represent it in a structured format. Yet, constructing ontologies requires substantial manual effort. To automate part of this process, large language models (LLMs) have been applied to solve various subtasks of ontology learning. However, this partial ontology learning does not capture the interactions between subtasks. We address this gap by introducing OLLM, a general and scalable method for building the taxonomic backbone of an ontology from scratch. Rather than focusing on subtasks, like individual relations between entities, we model entire subcomponents of the target ontology by finetuning an LLM with a custom regulariser that reduces overfitting on high-frequency concepts. We introduce a novel suite of metrics for evaluating the quality of the generated ontology by measuring its semantic and structural similarity to the ground truth. In contrast to standard metrics, our metrics use deep learning techniques to define more robust distance measures between graphs. Both our quantitative and qualitative results on Wikipedia show that OLLM outperforms subtask composition methods, producing more semantically accurate ontologies while maintaining structural integrity. We further demonstrate that our model can be effectively adapted to new domains, like arXiv, needing only a small number of training examples. Our source code and datasets are available at https://github.com/andylolu2/ollm.

End-to-End Ontology Learning with Large Language Models
Andy Lo University of Cambridge cya140cam. ac uk
Albert Q. Jiang University of Cambridge qj2130cam ac. uk
Wenda Li University of Edinburgh wenda. li@ed. ac uk
Mateja Jamnik University of Cambridge
mateja. jateja.jammiike janikkmcnn cam acc. uk
Abstract
Ontologies useful for automatic machine processing of domain knowledge as they represent it in a structured format. Yet, constructing ontologies requires substantial manual effort. To automate part of this process, large language models (LLMs) have been applied to solve various subtasks of ontology learning. However, this partial ontology learning does not capture the interactions between subtasks. We address this gap by introducing OLLM, a general and scalable method for building the taxonomic backbone of an ontology from scratch. Rather than fo- cusing on subtasks, like individual relations between entities, we model entire subcomponents the target ontology by finetuning an L.LM with a custom regu- lariser that reduces overfitting on high-frequency concepts. We introduce novel suite of metrics for evaluating the quality of the generated ontology by measuring its semantic and structural similarity to the ground truth. In contrast standard syntax-bact based metrics, our metrics use deep learning techniques to define more robust distance measures between graphs. Both our quantitative and qualitative results on Wikipedia show that OLLM outperforms subtask composition methods, producing more semantically accurate ontologies while maintaining structural integrity. We further demonstrate that our model can be effectively adapted to new domains, like arXiv, needing only a small number of training examples. Our source code and datasets are available at https: : //gi thub. com/com///o/ooc andylolu2yylloooo ollmmmllm..
Introduction
An ontology formal and structural way of representing domain-specific concepts and their relations [16]. They can be simple (e.g., Wikipedia categories) consisting of concepts and only small number of types of taxonomic relations (e.g., is-a relationships). or they can be complex (c.g.. Schema.org consisting of axioms many types of relations. For example, a simple ontology for programming languages might contain two concepts "Dynamically- and one relation "Dynamically-typel language -> Python", representing the knowledge that Python a dynamically-typer language. A more complex ontology might contain axioms too, for example, "all programming languages are either dynamically statically typed". In this paper, focus on ontologies with only concepts and taxonomic relations. Compared to typical deep learning models, which represent knowledge implicitly in its weights, ontologies capture knowledge in structured and explicit manner, making them reliable, easy to edit and human-interpretable Such benefits of ontologies have led their wide adoption in practice. For example, Wikipedia categories have been
language" and "Python",
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Dataset
Target
Backpropagate
Training
LLM
Mask-regularised loss
Output
Evaluation Inference
Sum and prune
LLM
Document
Concept
Is-a relation
Gold standard evaluation
Figure 1: OLLM: Using annotations of documents with their relevant concepts, we train an LLM to model relevant subgraphs of the target ontology with a custom regulariser. During inference, the generated subgraphs for each document are summed and pruned to give the final output ontology. For evaluation, we measure the similarity between the generated ontology and the ground truth.
used for entity ranking [46] and information retrieval [42), or Schema.org [40] is a core component of the Semantic Web [1] initiative.
While ontologies are useful, building ontologies often requires substantial manual effort. Ontology learning (OL) is the study of automating the construction of high-quality ontologies scale. For simple ontology, this amounts to discovering the concepts and taxonomic relations, usually based on source corpus. In this paper we aim to develop Nomain-indeeendent methods for OL that are scalable and produce better ontologies.
Traditionally, OL is viewed extraction. In particular, prior works have demonstrated that state-of-the-art large language models (LLMs) can solve such subtasks effectively [4]. While studying subtasks permits fine-grained analysis and evaluation, it does not directly indicate the subsequent impact on the quality of the final ontology. Moreover, there is potential room for improvement by combining several subtasks into one, such as by modelling concepts and relations in conjunction. In this paper, we instead develop and evaluate methods that construct ontologies an end-to-end fashion to answer the following research questions:
composition of subtasks [3], such as concept discovery and relation
1. How can leverage LLMs' knowledge base to build ontologies from scratch?
Does our method scale efficiently practical problem sizes?
3. How well does our method generalise to new domains?
We introduce OLLM, an end-to-end method for using LLMs to construct ontologies at scale. Rather than focusing on individual relations between concepts, we finetune an LLM to model entire sub- components of the target ontology. The output ontology is generated by taking the sum of generated sub-components and applying simple post-processing. An overview of the pipeline is shown in Figure 1. To train OLLM, we collect the categorisation metadata for subset Wikipedia articles. We attempt to adapt an L.L to model the relevant categorisation subgraph for a particular Wikipedia article, but discover that direct finetuning leads to poor generalisation due to overfitting to high-level, frequently occurring concepts. Instead, we propose custom regulariser that reweights each concept based on its frequency of occurrence, which substantially improves generalisation.
We evaluate OLLM by measuring the similarity of the generated ontology with the ground truth. Current approaches for comparing ontologies rely on mapping components of the two ontologies onto each other, most commonly by literal text matching [30, 45]. This is unreliable when the two ontologies are not already sufficiently similar. Instead, we propose suite of evaluation metrics suitable for comparing arbitrary labelled graphs. These metrics compare edges and subgraphs of the two ontologies using pretrained text embedders to test for semantic and structural similarity. Both our quantitative and qualitative results reveal that LLM can already outperform existing

---

extraction- methods out of the box, and the performance is further improved by finetuning with our custom regulariser. We additionally demonstrate that OLLM can be adapted to build the arXiv ontology using only a small number of training examples, suggesting that our model can be applied to new domains in data-efficient way. In summary, our contributions are:
1. We constructed two datasets based on Wikipedia and arXiv, which can serve as standard datasets for future work studying end-to-end OL.
2. We created OLLM, a method that utilises LLMs to build ontologies from scratch. OLLM produces high-quality ontologies and serves as a strong baseline for end-to-end OL.
3. We developed new evaluation metrics for assessing the quality of the generated ontologies.
2 Background
An ontology is a structured way of representing concepts and relations of a shared conceptualisation, that is, domain knowledge [15, 16]. Ontologies can span a wide range of complexities. fully- fledged ontology might contain concepts, relations, constraints, and axioms that enable complex automated reasoning. this paper, we focus on the core building blocks of an ontology: concepts and taxonomic relations which represent is-a or is-subclass-of relationships between concepts. In some cases, the is-purt-of relation is also considered a taxonomic relation. We treat such an ontology as a rooted labelled directed graph where nodes represent concepts, edges represent taxonomic relations and the root node is the special concept of all concepts. A strict ontology asserts that the taxonomic relation is asymmetric and thus the graph must be acyclic, though in practice some ontologies, such as the Wikipedia ontology studied in this paper, may contain cycles. We therefore do not assume that an ontology graph is necessarily acyclic. Examples of ontologies include WordNet [33] with 1 17,659 concepts and 89,089 taxonomic relations, and the Gene Ontology [2] with 42,255 concepts and 66,810 taxonomic relations.
Ontology learning is the automatic extraction of ontological elements [17]. The most studied source of input is unstructured text, though there are also works semi-structures data like HTML [22]. In this paper, the input is a set of documents, each consisting of some unstructured text. We additionally assume each document is associated with one more concepts in the ground truth ontology, which we utilise for training. The goal is to reconstruct the ground truth ontology given the set of documents.
Prior works view OL as a composition of subtasks, and study each subtask in isolation [3, 6]. A typical pipeline for building a simple ontology is to first perform concept discovery (identify the nodes), and then relation extraction (identify the edges) [8, 24]. A notable approach for relation extraction is Hearst patterns [18]. Hearst patterns are hand-crafted lexico-syntacti patterns that exploit natural language structure to discover taxonomic relations. For example, the pattern "[noun phrase] such as [noun phrase]" matches phrases like "dogs such as chihuahuas", and thus can be processed by regular expressions to identify the relation "dog - chihuahua", Hearst patterns suffer from low recall, as the relations must occur in exact configurations to be identified by the rules. Roller et al. [39] suggest smoothing techniques to alleviate this issue though at the cost of lower precision.
Recently, language models have been used for OL. REBEL [7] treats relation discovery as a translation task, and finetunes encoder-decode LLMs to extract both taxonomic and non non-taxonomic relations. Babaei Giglou et al. [4] benchmarked a wide family of L.LMs for concept and relation discovery, and showed promising results. However, the quadratic complexity of link prediction makes this approach unscalable to large ontologies. We provide more discussion in Appendix A.2.3. There are also proof-of-concept works for building ontologies end-to-end with L.Mss. Funk et al. [13] proposes to build an ontology by recursively prompting LLMs, while Trajanoska et al. [44] generate the entire ontology in one completion. However, both studies are limited in the scale of the task and evaluation: they only considered ontologies of up to 1000 concepts and relied on manual qualitative evaluation. We bridge this gap by proposing method that can scale to practical problem sizes and new metrics for systematic qualitative evaluation.
The evaluation of ontologies is an open research area. The main approaches are gold standard evaluation [51], which matches elements of the generated ontology with a predefined target ontology: task-based evaluation [36], which measures the usefulness of the ontology on specific application; and human evaluation [5, 37]. In this paper, we evaluate by the gold standard metric as is the most straightforward approach when ground-truth ontology exists. Prior works have considered
3

---

Main topic classifcations

 [IN[T) Title: Hybridity
Hybridity, its most basic sense 

Main topic classifications -> Human behavior
Humanities
behavior
-> Human activities -> Culture ->
Sociology of culture
Main topic classifications Humanities Politics -> Politics by issue <> Politics and race
Politics
Human activities
Main topic classifications -> Politics -> Politics by issue -> Politica and race Main topic classifications Culture Sociology of culture

issue
Politics and race
Sociarogy of culture
Figure 2: Example subgraph induced for the Wikipedia page "Hybridity" (left), where 4 and C {Politics and race, Sociology where text coloured in grey ignored as training targets, still present as context later tokens.
of culture). The corresponding training text sequence (right),
matching concepts [30] and direct or indirect relations [23, 45] by literal text comparison. Others have also considered edit-distance [12] or bag-of-word. distributional similarity for text comparison [51]. These techniques for measuring semantic similarity may be considered unreliable and have been superseded by current methods [9]. We instead rely on more modern techniques like pretrained text embedders [10] and graph convolutions [26] to match substructures between the two ontologies.
3 OLLM
now introduce OLLM, our novel, simple and scalable method for end-to-end OL with LLMs. On high level, OLLM uses LLM model concept subgraphs of the target ontology by utilising linearisation scheme to transform subgraphs into string sequences. In contrast to learning individual edges, modelling subgraphs allows the model to learn higher-order structures, such as the interactions between three or more nodes. To create the training dataset, OLLM relies on the annotations of documents concepts to generate Document-subbraph pairings. Such subgraphs are much smaller than the complete graph, so they can be learned by the model more easily. The generated subgraphs for each document are summed into a weighted graph, and simple post proccessossingg applied obtain the final predicted ontology.
3.1 Subgraph modelling
Here, describe the method for creating eecumentt-subrrage pairings. Given document and its associated set of concepts C, we define the relevant paths as the set of paths of at most length N from the root any of the concepts in C. The relevant subgraph is the set of nodes (concepts) and edges (taxonomic relations) that occur at least once in the relevant paths. An example shown in Figure (left). The choice of N is task-specific and we describe method for choosing N in Section 5.1.
To employ LLMs to model the subgraphs, we must linearise the graph into a string sequence. Existing methods for autoregressive graph generation employ BFS [50] DFS [14] ordering starting at an arbitrary node. We instead choose to linearise the subgraph as a list of relevant paths that produced the subgraph in the first place. We do so over BFS/DFS ordering for three reasons: 1) the subgraph is defined from the relevant paths, which makes them the most natural representation; 2) we hypothesise that the hicrarchy of concepts in each path is desirable inductive bias for the hierarchical nature of an ontology; and 3) the path- based representation is much easier to describe in natural language instructions that LLM nompting-based baselines may produce reasonable results without finetuning. The linearisation template can be found Figure in Appendix A.1.2.
3.2 Post-processing
The final output graph is obtained by summing all generated subgraphs for each document and pruning low-weighted components. Given the generated subgraphs G₁ (V,, Fr),... Gn = (V,, F.n) the raw output graph is defined as Graw (Vraw, Enow) ), where Vraw U Vn and Eng - En

---

Each edge (u, 2) E Eraw is additionally weighted by the number of times it occurs in the collection of subgraphs: w(u, v) = D [[(u,0) En]. A few simple post-processing steps are then applied to Graw in order to prune it:
1. Self-loop pruning: All edges (u, u) E Eraw are removed.
2. Inverse-edge pruning: For (u, v) E Eraw, if (v. u) Eraw and w(v u) > w(u, v), remove (u, v).
That is, bidirectional edges are turned into unidirectional ones.
3. Absolute thresholding: Edges in Fraw with weight below the a-th quantile are removed, where
0 S S is a hyperparameter. This removes edges that are globally less important.
the outgoing edges from u, sorted by weight in ascending order. Let the cumulative weight be C(e)) شj w(ee)/ شك- w(e)). The edges fe, C(e)) S }} are pruned, where 0 S B < 1 is hyperparameter. This is similar to top-p sampling [19]. which we use to remove edges that are less important than their neighbours. 5. Clean up: After pruning all edges, nodes with no incoming or outgoing edges are removed.
Relative thresholding: For each vertex u E Viww let e1,...
We choose the hyperparameters a and by tuning on the validation set (Section 5.1).
Evaluating end-to-end OL
Ontology evaluation is hard problem as there are no quantitative definitions of what constitutes a "good ontology", and metrics generally only capture one aspect (e.g., structure but not semantics) of an ontology. We approach evaluation by treating the ground truth as a proxy for good ontology, and comparing the generated ontologies against the ground truth. Here, we describe how the ground truth is obtained, and introduce new evaluation metrics that are used for measuring ontology similarity.
4.1 Dataset
We collect the datasets for the ontologies considered in this paper: Wikipedia categories and the arXiv taxonomy. We use Wikipedia for learning and in-domain evaluation, and arXiv for out-of- domain evaluation. To build the Wikipedia dataset, we perform BFS traversal from its root category "Main topic classifications" up to depth 3. For every category encountered, we retrieve the titles and summaries (the text before the first section) of up 5000 pages that belong in that category. The source data is obtained from the Wikipedia API. The arXiv taxonomy is available from its home page, and the source corpus is constructed from the title and abstract of all the papers uploaded to arXiv in the years 2020-2001 2022 with more than or equal to 10 citations. In total, the Wikipedia dataset has 13886 concepts, 28375 taxonomic relations and 362067 documents, while the arXiv dataset has 161 concepts, 166 taxonomic relations and 1 126001 documents.
Generating the train and test splits from the datasets is a non-trivial problem. Each train- ing example consists of a document and its rele- vant subgraph (Section 3.1). The naive approach of randomly selecting a subset of document- subgraph pairs for the training likely leads to data leakage as there might be a significant over- lap between subgraphs in the training set and the test set. Instead, we first split the full ontology into train and test graphs, and then generate the training suuurrpt- pairs. This ensures that there are sufficiently many unseen concepts (and thus relations) in the test split, as shown in Figure 3. Our method is as follows:
Train
Eval
3499
2127
Train
Eval
303
1125
226
61
2
6280
56
Test
Test
(a) Wikipedia
(b) arXiv
Figure 3: Intersection of concepts among the train,
validation and test splits of the datasets.
1. Let dld be the set of top-level nodes, that is, children of the root node. Randomly partition Vlup into train tlup validation Vrlap and test Velpp splits in 7:3:10 ratio.
'httpa://eer //en. Been.wikipede www.wiieia............ wttg://pi.pppppppe php "Citation counts obtained from https:// h/aps://app /ppi. semanticscholar.orgg
5

---

2. Let be the depth on the full graph, that is, the distance of the furthest node from the root. The nodes of the train graph are taken as the union of all the nodes that are within distance d from any node in min' plus Vran and the root. The edges are all the edges in the full graph that have both endpoints the train graph. Similar applies for Volp and Vesst val
4.2 Metrics
Many existing methods for comparing ontologies rely on syntactic measures like string edit dis- tance [12] as a proxy for semantic similarity, or require every concept to be tagged with descriptions or documents for distributional semantics comparison [51]. To obtain more robust and general evaluation results, we introduce suite of similarity metrics that use modern methods like text embeddings [38]. Multiple metrics used they trade off between interpretability and comprehensiveness and aim to make them complementary by capturing different aspects an ontology. For example, com- paring ontologies by literal text equality is easy to understand but may be unreliable. In Section 5.4, we provide further discussion on evaluation metrics in the context of our experiment results. We denote the ground truth ontology graph as G = (V, E)) and the generated graph G' -V', F*).
Literal Fl While literal text matching unreliable, is also the simplest and the most interpretable. We treat this metric reference metric for sanity check. The Literal Fl metric [23] is given by the harmonic mean the precision and recall of the edges:
Literal precision = IEN E'1
E F' E
Literal recall
EE
Fuzzy F1 The Literal Fl metric puts a strong emphasis on using the correct wording, while in prac- tice, are interested in evaluating the semantics of ontology. For example, using a synonymous phrase for a concept should not be penalised. We utilise embeddings from a pretrained sentence trans- former [38] and use the cosine similarity of the embeddings measure semantic similarity. Specifi- cally, let NodeSim (u, u') E V x V + 1, be the cosine similarity between the sentence embed- dings for and u'. The Fuzzy F1 score is obtained from the fuzzy precision and recall, defined as:
K(u', e E' 3(u, v) NodeSim u, u') > t NodeSim(e, v') {}
Fuzzy precision
E
H(u,v) E E 33' v') e NodeSim (u, w') > t A NodeSim(e) (u, v') {
Fuzzy recall
E
where is the matching threshold. We use all-MiniLM--L--22 [38, 47] as the embedding model, and choose as the median cosine similarity between the synonyms in WordNet [33], computed as 0.436.
Continuous F1 With fuzzy comparisons, the matches between the edges of the generated and the ground truth graph are no longer one-to-one. This is problematic: consider two graphs A + B and B + A > B', where B and B' match fuzzily. Such graphs will achieve perfect Fuzzy F1 score yet they significantly differ. Additionally, we found that the previous metrics fail to provide a useful signal for hyperparameter tuning, particularly for our baselines where the generated graphs are poor. The Continuous F1 metric solves these issues by computing the highest-scoring edge matching between the two graphs, where the similarity score between (u, r) and (u', vf) is given by min (ann(Nedeim(( u, u²) NodeSim (e, v')). Obtaining such matching is equivalent to solving the linear assignment problem [32], which can be computed by the Hungarian algorithm [27]. The Continuous Fl score is obtained from the continuous precision and recall, given by:
Continuous precision =
Scoat
Continuous recall
Scont
E
E
where Scont is the score achieved by the best edge matching.

---

<s*
 Title: List of yeneral amards in the hunenilies
of general awards hunanities ... from country. 

Noin topic classifications Society >> Humanities +> Hunanities awards
Nain topi: classiicaatios Acedenic disciplines -> Humenities Humenities awards
topic classtfications Society Culture Cultural
awords
Nein topi: classifications Culture -> cultural 11sts -> Lists of awards
Nain topis classifications
Cultural lists Lists awards Nain topic classirications Humenities Hunanities awerds
Nein topi: classifications Acedemic cisciplines -> Liberal arts educetion iumanities Hunenities awards
(a) Direct finetuning
<S>(INST] Title: List of general awards in the hunenities
This list of general awrrd in the humanities from thet country. 

Noin topic classtrications
Hunanities Hunantties
Nein Lopic clesifications Acadenic disciplines -> Hunenilirs -> Humenilies ewards
Nain topis classtfications Society
Cultural lists Noin topic classtricattons Culture -> Cultural 1ists Lists of awords
Nain topi: tlassifications Lists -> Caltural lists +> Lists of awards
topic classtrications Hunanities Hunanities
Nein topic classifications Acedenic disciplines -> Liberal axts education Humanities -> Hunenities awards
(b) Finetuning with masked loss
Figure 4: Per token loss on a test set example of the final model trained with and without the custom masked loss objective. A stronger red colour represents higher cross-entropy loss. Within the top-level concepts (children of the root) shown here, "Culture" and "Humanities" are in the training while others are not. Using the masked loss objective improves generalisation on the high-level relations (e.g.. "Main topic classifications" - "Academic disciplines") while maintaining performance on lower-level relations.
Graph F1 Instead of individual edges, this metric aims to capture the wider structure of the two graphs. Intuitively, we want know how concepts are related their local neighbourhood. We do so by using simple graph convolutions [49] with K = 2 to compute graph-aware node embeddings after embedding each node with the pretrained embedder. Such embeddings in are compared against those in G' by cosine similarity, and the highest-scoring node matching, similar to the Continuous Fl metric, gives the graph similarity score. The Graph Fl score is computed from the graph precision and recall, defined as:
Sgraph
Sgraph
Graph precision
Graph recall
[V]
where 8 graph is the score achieved by the best node matching.
Motif distance Taking inspiration from classical network analysis, to evaluate the structural integrity of the generated graphs. Network motifs are reoccurring subgraphs in larger graph, most commonly 3-vertex subgraphs. They are typically indicative of the structural characteristics of the full graph. We define the motif distance as the total variation distance between the distribution of all 3-vertex subgraphs in G and G'.
network motifs [34, 41]
5 Experiments
We design our experiments to answer the following research questions:
1. Does OLLM produce better ontologies than traditional methods by subtask composition?
Can OLLM be easily adapted new domain?
We approach the questions by training OLLM on the Wikipedia dataset, and further transfer the model arXiv with small number arXiv samples. As baselines, we use two relation extraction methods, Hearst patterns [18, 39] and REBEL [7]. Relation extraction depends on successful concept discovery to produce high-quality ontologies. To estimate a ceiling to such baselines, we give the baselines substantial udvantage by providing them with the ground truth concepts in the test graph. The results show that even with such an advantage, OLLM outperforms the baselines on many metries, demonstrating the potential of OLLM for end-to-end OL (Section 5.3).
7

---

5.1 Implementation details
Analysing the per-token loss on the test split sequences of directly finetuned model (Section 3.1) shows that the model tends to memorise high-level relations from the training set, leading to poor generalisation, as shown in Figure 4 (top). The crux of the problem is that low-level relations are substantially more diverse than high-level ones: since we present both types of relations at the same rate to the model, it tends to overfit on high-level relations while underfitting on low-level ones. To alleviate this issue, we introduce a new training objective that randomly masks the loss contribution of frequently occurring relations. Suppose relation + is present times in the training set. During training. when u + V appears in one of the relevant paths, we mask the loss contribution of the tokens for e with probability max(1 M/n, 0), where M is constant for the average number of times relation is present in the training set. Intuitively, this regulariser ensures that frequent relations are only seen M times targets throughout training. hence reducing overfitting as shown in Figure 4 (bottom). Note that while v is masked from the target, its tokens are still present in the input sequence as context for later tokens. A concrete training example can be found in Figure (right).
We finetune Mistral 7B v0.2 [21] with Low-Rank Adaptation [20] on the masked loss objective. The model is trained the Wikipedia dataset for two epochs with Adam [25]. During inference, the outputs are generated with temperature 0. I and nucleus sampling [19] top-p of 0.9. We include a finetuning baseline without the masked loss objective, denoted as Finetune. To adapt OLLM for arXiv, we further finetune the model on 2048 document-subgrape pairs from arviv. We initialise new low-rank adaptors and train until the loss stops improving the validation set. We name these models OLLM (transfer) and Finetune (transfer) for training with and without the masked loss objective, respectively. Full details for the Wikipedia and arXiv experiments can be found in Appendix 1.1.2.
The hyperparameters for the post-processing steps are tuned by grid search on the validation set. We sweep over a E 1 geomspacc(L/|EEmg i 1, 21) and B E geomspace the values that maximise Continuous F1. For Wikipedia, we choose the subgraph modelling path length N = 4 as it is the smallest N such that almost all edges (> 99%) occur in at least one relevant subgraph. Such criterion is used since smaller N results in smaller subgraphs, which we expect to be easier to model accurately. We choose N for arXiv for the same reason.
21) 0.1, and use
5.2 Baselines
We give brief overview of the baseline methods here (in addition to Finetune and Finetune (transfer)). The full implementation details can be found in Appendix 1.1. All baselines produce weighted directed graphs which we apply the same post-processing steps as in OLLM (Section 3.2) to obtain the final predicted graph.
Memorisation Simply memorising the train graph is a surprisingly strong baseline due to the overlap between train and test graphs, especially for Wikipedia. The weight of each edge is given by the number of relevant subgraphs in which it appears.
Hearst We follow the improved implementation of Hearst patterns by Roller et al. [39]. The authors propose spmi, method which uses low-rank approximations to smooth the relation matrix so that two concepts can be compared even if there are no direct matches between them. We use the smoothed relation matrix to weigh the relations between the ground truth concepts. The additional hyperparameter for the rank of the smoothed matrix is tuned by grid search over the validation set.
The REBEL-large model [7] is an L.M trained to extract many types of relations from Wikipedia articles. We only take the "subclass of", "instance of", "member of" and "part of"" relations that were extracted. Similar to Hearst, we find that it fails to find many direct relations between ground truth concepts. The same low-rank smoothing technique is applied to improve recall.
REBEL
Prompting We test the Serro/One/Three-/iee performance of instruction-turae LLMs on the subgraph modelling task described in Section 1.1. To obtain more comparable results, we use Mistral 7B Instruct v0.2, the instruction-tunal version of the base model of OLLM, as the LLM for our prompting baseline. The prompt template used is shown in Figure 6 in Appendix A.
Finetune To test the effectiveness of our masked-loss objective, we introduce a direct finetuning baseline using the same configuration as OLLM except it is trained without loss masking.
8

---

Table 1: Evaluation metries of OLLM and baselines on Wikipedia and arXiv. OLLM performs
particularly well in modelling semantics, and remains competitive syntactically and structurally.
Dataset
Method
Literal
Fuzzy
Cont. FI f
Graph
Motif Dist.
Wikipedia
Memorisation
0.134
0.837
0.314
0.419
0.063
Hearst
0.003
0,538
0,350
0,544
0.163
Rebel
0.004
0,624
0.356
0.072
0.132
Zero-shot
0.007
0.871
0.455
0.639
0.341
One-shot
0.031
0.888
0.477
0.610
0.314
Three-shot
0.031
0,880
0.475
0.622
0.354
Finetune
0.124
0.884
0.470
0,588
0.050
OLLM
0.093
0.915
0.500
0,644
0.080
arXiv
Memorisation
0.000
0.207
0.257
0.525
0.037
Hearst
0.000
0.000
0.151
0,553
0.098
Rebel
0.000
0.060
0.281
0.546
0.088
Zero-shot
0.025
0,450
0,237
0.414
0.145
One-shot
0.072
0.460
0.290
0.433
0,293
Three-shot
0.051
0,405
0.212
0.385
0.124
Finetune (transfer)
0,000
0.440
0.225
0.441
0.148
OLLM (transfer)
0.040
0.570
0.357
0.633
0.097
5.3 Results
We first evaluate whether OLLM can accurately create ontologies with many concepts and relations, such as the Wikipedia categories. Computationally, OLLM required 12 A100-hours for training and 7 A100-hours for inference to generate an ontology for Wikipedia. This is modest cost in current standards, which demonstrates the scalability of OLLM for real-world problems. In terms of performance, OLLM produces the most semantically accurate ontology in comparison to our baselines as presented in Table 1. Across all of Fuzzy F1, Continuous Fl and Graph F1, we observe the trend that OLLM scores the best, followed by Finetune and Prompting, and lastly Hearst and REBEL. This surprising. as it suggests that the combination of LLMs with our subgraph modelling framework is sufficiently strong inductive bias for LLMs to outperform traditional methods even without finctuning. However, prompting alone is not sufficient to build high-quality ontologics. On the Motif Distance metric, prompting methods score poorly 0.314 0.354 in comparison to 0.050 and 0.080 for Finetune and OLLM respectively. This shows that using LLMs out-of-the--be for subgraph modelling results in poor structural integrity, though this issue is solved by finetuning. Qualitatively, we observe that OLLM can adhere to the clear, explicit naming style of Wikipedia, even on unseen topics in the set. For example, it generates "Mathematical categories" and "Groups (mathematics)" under the parent concept "Mathematical structures" to distinguish from the natural language sense of categories and groups (Figure 1 Ic). Such style is not learned by the prompting baselines: Three-shot generated "Elections + France", while it most likely meant "Elections + Elections France" (Figure 18c). More sample outputs are shown in Appendix 1.4.1.
The Xiv task differs from the Wikipedia task as it has much fewer relations, and there is even less overlap between the train and test split. This imposes great challenge Finetune and OLLM as they need to generalise with a limited diversity of training samples. Despite such constraints, OLLM is substantially better than other methods in modelling the semantics of the test graph. On the Fuzzy FI, Continuous F1, and Graph Fl metrics, OLLM performs the best among all methods with 0.570, 0.357, and 0.633, significantly higher than the next-best of 0.460, 0.290 and 0.546 respectively. Inspecting the generated ontologies (Appendix A.4.2), we observe that prompting baselines tend to produce repetitive concepts such as "Machine Learning and Artificial Intelligence" and "Artificial Intelligence and Machine Learning" (Figure 27), while Hearst and REBEL put almost all concepts under the same parent concept(s) (Figures 23 and 24). We also found that OLLM's output for arXiv contains concepts from Wikipedia, but restructured in way that fits the arXiv ontology. For example, "Life sciences" and "Biological evolution" appear in the Wikipedia training set under the same parent category "Life" with no direct links between them. On the generated graph for arxiv, "Life sciences" instead promoted to one of the top-level concepts with "Biological Evolution" as one of children, which better fits the "fields of science" style of the arXiv ontology (Figure 20). This demonstrates that OLLMM can adapt to produce new type of ontology by restructuring its learned concepts, all using just small number of training samples.

---

In summary, OLLM scores the best or is competitive across all metrics in both tasks, with the notable exception of the Literal Fl metric. We attribute this to the fact that Literal F1 is sensitive factors like casing and choice of words, and generally only measures syntactic similarity. For example, we see that a suboptimal baseline like Memorisation scores the best on this metric with 0.134 on the Wikipedia task. This reflects that syntactic similarity generally does not entail semantic similarity, so syntax-based metrics should not be used as stand-alone measures for ontology quality.
5.4 Meta-evaluation
In this section, we analyse the usefulness of our new metrics for measuring graph similarity and discuss the limitations of existing metries. On the Wikipedia task, Memorisation, despite being clearly the worst in Continuous Fl and Graph F1, performs the best on Literal Fl and the second-best on Motif Distance. This can attributed to the fact that Literal Fl is sensitive to semantically insignificant syntactic differences such as casing and word form, and thus when the training and test set has non-trive non-trivial overlap (Figure 3), it is biased towards methods that overfit. Similarly, as per the method described in Section 4.1, the data splits are constructed with structural symmetry, hence we expect the train and test splits to have a similar graph structure even though the represented concepts different. As result, methods that tend to overfit, for example, Memorisation and Finetune, achieve the best scores on Motif Distance. This demonstrates that Literal F1 and Motif Distance only capture syntactic and structural similarity respectively, and thus should not be used as stand-alone metrics for evaluation.
Analysing the edge and node matchings found by our Continuous Fl and Graph Fl metrics on arXiv reveals that they successfully capture some human intuition on semantic similarity between the two ontologies. In Figures 9 and 10, visualise the ontology generated by OLLM and the ground truth and observe that semantically similar components in the two graphs indeed get matched. For example, the "Physics" and "Mathematics" clusters in the generated graph get matched with the "Mathematics" cluster in the ground truth, "Data Analysis" and "Information" get matched with "Statistics", "Economics" with "Quantitative Finance", and "Life Sciences" with "Quantitative Biology". This suggests that our edge/node matching procedure is capturing a "semantic graph isomorphism" that allows one to compare similar components in the two graphs, even if they do not exactly share the same concepts. We believe this example of a semantic mapping from one ontology to another is strong evidence that our metrics are capturing meaningful qualities of the ontologies.
Discussion
Limitations We only study and evaluate the construction of simple ontologies with only concepts and taxonomic relations. A potential approach to extend OLLM to produce non-taxonomic relations is to add tags indicating the relation type to each edge when linearising the subgraphs for sequence modelling. New evaluation metrics might also be required to handle multiple types of relations. Another limitation is that the taxonomic relations in the generated ontologies are not necessarily transitive due to the existence of cycles. This is general problem for many OL methods and there are existing works on cycle removal algorithms for cleaning hierarchies [43, 52]. We ablate this in Appendix 1 and found that the generated ontology can be made consistent by removing a small number of edges. Furthermore, we were unable to fully control for data contamination as the pretraining dataset of Mistral 7B is not publically known. We do, however, observe that the generated ontologies are sufficiently different from the ground truth, indicating that OLLM is not directly remembering samples from its pretraining stage.
Conclusion In this paper, we introduce a general method for building ontologies in an end-to-ent fashion. We propose a set of metrics for end-to-end OL that measures the semantic and structural similarity between arbitrary labelled graphs. Our model, OLLM, outperforms traditional subtask composition methods in reconstructing the Wikipedia categories, and can be transferred to build ontologies for arXiv after finetuning on a small number of examples. Using LLMs as the backbone for subgraph modelling opens up exciting avenues for future research. For example, one may generate ontologies from corpora with images using vision language models [1]].
10

---

7 Acknowledgements
We thank Dr Thomas Sauerwald for suggesting network motif's as basis for evaluation. AQU acknowledges the support of a Peterhouse Graduate Studentship.
References
[1] Grigoris Antoniou and Frank Van Harmelen. A semantic web primer. MIT press, 2004.
[2] Michael Ashburner, Catherine A Ball, Judith A Blake, David Botstein, Heather Butler, J Michael Cherry, Allan P Davis, Kara Dolinski, Selina S Dwight, Janan T Eppig, et al. Gene ontology: tool for the unification biology. Nature genetics, 25(1):25-29, 2000.
[3] Muhammad Nabeel Asim, Muhammad Wasim, Muhammad Usman Ghani Khan, Waqar Mah- mood, and Hafiza Mahnoor Abbasi. A survey of ontology learning techniques and applications. Database, 201 2018bbay000 2018.
[4] Hamed Babaei Giglou, Jennifer D'Souza, and Sören Auer. Llms4ol: Large language models for ontology learning. In International Semantic Web Conference, pages 208-427. Springer, 2023.
[5] Janez Brank, Marko Grobelnik, and Dunja Mladenic. A survey of ontology evaluation tech- niques. In Proceedings of the conference on data mining and data warehouses (SiKDD 2005), pages 166-170. Citeseer, 2005.
[6] Paul Buitelaar, Philipp Cimiano, and Bernardo Magnini. Ontology learning from text: methods, evaluation and applications, volume 123. IOS press, 2005.
[7] Pere-Lluis Huguet Cabot and Roberto Navigli. Rebel: Relation extraction by end-to-end language generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2370-2381, 2021.
[8] Philipp Cimiano and Johanna Völker. Text2onto: A framework for ontology learning and data-driven change discovery. In International conference on application of natural language information systems, pages 227-238. Springer, 2005.
[9] Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes, Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:
2017.
[10] Jacob Devlin, Ming- deep bidirectional transformers for language understanding. arXiv preprint arXiv: 180000880 2018.
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of
[11] Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-tere recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2625-2634, 2015.
[12] Marc Ehrig, Peter Haase, Mark Hefke, and Nenad Stojanović. Similarity for ontologies a comprehensive framework. In European Conference Information Systems, 2005. URL
http://pp...mmaaticscchcoooooooo wwwoooo//oooo CCrrporrpusD:D:. 9982461.
[13] Maurice Funk, Simon Hosemann, Jean Christoph Jung, and Carsten Lutz. Towards ontology construction with language models. arXiv preprint 200.09230, 2023.
[14] Nikhil Goyal, Harsh Vardhan Jain, and Sayan Ranu. Graphgen: A scalable approach to domain-agnostic labeled graph generation. In Proceedings of The Web Conference 2020, pages 1253 1263, 2020.
[15] Thomas R Gruber. A translation approach to portable ontology specifications. Knowledge acquisition, 199-229,, 1993.
[16] Thomas Gruber. Toward principles for the design of ontologies used for knowledge sharing? International journal of human- comput-r studies, 43(5-6):907-928 1995.
11

---

[17] Maryam Hazman, Samhaa R El-Beltagy, and Ahmed Rafea. A survey of ontology learning approaches. International Journal of Computer Applications, 22(9):36-43, 2011.
[18] Marti A Hearst. Automated discovery of wordnet relations. WordNet: an electronic lexical database, 2, 1998.
[19] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv: (e00. 09751, 2019.
[20] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 10X.::2/06.09685, 2021.
[21] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprint arXiv:2310.06825 2023.
[22] Lobna Karoui, Marie-Aude Aufaure, and Nacera Bennacer. Ontology discovery from web pages: Application to tourism. In In the Workshop of Knowledge Discovery and Ontologies. Citeseer, 2004.
[23] Vipul Kashyap. Cartic Ramakrishnan, Christopher Thomas, and A. Sheth. Taxaminer: an experimentation framework for automated taxonomy bootstrapping. Int. J. Web Grid Serv,, 1: 240-266, 2005. URL https: www.ntiisccooiii.o.........i.coe .rrg prrgCorppusDD. 5549251.
[24] Neha Kaushik and Niladri Chatterjee. Automatic relationship extraction from agricultural text for ontology construction. Information processing in agriculture, 5(1):60-73, 2018.
[25] Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint
1421::1412.6980, 2014.
[26] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint 1609:0/609.0000000 2016.
[27] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
(28) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 1-626, 2023.
[29] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-scquence pre-training for natural language generation, translation, and comprehension. arXiv preprint 1rXiv:1910.134611 2019.
[30] Alexander Maedche and Steffen Staab. Measuring similarity between ontologies. In Asunción Gómez-Pérez and V. Richard Benjamins, editors, Knowledge Engineering and Knowledge Management: Ontologies and the Semantic Web, pages 251-263, Berlin, Heidelberg, 2002.
Springer Berlin Heidelberg. ISBN 978-3-540-45810-41
[31] Christopher Manning, Mihai Surdeanu. John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. The stanford corenlp natural language processing toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pages 55-60, 2014.
[32] Silvano Martello and Paolo Toth. Linear assignment problems. In North-Holland Mathematics Studies, volume 132, pages 289-282. Flsevier, 1987.
[33] George Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11): 39-41, 1995.
12

---

[34] Ron Milo, Shai Shen-Ore Shalev Itzkovitz, Nadav Kashtan, Dmitri Chklovskii, and Uri Alon. Network motifs: simple building blocks of complex networks. Science, 298(5594):824-827)) 2002.
[35] Simone Paolo Ponzetto, Michael Strube, et al. Deriving large scale taxonomy from wikipedia. In AAAI, volume 7, pages 1440-1445, 2007.
[36] Robert Porzel and Rainer Malaka. A task-based approach for ontology evaluation. In ECAI Workshop on Ontology Learning and Population, Valencia, Spuin, volume 1. Citeseer Valencia, Spain, 2004.
[37] Joe Raad and Christophe Cruz. A survey on ontology evaluation methods. In Proceedings of the International Conference on Knowledge Engineering and Ontology Development, part of the 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management, 2015.
[38] Nils Reimers and Iryna Gurevych. Sentence-bert Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 11 2019. URL htwp:htt:p:/eeee /atpiv/aaaivvaa//a org/ abs/1908. 1000/ 10084.
[39] Stephen Roller, Douwe Kiela, and Maximilian Nickel. Hearst patterns revisited: Automatic hypernym detection from large text corpora. arXiv preprint 106.:1806.03111, 2018.
[40] Schema.org. Schema.org, 2011. URL https:tt//// //www. schema. org/
[41] Shai S Shen-Orr, Ron Milo, Shmoolik Mangan, and Uri Alon. Network motif's in the transcrip- tional regulation network of escherichia coli. Nature genetics, 31(1):64 68, 2002.
[42] Philipp Sorg and Philipp Cimiano. Exploiting wikipedia for cross-lingual and multilingual information retrieval. Data Knowledge Engineering, 74:26-45, 2012.
[43] Jiankai Sun, Deepak Ajwani, Patrick Nicholson, Alessandra Sala, and Srinivasan Parthasarathy. Breaking cycles in noisy hierarchies. In Proceedings of the 2017 ACM Web Science Conference, pages 151-160, 2017.
[44] Milena Trajanoska, Riste Stojanov, and Dimitar Trajanov. Enhancing knowledge graph con- struction using large language models. arXiv preprint arXiv:2305.04676, 2023.
[45] Pucktada Treeratpituk, Madian Khabsa, and C. Lee Giles. sraph-based approach to auto- matic taxonomy generation (grabtax). ArXiv, abs/1307.171s 2013. URL https://api. semanticscholat prrg/CorpusI::8600 8625171.
[46] Anne-Marie Vercoustre. Jovan Pehcevski, and James A Thom. Using wikipedia categories and links in entity ranking. In Focused Access to XMI. Documents: 6th International Workshop of the Initiative for the Evaluation of XML Retrieval, INEX 2007 Dagstuhl Castle, Germany, December 17-19, 2007. Selected Papers 6, pages 321-335. Springer, 2008
[47] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776-5788, 2020.
[48] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le. Denny Zhou, et al. Chain-of of-thuht prompting clicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[49] Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning. pages 6871- PMLR, 2019.
[50] Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. Graphrnn: Generat- ing realistic graphs with deep auto-regressive models. In International conference on machine learning, pages 5708-5717. PMLR, 2018.
13

---

[51] Elias Zavitsanos, Georgios Paliouras, and George A. Vouros. Gold standard evaluation of ontology learning methods through ontology transformation and alignment. IEEE Trans- actions on Knowledge and Data Engineering, 23:1635 23:1635-1648, 2011. URL. https://api. semanticscholar.org/ :
[52] Torsten Zesch and Iryna Gurevych. Analysis of the wikipedia category graph for nlp applications. In Proceedings of the Second Workshop on TextGraphs: Graph- Based Algorithms for Natural Language Processing, pages 1-8, 2007.
14

---

A Appendix supplemental material
A.1
Experiment details
A.1.1
Wikipedia
Some prior works that use Wikipedia categories as a dataset perform additional filtering of concepts considered "meta- entea-cate mainly used for page management [35]. We instead decided not to further filter the source data to minimise external bias. We note that it is often not clear-cut whether Wikipedia category is just for page management. For example, the Wikipedia categories of the form "Lists of [subjecr]" refer to the special type of articles where the main body is bullet point/table listing of the subject, which is useful concept in the Wikipedia domain.
A.1.2 OLLM
For the Wikipedia experiment, we use Mistral 7B v0.2 (not instruction-tuned [21] as the base model. We attach LoRA [20] adaptors to all attention and feed-forward layers with parameters r = 32 and 16. The model is trained for 2 epochs (Rs 17K steps) with batch size 16, context length 2048, and is optimised with Adam using a constant learning rate of le-5 with warm-up from zero for the first 100 steps. Finetune uses the same configuration. Training takes 12 A100-hours.
For the arXiv experiment, we further finetune the model trained on Wikipedia with masked loss objective on 2048 document-subgraph pairs from the arXiv training set. We merge the LoRA adaptors from the Wikipedia experiment and initialise new ones with r 8 and a The model trained with batch size 16 and Adam with constant learning rate 3e-6 and warp-up from zero for the first 10 steps. Training terminates when the loss stops improving on the evaluation set, which happened at step 288. Finetune (transfer) uses the same configuration Early stopping happened at step 192.
For both experiments, finetune the model with the instruction template similar that of Mistral 7B instruct v0.2. The format is shown below:

 
]
Title: {{ title }} {{ abstract 
]
(% for path in paths 1)
{ path I join(" -> ") }}
1% endfor KF
</a>
Figure 5: Linearisation template for OLLM training.
For inference, we use the vLL.M [28] server which achieves a throughput of ~ 10 documents per second. Inference on the validation and test splits of both datasets takes 12 A100-hours total.
A.1.3 Hearst
The Hearst baseline follows the implementation by Roller et al. [39]. Using the tokenization, part- of-speech tagging, lemmatisation, and token regex functionality of the CoreNLP pipeline [31], taxonomic relations are extracted according to the 28 Hearst patterns used by the authors. Processing all documents takes 10 CPU-hours.
Following the spmi method, low-rank smoothing is applied to the relation matrix to allow com- parison between any two concepts even if they are not directly related by an extracted rela- tion. The rank of the smoothed matrix, r. is hyperparameter which we tune by sweeping over r E (5, 10, 15, 20, 25, 50, 100, 150, 200, 250) on the validation set. This defines dense weighted graph as the raw output. Unfortunately, computing Continuous Fl on a dense graph is very slow, especially for Wikipedia. This because the Hungarian algorithm used for solving the optimal matching between edges has time complexity ((N3), where N is the number of edges. To bypass this issue, we perform a pre-filtering step of only exporting the top 10 weighted edges in the smoothed relation matrix, where [ is the number of nodes in the graph. For the datasets considered,
15

---

this density of edges is still much higher than that of the ground truth, and thus, we expect this to have minimal impact on the final output after post-processing.
A.1.4 REBEL
We use REBEL-large [7] in the implementation. The model is encoder-decoder transformer based on BART-large [29] with 406M parameters. We sample the model with the default configuration used by Cabot and Navigli [7]. The model is trained to predict 220 types of relations, most of which are taxonomic relations. We filter the extracted relations and only keep those tagged with "subclass of", "instance of"" "member of", and "part of"" relation types. The same low-rank smoothing method as Hearst is applied to the raw extractions. Processing all documents takes A100-hours.
A.1.5
Prompting
To obtain more comparable results, we use Mistral 7B Instruct v0.2, the instruction-tunet version of the base model of OLLM, as the LLM for our prompting baseline. For One-shot and Three-shot, we randomly sample examples from the training set for each query. The output is parsed using regex and results that do not match the regex are discarded. We perform manual prompt engineering by inspecting individual responses. The final prompt template is shown in Figure 6. The total inference cost for prompting baselines is se 50 A100-hours.
16

---

The following article's title and abstract. Your task is assign this article suitable category hierarchy. category is typically represented by a word or a short phrase, representing broader topics/concepts that the article is about. category hierarchy represented by a collection of paths from the generic root category "Main topic classifications" to specific category suitable for the article. specific as you move from the root the leaf.
topics titles should become more and more
{% if examples
[eemple llength 0 % (% for example in examples %) EXAMPLE {{ loop.index
###
### ARTICLE ###
Title: {{ example[Ptitle]] }}
{{ (aample(abbttractt) }}
### END ARTICLE ###
{% for path in example pemplel'pattee ['paths']
{ path join(" -> ") }}
{% endfor %}
### END EXAMPLE { loop. index }} ###
(% endfor %}
(% else %}
You must answer in the format of: Main topic classifications -> Broad topic 1 -> Subtopic -> -> Most specific
topic
Main topic claseifications -> Borad topic 2 -> Subtopic -> -> Most specific topic 2
0% endif %
#*# ARTICLE ###
Title: {( title }} {{ abstract }}
### END ARTICLE ###
Provide a category hierarchy for the above article. (% if examplesllengs llength > 0 %}
Use the
format the examples above. 0% else X}
Use the format described above.
5% endif X}
Figure 6: Prompt template used for the Zero/One/Three-shot baselines.
A.1.6 Hyperparameters
The raw generated outputs of all methods post-processed with the same scheme described in Section 3.2. The best hyperparameters for the post-processing step found by grid search on the validation reported in Table 2.
17

---

Table 2: Values of the best hyperparameters found by grid search. is the rank of the low-rank smoothing, only applicable to Hearst and REBEL. a 3 = 0 means no edges are pruned from the raw output apart from self-loop and inverse edge removal.
Dataset
Method
B
Wikipedia
Memorisation
0.058489
Hearst
0.786685
0
5
REBEL.
0.872544
20
Zero-shot
0.976781
0.298107
One-shot
0.990906
0.346684
Three-shot
0.991955
0.530957
Finetune
0.883848
0.058489
OLLM
0.974330
0.025893
arXiv
Memorisation
0.340246
0
Hearst
0,595878
0
150
REBEL
0.836685
100
Zero-shot
0,999896
0.346684
One-shot
0,999611
0.401187
Three-shot
0,999851
0.298107
Finetune (transfer)
0.988129
0.346684
OLLM (transfer)
0.983681
0.123872
A.2 Ablations
In this section, we present the results of our ablations regarding output consistency, the benefits of more advanced prompting techniques, and comparison against L.LMs4OL. [4).
A.2.1 Consistency
common assumption of taxonomic relations is its transitivity and anti-symmetry. One limitation of many OL methods, including OLLM, is that they do not guarantee that the generated ontology cycle- free, leading to inconsistent taxonomic relations. To achieve consistency, generic post-processing techniques [43] can be applied to remove such cycles.
We analysed the ontologies generated by OLLM and found only 97 simple cycles Wikipedia and none in arXiv. Using the greedy algorithm of repeatedly removing the edge that breaks the most simple cycles (a heuristic to the smallest set of edges whose removal makes the graph acyelic), prune all such cycles and make the ontology consistent by removing just 26 of 10414 edges in Wikipedia. This is surprising considering we did not explicitly optimise our model to satisfy consistency.
A.2.2 Chain of thought prompting
More sophisticated prompting techniques, such as chain-of-thought (CoT)) [48] have been shown bring significant improvements in LLM inference. We explore whether we establish strong baselines here by employing CoT in our prompting methods.
We extend the zero-shot prompting method such that prediction involves two rounds of inference: In the first round, we ask the model to describe the possible relevant concepts for the input document and to explain its reasoning. Then, ask the model to predict the subgraph in the specified format given the additional, self- geferated context. The prompts used are shown below:
We tested the CoT method on Wikipedia and found no significant difference from basic zero-shot prompting, as shown Table 3. We attribute this to the fact that CoT prompting primarily aims to improve logic and reasoning. We hypothesise that the performance in OL is more dependent on the model's understanding of natural language than its ability to perform multi-ster reasoning, hence we do not observe any significant improvement from COT.
18

---

The following article's title and abstract. Briefly break down the topics ( both specific and general concepts) relevant to this article. Explain your reasoning step by step.
### ARTICLE ###
Title: { title }} { abstract }} ### END ARTICLE ###
Figure shain-of-thoughh first prompt
Your task now is to assign this article to suitable category hierarchy. A category is typically represented by word short phrase, representing broader that the article about. A category hierarchy represented by a collection paths from the generic root category "Main topic classifications' to a specific category suitable for the article. The topic titles should become more and more specific you move from the root to the leaf.
You must answer in the format of: Main topic classifications -> Broad topic 1 -> Subtopic -> -> Most specific
topic 1
Main topic classifications Broad topic 2 Subtopic
Most specific
topic
Figure Chain-of of-thought second prompt
A.2.3 Comparison against LLMs40L
In this ablation, evaluate whether the improvement by OLLM is due the improved methodology (end-to-end modelling) or simply due to the use of LLMs. One way to construct ontologies with LLMs proposed by LLMs40L is to first prompt LLMs for possible concepts in document, then link prediction by prompting for yes/no response. Unfortunately, constructing a baseline from such two subtasks is non-trivial. We encountered significant scalability issues in the link prediction stage it required (n²²) inferences. We make two modifications to overcome such limitation:
After the concept discovery stage, we only discard all but the n most frequent concepts to limit the number of inferences required during link prediction, where n is the number of concepts in the ground truth.
Instead of using zero-shot Mistral 7B as the link predictor, we use a finetuned BERT as the link predictor as it runs much faster. Given that LLMs4OL demonstrated that finetuned models perform much better than zero-shot inference on link prediction, we expect the finetuned BERT to be at least good, if not better, than zero-shot Mistral 7B this subtask.
We design this ablation such that it is comparable to zero-shot end-to-end modelling: both use zero-shot Mistral 7B the backbone, just utilised in different ways. We tested this method Wikipedia and found that it is worse than zero-shot end-to-end modelling on all metrics except Motif Distance, as shown in Table 4. This is evidence that our end-to-end modelling approach is clear improvement over traditional subtask-based OL. Not only does LLMs40L suffer from significant scalability bottlenecks thus unlikely be scalable to solve large problems, its performance is also worse. The results suggest that we can more effectively and efficiently leverage the capabilities of LLMs beyond just solving subtasks, such as by predicting subgraphs.
19

---

Table 3: Comparison of Zero-shot with and without Chain-offthougee prompting. significant difference in performance.
There is no
Dataset
Method
Literal FI f
Fuzzy FI f
Cont.
Graph FI
Motif Dist.
Wikipedia Zero-shot
0.007
0.871
0.455
0.639
0.341
shot CoT
0.007
0.873
0.449
0.635
0,357
Table Comparison of Zero-shot end-to-end modelling and LLMs4OL-style modelling with zero-
shot concept discovery and fine-tuned BERT link prediction. LLMs40L generally performs worse than zero-shot.
Dataset Method
Literal FI f
Fuzzy F1 1
Cont. FI f
Graph FI 1
Motif Dist. +
Wikipedia Zero-shot
0.007
0.871
0.455
0.639
0.341
LLMs40L 0.003
0.841
0.428
0.482
0.092
A.3 Visualising evaluation metrics
A.3.1 Visualisation of node matching in Graph F1
Galle
Figure 9: Highest scoring node matching from the Graph F1 metric between the ontology generated by OLLM (teal) and the ground truth ontology (black). The matching between nodes shown in red, where the opacity of the edge indicates the similarity score (weaker links Visually, the matching defines a clear alignment of the two graphs: from the centre to the left we have the Mathematics-relatee concepts; at the top right we have Biology-related concepts; and the bottom right we have Economics-related concepts.
transparent).
20

---

A.3.2 Visualisation of edge matching in Continuous F1
Sime Pooey
Figure 10: Highest scoring edge matching from the Continuous F1 metric between the ontology generated by OLLM (teal) and the ground truth ontology (black). The matching between edges is shown in red, where the opacity of the edge indicates the similarity score (weaker links are more transparent). Visually, the matching defines clear alignment of the two graphs: in the bottom left and centre we have the Mathematics-relatee concepts; at the right we have Biology-related concepts; and at the top left we have Fconomics-related concepts.
A.4 Visualisation of generated ontologies
A.4.1 Wikipedia
We include some generated outputs for Wikipedia here. Since the full generated output is too large to visualise, we plot subgraphs of the output instead. We sample the subgraphs by the following method:
1. Pick random node in the generated graph.
2. Get the induced subgraph by the 1-hop neighbourhood of the chosen node.
3. Include the shortest path from the root "Main topic classifications" to the chosen node such path exists.
4. Repeat from step if the subgraph has more than 30 nodes or less than 5 nodes.
We apply the filtering step (step 4) as subgraphs with too many nodes are difficult to inspect manually, and those with too few uninformative. For Hearst, we choose the filtering upper bound to be 50 nodes as we fail to find subgraphs smaller than 30 nodes quickly. We additionally colour each edge black if it occurs literally in the training graph, blue if it occurs literally in the test graph, and red otherwise.
21

---

BlograpNes bielogost
Brologica research stubs
Biology
olbeical evolution
Brological research
Blotochnology
Biclogy
Life
Hain topic clesstiratioos
Bioloical dlasificatoon
Bioiog sts
Biological concepts
Biclogy
Biology organisations
Bloogy
(a) Biology
Languape tty
Lang polcycorg
Langugg
(b) Language policy
Algebraic structures
Mathematical objects
Topological
Mathematical categories
Mathematical structures
Mathematics
topic classifications
Algebras
Groups (mathematics)
Ordered sets
(c) Mathematical structures
Figure 11: Sub-ontologies for Wikipedia generated by OLLM, centred various topics.
22

---

Cuibonn Srance
criett
Energy consuration
Energe
Main tople Chee Poaatone
Onergy ecoromics
Benewable energy aconomy
Energy
Enery palyy organzations
Eneegy accidnts and incidests
vs erengy economy
Fuel Tyyor
(a) Energy economics
Internet activism stubs
Main topic classifications
Politics Political activism
Internet activism
Internet activism organizations
Internet activists
(b) Internet activism
Hortvesntionl
Foot
dyhaneticaa obcests
Theores
Terwwro Becme
Motheratic
Scovt's Theole
Main clataioattoon
SXl theenss
Thecria swayyhing
(c) Theories
Figure 12: Sub-ontologies for Wikipedia generated by Finetune, centred various topics.
23

---

Goods
Coins
Historical objects
Archocological artitccts
Buildings and structures
ExdNLmbb
Satellttes
Swlass
Main togic dlasifticaiions
Cultre
objects
Carembiiaal objects
Machines
Synthetic biology
Gemetically modilled organisms.
(a) Aritificial objects
Confidence tricks
Internet froud
Froud
Deception * Human behavior
Main topic classifiiaaions
Fake news
(b) Fraud
spirits
Light and religion
Sacred natural sites
Water and religion
Nature and religion
Roligian
Main topic classifications
Nature deitles
Sanamahism
Fire religion
Animals in religion
(c) Nature and religion
Figure 13: Sub-ontologies for Wikipedia generated by Memorisation, centred various topics.
24

---

Gooug
WAIN MAAi
Pokeen
Drugs
(b) Government
Denikh
(c) Society
Figure 14: Sub-ontologies for Wikipedia generated by Hearst, centred various topics.
25

---

Somporal
aa Elections
Game Compuser neerin
Centensorio
Wedcl cotuan
Excormarill meDhendiis
Algetra
Socil yelpos
Corputaiinall nuzhehlli
Wescch miid
Ocer thayy
Spacchress
(b) Money
Vocil mort
Orondid vatoreto
Conbiraria
(c) Vocal music
Figure 15: Sub-ontologies for Wikipedia generated by REBEL, centred on various topics.
26

---

Thaze and Comsiios
Trak Sertancing
daa
PRICSE Fisowe
Enfecament
Parldmrent
Denections Rehawatioooon
(a) Criminal Justice System
People
Danish Politics
Greenland
Danish Govemment:
Atfietes
topic classifications
Darish
University Capenhagen
Political History
Govermment and Politics
Danish History
Copenhagen
Regions
(b) Denmark
Supervised Learnnng
Computer Science
Deep Learinng
Machine Learinng
Main topic classifications
Natural Language Processing
Neural Networks
Pattern Recognition
(c) Machine Learning
Figure 16: Sub-ontologies for Wikipedia generated by Zero-shot, centred on various topics.
27

---

World Championships
Track and field
Competitions
Events
Track and field
Sports
Athletics
Main classifications
Countries
Running
Long distance running
Football
(a) Athletics
Criminal law
Nationality law
International law
Administrative law
law
Legal studies
Main topic classifications
Comparative law
Legal history
Constitutional
Jurisprudence
(b) Legal studies
physiology
Muscle physiology
Cardiovasculat system
Main topic classifications Science
Biology
Physiology
Metabolism
Respiratory system
Neurophysiology
(c) Physiology
Figure 17: Sub-ontologies for Wikipedia generated by One-shot, centred various topics.
28

---

Space technology
Satellites
Space exploration
Technology
Aerospace technology
topic classifications
Spacecraft
Rocket technology
(a) Aerospace technology
Natural language procossing
Speeehs recogrition
Technology
Neural networks
leaming
Artindal intelligence and machine learning
tapic
Computervision
Krowledge essentaation
Cnowledge representationn
(b) Artificial intelligence and machine learning
Campaiga
by spelecton
Ugi poliianiooloooo
Aarreta
Worg solars
(kited Suits
Mo hode casollatioo
Peotheia clattors
States
hane
aolleaatoo anc.
Wet tureet
Weggreteh
Fowirid eistves
(c) Elections
Figure 18: Sub-ontologies for Wikipedia generated by Three-shot, centred on various topics.
29

---

A.4.2 arXiv
Computational France
Ganeral EcunonEcs
Goneral Finance
Muthematical Frierce
Managoment
Quantrative Satigiio
Pricing of Secutttes
Loga Agebraic Topelogy
ca cal Systems
Group Theory
Trdding and Mooestmuture
Opimppaiion
Other Stotitties
Spectri Theao)
humotical
Metrooology/
of Ceometyy
Pyliitum Agueii
Gedmetric Epprogy
History Stabitica Theory "Meir sutrrpe dwwteiii
Learning
Classrod Analysts and DEES Number
Operater Alpebrez
Appllcations
Fundiional Belliieee Geometry
Camplex Vo lastes
Re geeeee Hasjf POE:
Cur purpptation
Genera Topelogy
Prbsiility
Other Quandiative ve Bickay/
Commutative Aigebra
<Theary
Informtionn Theory
Quarttative Brgeyy Natoorts
Sromelecdees
Subcediular
Neurons and Cogsttion
Tisues and Organs
Behavior
Qua nittttttvv Nethods
Figure 19: Ground truth test split ontology for arXiv
30

---

Information
Information Tochnology
Equations
Systems Theory
Mathematical Systems Control
Information Systems Theor Information
Society
Mathematical Logic
Neuroscience
Differential Equations
Mathematics
Biological
Information Systems
Mothematical Logic and Foundetions
enttccia//maalligen
Life Sciences
Mathematical Physics
Biological Data Science
Maehine Learning
Analya topic classifications
Biological Evolution
Eleccical Engineering and Systems Science
Information Theory
Sigrbll Processing
Economics
Physics
Econamics and Ananco
Data Analysis
Quantum Physics
Time Series Analysis
Computational Physics
Statf Inference
Data Miring
Shatistical Mechanics
Figure 20: Ontology for arXiv generated by OLLM
Information Theootrrorrmttion
Biological Sciences
Biology
Economics and Triediconomiics
Life Sciences
Computational Physics
Biological Physics
Economy
Methods
Linguistics
Main togic classifications
Condersed Matter
Physics étatistical Mechanics
Anthropology
Humanities
Quantum Physies
Artificial Intelligence
Optics
Nonlinear Scien Bluid Dynamics
Medicel
Figure 21: Ontology for arXiv generated by Finetune
31

---

Image Poceuinn
quantan Mabi
Mecieraa canese Weylls
Maciicaa bnginesting and kystems
Approdduutiiiyyy condecced MARI
Phpaics
Sigill froouaing
Cewer Relatviyy Doart Rearez Spp
Stcket
Maturaiica Pipla
Eeaaged Pipia
Nentrau Sorrot
aiganiiy Pplics 04
Projica
Mateor Eeprment
Dengy Plysio Peepptaiiee//tppoo
Equeeloy and Mor plact tomaleskk
Dygine Astoph dil Thenommnng
Methis dorulayks
Figure 22: Ontology for arXiv generated by Memorisation
32

---

Genomics
Economics
Neurors and Cognition
Quantitative
Mathemoniss
Information Theory
Ganegory Theory
Portfolii Meneggemen
Rings Algebras
Statistics
Aiomeloss des
Compuration
Number Theory
Mesternn rel
Tirance
k-Theory and Homology
Model
Networks
Differential Geomeety
Clpc
Analysis of PDEs
Algebraic Geornety
Risk Manegemggeretoor Algebras
General Topology
Matheinatie'' Metric Gearnetry
Quantom Algebra
Commutative Algebra
Trading Market Mcrostrcture
Machinalgnnree Tapalar
Gresn
Statiates
Spectral Theory
Subcellular Processes
Computatiooa
Symplectic Geometry
Sptrance
Matherattial Physics
enicing
Statistical peicce
Camplex Variables
mobabbiity
Optimization Contoo
Ri Represertaton
Other Statistics
Populations and Evolution
Qunitiaaiie Biology
General Finandappiicaions
Combinatoric
Numerical Analysis
Mein tupic clessifications
Functional Analysis
History
Other Quartitative Biology
Mothodology
Figure 23: Ontology for arXiv generated by Hearst
33

---

clacsincatinn
Sratstics
Molecular Networks
Spectral Theory
Combinstorics
2ieimmatiaal Physlys
Quaniuu Alpcbri
Economics
Nourans and Cognition
Computational france
other Quantrative B geaweer
B oonolecules
Pricing of Securites
Succellutor Processes
Popuittions and Evotution
Geromics
Other Statinlis
Optimisation and Canted
informat Theory
Statistical Rnonce
Mathgmatical Bnance
Sympletic Gesrintty
Aurainttal Andlys
Toding Misrosaani gane
Naclina Leamiing
Classical Ana
Muthotology
France
Rosbwiility
Motric Goomatry
Algebras
Wathemagis
Catedory Theory
oysmitttive Regnce
Portfolio Management
Gave winl Topulioy.
Theory
Binga und Nigebren
Compotation
WeTwwgy and Hamm bcometry
Ipprosertation Thwory
Quantitative Sollgy
POT Tocology
History and Dvendtew
Genmral Topakay
Logc
Dynamica Systems
Group
Functional Andlysis Complex Commutative Algebrs
App copicaaions
Figure 24: Ontology for arXiv generated by REBEL
Artificial Intelligence
Runctional Analysis
Adtifiiial Melliaange and Machine Learning
Computer Science
Analysis
Physics
Machine Learnlegtain topic classifications
Optimization
Mathematics
aaadd Deep Learnigähine Learning Lnd Lęaming Artificial Intelligence Probability Theory and Statistics
Al
Neural Networks
Communications Technology
Theoretical Marhhppiedioaathemmaiic
Figure 25: Ontology for arXiv generated by Zero-shot
34

---

Artificial Intelligence and Machine Leaming
Probability Theory Statistics
Science
Optimization
Machine Learning and Al
Analysis
Deep Learning
Computer Science
MathernaticsFunciionall Analysis
Main topic classifications
Machine Leaminng
Artificial Intelligence
Mathematical Physics Mathhmaaaati Analysis
Neural Networks
Applied Mathematics
Physics
Theoretical Physics
Electrical Engineerir@ngineering
Figure 26: Ontology for arXiv generated by One-shot
Engineering Technology
Probability and Statistics
Machine Learning
Physics
Deep Learing
Machine Leaming Artificial Intelligence
Mathematical Physics
Mathematics
Main topic classifications
Machine Learning and Al Computer Sclence
Applied Mathematics
Engineering Attiiccal Intelligence and Machine Leaming
Analysis
Artificial Intelligence
Electrical Engineering Systems
Figure 27: Ontology for arx generated by Three-shot
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims of this paper are: 1. OLLM is effective method for building ontologies from scratch, and 2. the evaluation metrics we introduce are robust and useful for gold standard evaluation of ontologies. We justify the first claim by demonstrating that OLLM outperforms our baseline methods on Wikipedia and arXiv according to our metrics. We justify the second claim by showing that existing metric (Literal F1) score the sub-optimal Memorisation solution highly, while our metrics are not subject to this issue. Our metrics also suggest results that align with qualitative analysis.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations and the possible resolutions in Section 6. This includes the scope of the work, where we only study ontologies with concepts and taxonomic relations, the inability guarantee taxonomic relation transitivity, and the inability control for data leakage from the pretraining stage of the LLM base model.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
36

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We describe the data collection procedure in Section 4. and the full experiment details in Appendix 1. We also include the code and dataset in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
37

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The code and data used for this project are provided in the supplementary material. The code includes README which details the steps for reproducing our results.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Sewiidde/CoodSuumiiise wweeiisssooppooocooooo for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/ cc/ppblic/guu wtwp//gg/////eeee/eeeeeee/.eeeee /odee///ooedbmiii ss aasionPooi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We give all the experimental details in Appendix A.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We did not perform repeated experiments due to compute constraints, therefore there are error bars.
Guidelines:
38

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe the compute requirements for each experiment in Appendix A.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: h/ppprr////urips// cc.ppp.cc......c.oooooo sc/pubbic// httteee/ttiiiiieeeeeeeeeeeeeeeae ines?
Answer: [Yes]
Justification: This paper does not involve human subjects and does not use data that is not already in the public domain. We clearly describe our data collection procedure in Section 4.1. Our method is highly specialised in building ontologies and solving related tasks, thus having minimal socictal impact or any potentially harmful consequences.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
39

---

Answer: [NA]
Justification: Our method is highly specialised in building ontologies and solving related tasks only. We do expect our work to have wider impact than improving the quality of existing or new ontologies.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks. The trained model specific to building ontologies only.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The two datasets collected in this project use data from Wikipedia and arviv, which are the public domain under the CC BY-SA 4.0 and CCO 1.0 Deed licenses respectively. The REBEL-large model is available under CC BY-NC-SA 4.0 license.
40

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release the dataset and code for this paper. The data collection procedure is clearly described in Section 4.1. The training procedure clearly described Appendix A.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
41

---

Answer:
[NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
42


Reviews:
[{"review_id": "3jvObdLDsV", "reviewer": "Reviewer_Qo2b", "summary": "The paper aims to address the challenge of constructing ontologies, which traditionally require substantial manual effort. Ontologies are structured representations of domain knowledge used for automatic machine processing. Previous methods for ontology learning (OL) using large language models (LLMs) focused on solving individual subtasks, but this approach failed to capture the interactions between these subtasks. Experimental results on Wikipedia demonstrate that the authors' proposed approach, called OLLM, outperforms traditional subtask composition methods, producing more semantically accurate ontologies while maintaining structural integrity. The model also shows effective adaptation to new domains, such as arXiv, with only a small number of training examples, indicating its scalability and domain-independence.", "questions": "Given that there are so many ontologies and corpora out there, why only limit to these two very well-known datasets? I feel that the approach could have been better evaluated with multiple ontologies, some of which are unusual and hence not captured as well by LLMs.", "limitations": "The study focuses on constructing simple ontologies that include only concepts and taxonomic relations. To extend OLLM to produce non-taxonomic relations, tags indicating the relation type could be added to each edge during the linearization of subgraphs for sequence modeling. New evaluation metrics may also be needed to handle multiple types of relations. Another limitation is that the taxonomic relations in the generated ontologies are not always transitive due to the presence of cycles, a common issue in many ontology learning methods. There are existing algorithms for cycle removal to clean hierarchies. Additionally, the study could not fully control for data contamination since the pretraining dataset of Mistral 7B is not publicly known. However, the generated ontologies were found to be sufficiently different from the ground truth, suggesting that OLLM does not directly remember samples from its pretraining stage.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This is an interesting problem, and could serve as the foundation for important neurosymbolic applications in the future. \n\nThe authors introduce OLLM, a general and scalable method that builds ontologies from scratch by fine-tuning an LLM. Instead of focusing on individual relations between entities, OLLM models entire subcomponents of the target ontology, reducing overfitting on high-frequency concepts through a custom regularizer. The principles of the approach are moderately innovative. \n\nThe paper also introduces new metrics for evaluating the quality of the generated ontology, measuring its semantic and structural similarity to the ground truth. This is laudable, but in going through the metrics, I noted some significant potential flaws that I note below.", "weaknesses": "I would question the validity of the  fuzzy F1 metric. Using embedding similarity is not a good fit here, as for difficult samples, it would likely be wrong. There is also bias that comes from pretraining (so the metric can't really be used in 'unusual' domains for which such embeddings are not available; but that's where the true applications of this proposal would be!) The gap between the literal and fuzzy F1 in the performance table should have been a warning to the authors. \n\nThe experiments could have been much stronger. A more difficult benchmark than either of the ones the authors selected would have been more valuable for assessing the true potential or limitations of the method. \n\nThe baselines could similarly have been stronger. It doesn't look like the authors considered some form of sophisticated (e.g., chain of thought) prompting, which could have made a big difference here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback.\n\n> Using embedding similarity is not a good fit here, as for difficult samples, it would likely be wrong. There is also bias that comes from pretraining (so the metric can't really be used in 'unusual' domains for which such embeddings are not available; but that's where the true applications of this proposal would be!)\n\nFor the tasks considered in this paper, we did not observe instances where concepts with clearly different meanings were given similar embeddings. This can be credited to the quality of the pretrained embedding model and relatively common concepts present in Wikipedia and arXiv. Note that the pretraining of the embeddings model aims to cover every domain and to find the best generalising embeddings. We expect such embedding to remain informative even when applied out-of-distribution. \n\nWe agree that for more exotic domains (e.g., protein ontology), a generic embedding model like the one used in this paper is unlikely to give accurate results. However, we argue that the evaluation framework proposed in this paper is still applicable as long as a more specialised embedder is available. We believe applications with accurate embeddings are much more prevalent than those with accurate ontologies.\n\n> The gap between the literal and fuzzy F1 in the performance table should have been a warning to the authors.\n\nWe take this as an argument _for_ using embeddings. The cases where Literal and Fuzzy F1 disagree the most (e.g., when Literal F1 is comparatively high but Fuzzy F1 is comparatively low) are on methods that show signs of overfitting, particularly Memorisation and Finetuning. We observe Literal F1 has a strong bias towards methods that memorise the training set. This is because literal F1 is sensitive to semantically insignificant syntax differences such as casing and word form, whereas Fuzzy F1 (as well as Continuous F1 and Graph F1) are generally agnostic to syntax.\n\nWe also emphasise that we do not claim that any single metric can truthfully reflect the quality of an ontology. Given the many aspects of what constitutes a good ontology, it is an oversimplification to summarise the performance with a single value. We thus use multiple metrics to get a holistic understanding of the output. From this perspective, it is not surprising that different metrics will arrive at different conclusions, otherwise it would not have been useful to use multiple metrics in the first place.\n\n> The experiments could have been much stronger. A more difficult benchmark than either of the ones the authors selected would have been more valuable for assessing the true potential or limitations of the method.\n\nWe chose Wikipedia as our main benchmark as it covers a wide range of topics, so many specific domains of interest can likely be found as a subgraph of the Wikipedia ontology. We believe the diversity of topics makes it more challenging than specialised ontologies like WordNet (focusing on word-level relations) or GeoNames (focusing on Geography). We would like to know whether the reviewer has any particular benchmark in mind that they believe would be a strong addition to the paper. We are open to including more benchmarks if necessary.\n\n> The baselines could similarly have been stronger. It doesn't look like the authors considered some form of sophisticated (e.g., chain of thought) prompting, which could have made a big difference here.\n\nWe designed our prompting baselines to study the merits and failure modes of using LLMs to build ontologies out-of-the-box. The main weakness of zero-shot prompting (as discussed in the paper) is the inability to produce ontologies that are structurally similar to the target. Many prompting techniques, such as chain-of-thought, focus on improving reasoning and logic (e.g., the original CoT paper) which do not appear to be the main bottleneck in ontology learning. \n\nWe nonetheless thank the reviewer for this suggestion and have included an additional ablation baseline using a more deliberate prompting method (inspired by zero-shot CoT [1]). A detailed description of the experiment setup can be found in the general response. The results support our hypothesis above, showing no significant improvement over basic zero-shot prompting.\n\n> Given that there are so many ontologies and corpora out there, why only limit to these two very well-known datasets? I feel that the approach could have been better evaluated with multiple ontologies, some of which are unusual and hence not captured as well by LLMs.\n\nWe designed OLLM to be very general and demonstrated its effectiveness on Wikipedia as a proof-of-concept, and on a different dataset (arXiv) to show its generalisation. We believe they are sufficient in answering our primary research question “How can LLMs be used effectively and scalably for OL?”. While we agree that it would be useful to study which domains are well-captured by LLMs, it is beyond the scope of this paper.\n\nHowever, as mentioned above, we are open to suggestions from the reviewer if they think a particular dataset would be a strong addition to the paper.\n\n---\n\nWould the reviewer please consider raising the score if we have addressed their concerns?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NSBof03QIh", "reviewer": "Reviewer_cxnM", "summary": "The paper introduces a novel method called OLLM (Ontology Learning with Large Models) for automating the construction of ontologies using large language models (LLMs). Ontologies are crucial for structuring domain-specific knowledge, but their manual construction is labor-intensive. The authors aim to address the limitations of partial ontology learning approaches by proposing an end-to-end method to build ontologies from scratch.\n\nThis paper demonstrates OLLM's effectiveness through quantitative and qualitative results on Wikipedia data, showing that it outperforms subtask composition methods in producing more semantically accurate ontologies while maintaining structural integrity. Additionally, OLLM's adaptability to new domains like arXiv is showcased, requiring only a small number of training examples for effective adaptation.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1) The authors propose a custom regularizer that reweights concepts based on their frequency of occurrence, which helps to mitigate overfitting on common concepts and enhances the model's ability to generalize to new, unseen data.\n2) The paper introduces a novel suite of evaluation metrics that use deep learning techniques to measure semantic and structural similarity between ontologies. These metrics provide a more robust and nuanced assessment compared to traditional methods.\n3) The authors demonstrate the effectiveness of OLLM through comprehensive experiments on Wikipedia and arXiv datasets. The results show that OLLM outperforms existing subtask-based methods, indicating its practical utility.", "weaknesses": "1) The paper focuses on building ontologies with concepts and taxonomic relations, which is only a part of the full spectrum of ontology components. A comprehensive ontology includes not only hierarchical relationships but also properties, semantic relations, constraints, and other elements that provide a richer and more explicit semantic content. The paper does not address how these additional aspects are captured and integrated into the learned ontologies.\n2) The paper uses Wikipedia categories and arXiv taxonomy as the basis for ontology learning. However, these structures are more accurately described as taxonomies or folsonomies rather than full-fledged ontologies, which can lead to confusion about the capabilities and outputs of the proposed method.\n3) The paper does not provide a clear and detailed explanation of the end-to-end ontology learning model's training process. Key aspects such as the construction of the training dataset, filtering of irrelevant categories, and the specific model architecture and training regimen are not thoroughly described. For example, it is not clear how the model deals with Wikipedia categories that have little semantic content but are used for page management.\n4) The paper does not include a comparative analysis with other ontology learning methods, such as the LLMs4OL (LLMs4OL: Large Language Models for Ontology Learning, https://arxiv.org/abs/2307.16648.). Without such comparisons, it is difficult to assess the relative advantages and disadvantages of the proposed method in terms of effectiveness, efficiency, and scalability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback.\n\n> The paper focuses on building ontologies with concepts and taxonomic relations, which is only a part of the full spectrum of ontology components. A comprehensive ontology includes not only hierarchical relationships but also properties, semantic relations, constraints, and other elements that provide a richer and more explicit semantic content. The paper does not address how these additional aspects are captured and integrated into the learned ontologies.\n> The paper uses Wikipedia categories and arXiv taxonomy as the basis for ontology learning. However, these structures are more accurately described as taxonomies or folsonomies rather than full-fledged ontologies, which can lead to confusion about the capabilities and outputs of the proposed method.\n\nWe agree that our method only focuses on the basic components of an ontology, and such structures are perhaps better described as taxonomies. We thank the reviewer for raising this point and will refrain from phrases like “solving the full task of building an ontology” (L7) in the final version of the paper for better clarity. \n\nThe main contribution of this paper is to use LLMs to construct basic ontologies end-to-end as one task, and doing so yields more accurate concepts and taxonomic relations. We view more complex aspects of ontologies as extensions and thus beyond the scope of this paper. We hope the reviewer can view our achievements in comparison to existing methods for modelling concepts and taxonomic relations, and agree that improving the quality of the fundamental components of an ontology is a sufficient contribution already.\n\n> The paper does not provide a clear and detailed explanation of the end-to-end ontology learning model's training process. Key aspects such as the construction of the training dataset, filtering of irrelevant categories, and the specific model architecture and training regimen are not thoroughly described.\n\nIn Section 4.1, we give a detailed description of how we construct our datasets, including the process of obtaining the raw data and how we construct the data splits. Section 3.1 describes how we obtain document-subgraph pairs from the source data. We did not perform any additional filtering of categories, hence not such mention in the paper. Due to space limitations, we did not include all implementation details. We will consider rewording relevant parts of the paper for better clarity and include more details in the appendix. \n\nSimilarly, the specific model architecture and training details are broadly mentioned in the main text. As described in Section 5.1, OLLM is a LoRA finetune of Mistral 7B which has its architecture described in [5]. We also included a detailed description of all the LoRA and training hyperparameters in Appendix A.1.1 to aid reproducibility.\n\nWe also share our dataset, model and code (including dataset construction and training), which hopefully resolves all ambiguities.\n\n> For example, it is not clear how the model deals with Wikipedia categories that have little semantic content but are used for page management.\n\nWe did not apply any filtering to the source data, hence our model will learn to construct and organise every concept in the dataset as part of at subgraph modelling stage. We decided to not apply any filtering to the source data to minimise external bias on how the data “should” look like. We note that it is often not clear-cut whether a Wikipedia category is just for page management: for example, the Wikipedia categories of the form “Lists of [subject]” refer to the special type of articles where the main body is a bullet point/table listing of the subject, which is a useful concept in the Wikipedia domain.\n\n> The paper does not include a comparative analysis with other ontology learning methods, such as the LLMs4OL (LLMs4OL: Large Language Models for Ontology Learning, https://arxiv.org/abs/2307.16648.).\n\nWe thank the reviewer for their suggestion and have run additional ablations based on LLMs4OL. The full experiment setup can be found in the general response. Here, we summarise the results:\n\n1. The link prediction-based approach studied in LLMs4OL suffers from serious scalability bottlenecks. In particular, it requires $O(n^2)$ inferences for link prediction which is impractical for large models and/or large number of concepts. This is the reason why this baseline was not used originally. For the ablation, we made ad-hoc fixes to the method to make it manageable for our large problem sizes (see general response).\n2. The results show that it is generally worse than our zero-shot prompting method based on subgraph modelling. \nWe will include the above discussion points regarding LLMs4OL in the final version of the paper.\n\n---\n\nWould the reviewer please consider raising the score if we have addressed their concerns?\n\n[5]: Jiang, Albert Q., et al. \"Mistral 7B.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uayn4Yi2cI", "reviewer": "Reviewer_3YVJ", "summary": "The paper is well-written, addresses a valuable real-world problem, and proposes a simple, intuitive, and effective method. The experiments are rigorously designed with comprehensive evaluation metrics. Overall, this is a high-quality work with notable contributions.\n\nHowever, there are concerns about whether the improved performance is due to end-to-end modeling or LLM capabilities, the assumption that each document is associated with at least one concept, the consistency of the generated ontology, and the post-processing procedures.", "questions": "Please refer to W1-W5.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- S1: The paper is well-written and easy to follow\n- S2: The investigated ontology learning problem is of great value for real-world applications\n- S3: The proposed method is simple, intuitive, yet problem-oriented, novel, and effective\n- S4: The experiments are well-designed, in particular, the baselines and the data-splitting strategy are rigorously\n- S5: The set of evaluation metrics is comprehensive and covers various aspects of performance", "weaknesses": "- W1: The paper's main claim is that end-to-end modeling is better than pipelined methods. However, it is unclear whether the improved performance is credited to the end-to-end modeling approach or the capabilities of LLMs. Although I am convinced that end-to-end + LLM is a good solution, I am curious if applying LLMs for the subtasks in the pipeline will work even better, especially considering LLMs are good at decomposing problems and solving them step-by-step.\n- W2: The authors assumed each document is associated with at least one concept in the ground truth ontology. However, in real-world applications, it would be very possible that the whole document does not contain any relevant edges or even concepts. Can the proposed OLLM say no if a random irrelevant document is given?\n- W3: Consistency is a crucial property of ontologies, which is the prerequisite to run most logical reasoners. Is the generated ontology guaranteed to be consistent?\n- W4: I am concerned about the post-processing procedure. Are there any entity (concept) resolution mechanisms involved to merge semantically equivalent concepts? For example, if “English” and “english” are both there, are then treated as the same concept? And for “English Language” and “The Language Named English”, are they merged?\n- W5: Why using 3 prompts is generally worse than 1-shot?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback.\n\n> W1: The paper's main claim is that end-to-end modelling is better than pipelined methods. However, it is unclear whether the improved performance is credited to the end-to-end modelling approach or the capabilities of LLMs. Although I am convinced that end-to-end + LLM is a good solution, I am curious if applying LLMs for the subtasks in the pipeline will work even better, especially considering LLMs are good at decomposing problems and solving them step-by-step.\n\nWe thank the reviewer for their suggestion and have run additional ablations based on applying LLMs for subtasks. The full experiment setup can be found in the general response. Here, we summarise the results:\n1. The concept discovery + link prediction-based approach (studied in LLMs4OL [2]) suffers from serious scalability bottlenecks. In particular, it requires $O(n^2)$ inferences for link prediction, which is impractical for large models and/or large number of concepts. This is the reason why this baseline was not used originally. For the ablation, we made ad-hoc fixes to the method to make it manageable for our large problem sizes (see general response).\n2. The results show that it is generally worse than our zero-shot prompting method based on subgraph modelling. \n\nWe believe this is further evidence that our subgraph modelling method is a good solution.\n\n> W2: The authors assumed each document is associated with at least one concept in the ground truth ontology. However, in real-world applications, it would be very possible that the whole document does not contain any relevant edges or even concepts. Can the proposed OLLM say no if a random irrelevant document is given?\n\nWe do not view this assumption as a fundamental limit to OLLM. One simple modification we can make is to include empty targets in the training set such that the model learns to output an empty graph if the document does not contain any relevant concepts. We made this assumption as it applies to our datasets and simplifies the implementation.\n\n> W3: Consistency is a crucial property of ontologies, which is the prerequisite to run most logical reasoners. Is the generated ontology guaranteed to be consistent?\n\nEnsuring consistency is a challenge for many OL methods and is not specific to OLLM. There exist generic post-processing methods that can guarantee consistency [6]. We performed further analysis of the generated output by OLLM and found that they are near-consistent: the output ontology for Wikipedia consists only of 97 simple cycles and no cycles were found in arXiv. Using the greedy algorithm of repeatedly removing the edge that breaks the most simple cycles (a heuristic to the smallest set of edges whose removal makes the graph acyclic), we prune all such cycles and make the ontology consistent by removing just 26 out of 10414 edges. This is quite surprising considering we did not explicitly optimise our model to satisfy consistency.\n\n\n> W4: I am concerned about the post-processing procedure. Are there any entity (concept) resolution mechanisms involved to merge semantically equivalent concepts? For example, if “English” and “english” are both there, are then treated as the same concept? And for “English Language” and “The Language Named English”, are they merged?\n\nWe do not perform any concept-merging in our post-processing steps. While we agree that applying such normalisation strategies can produce better ontologies, we do not think it contributes to answering the research question of “Does OLLM produce better ontologies?”. Instead, we decided to restrain the post-processing steps to be simplistic and minimal without relying on many heuristics. This allows us to attribute any differences in output quality to the method itself without being confounded by interactions with the post-processing procedure.\n\n\n> W5: Why using 3 prompts is generally worse than 1-shot?\n\nWe thank the reviewer for raising such concerns. In response, we did a short investigation focusing on arXiv (as that’s where 3-shot seemed to perform worse than 1-shot). We found that 3-shot appears to generate unnecessarily long responses, which we hypothesise leads to noisier outputs. Across the training set, the ground truth subgraph targets contain a median of 4.0 concepts (IQR: 4.0). In contrast, 1-shot predicts a median of 9.0 (IQR: 8.0) and 3-shot predicts a median of 11.0 (IQR: 9.0). We see that giving more examples tends to result in longer responses, which might be an inherent bias of Mistral 7B.\n\nWe note that the performance gap between 1-shot and 3-shot is generally insignificant so we do not believe the claim “3-shot is generally worse than 1-shot” is fully justified.\n\n---\n\nWould the reviewer please consider raising the score if we have addressed their concerns?\n\n[6]: Sun, Jiankai, et al. \"Breaking cycles in noisy hierarchies.\" Proceedings of the 2017 ACM on Web Science Conference. 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NYizVmQtT7", "reviewer": "Reviewer_Yksm", "summary": "This paper proposes a new methodology of Ontology Learning using Large Language Models and ontology paths which is called OLLM. After introducing two ontology-datasets for Wikipedia and ArXiv, they train an LLM using the titles, summaries, and ontology paths to allow for sub-graph generation. The method is additionally improved using post-processing techniques for summing and pruning the subgraphs with the purpose of building a complete ontology. New metrics are also presented for facilitating the comparison of the generated ontology with the ground truth and other baseline techniques.", "questions": "•\tUsually training sets consist of the biggest part of a dataset than validation and test sets. Why did you choose such small portion of the datasets for training? Could that be one reason for the initial poor generalisation? \n\n•\tIt is also unclear how much the overlapping parts of training/validation/test sets contribute to the final result.", "limitations": "The main limitation, to my opinion, is that OLLM cannot generate a complete ontology for a different domain without already having an initial ontology for extracting the paths first. This means that there is still the need of reusing or constructing some part of an ontology.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "•\tThis work presents a novel technique leveraging the power of LLMs and ontology paths for ontology learning.\n\n•\tTwo different ontology-datasets (Wikipedia and ArXiv) are constructed and used for the experiments.\n\n•\tNew metrics for ontology evaluation are proposed based on semantic similarity. \n\n•\tOLLM seems to outperform most of the baseline methods in the experiments.\n\n•\tOverall, the paper is well-written.", "weaknesses": "•\tWhile the OLLM results are better than most of the baseline methods that is not the case using the motif distance metric (Wikipedia: Finetune = 0.05 while OLLM = 0.080, Arxiv: Memorisation = 0.037 while OLLM = 0.097). That is not well explained in the paper.\n\n•\tPerformance seems to drop significantly when using a smaller dataset such as arXiv.\n\n•\tIt is unclear whether OLLM can generalize well to different domains without having an initial ontology for extracting the paths.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback.\n\n> While the OLLM results are better than most of the baseline methods that is not the case using the motif distance metric (Wikipedia: Finetune = 0.05 while OLLM = 0.080, Arxiv: Memorisation = 0.037 while OLLM = 0.097). That is not well explained in the paper.\n\nWe thank the reviewer for raising this point. As the data splits are constructed symmetrically (i.e., the train, validation, and test splits have the same distribution), we expect all data splits to have similar structures. This implies that Motif Distance is biased towards overfitting, since the structure of the train and test splits tend to be similar, even though the represented concepts are different. This explains why methods that tend to overfit (i.e., Memorisation and Finetune) achieve the best scores. We will include this discussion in Section 5.3 in the final version.\n\n> Performance seems to drop significantly when using a smaller dataset such as arXiv.\n\nWe would like to emphasise that the two experiments are designed to answer different questions and thus the results are not suitable for direct comparison. We use Wikipedia to test the model’s ability to generalise in-domain, whereas the arXiv experiment tests the transferability using only a small number of training examples. The arXiv task also appears to be more difficult in general, as indicated by the worse performance of the baselines.\n\n> It is unclear whether OLLM can generalize well to different domains without having an initial ontology for extracting the paths.\n\nWe agree that training examples are needed to give the best performance. Ontologies come in different “styles” (e.g., the granularity of concepts) which is difficult to specify without training examples. However, we argue that our end-to-end modelling approach is useful in other domains:\n\n1. In the arXiv experiment, we only used 2000 finetuning examples to transfer from Wikipedia to arXiv and achieved good performance. This is a modest cost if one were to apply our model to a new domain.\n2. Even without training examples, our 0/1/3-shot prompting method outperforms existing baselines (including the new ablation methods, see general response).\n\n> Usually training sets consist of the biggest part of a dataset than validation and test sets. Why did you choose such small portion of the datasets for training? Could that be one reason for the initial poor generalisation?\n\nWe deliberately chose to use a larger test split as we wanted to ensure that sufficiently many concepts and relations were unseen in the training split. This tests the model’s ability to generalise in-domain. Using a larger split for evaluation also helps to get more reliable metrics, as we did not do repeated runs.\n\nAs described in Section 5.1, the fundamental cause of poor generalisation of direct finetuning is due to the imbalanced sampling of high and low-level concepts. This results in the model learning high-level concepts much faster, resulting in overfitting high-level concepts while underfitting low-level ones. Increasing the training set size does not solve such issue.\n\n> It is also unclear how much the overlapping parts of training/validation/test sets contribute to the final result.\n\nIt is inevitable and desirable that there is an overlap between the data splits. We attempt to quantify the contribution of such overlap to the final result by introducing the “Memorisation” baseline method, which demonstrates the quality of resultant ontology if one were to blindly exploit such overlap. The fact that Memorisation performs poorly suggests that the overlaps do not contribute significantly to the final result.\n\n---\n\nWould the reviewer please consider raising the score if we have addressed their concerns?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their time. The feedback is constructive and helpful.

We are happy to see that most reviewers found the end-to-end OL task interesting, and agree that our core contribution, OLLM, is a novel approach to leveraging LLMs for OL. Reviewers also agree that the paper is well-written.

Most of the reviewers’ concerns revolve around our experiment results and evaluation strategy, such as inconsistent ranking among different metrics, and insufficient baselines. We thank the reviewers for raising such points and in response have performed additional ablations and analysis. We give a summary of our new results here. Further discussions can be found in our response to each reviewer in the context of their concerns. 

# Consistency

Reviewer 3YVJ questioned the consistency of the output ontology. For example, it does not guarantee an anti-symmetric relation under transitivity (i.e., there may be cycles). We analysed the ontology generated by OLLM and found that they are almost consistent, with only 97 simple cycles in Wikipedia and no cycles in arXiv. All the cycles in Wikipedia can be broken if we remove just 26 of the 10414 edges. This is quite surprising considering we did not explicitly optimise our model to satisfy consistency.

# Ablations

See supplementary pdf for full metrics.

## Chain of thought (CoT)

**Motivation**: Reviewer Qo2b suggested that a stronger baseline can be established if we employ more sophisticated prompting techniques, such as CoT.

**Method**: Prediction involves two rounds of inference: In the first round, we ask the model (Mistral 7B instruct) to describe the possible relevant concepts for the input document and to explain its reasoning. Then, we ask the model to predict the subgraph in the specified format given the additional, self-generated context.

**Result**: We tested the CoT method on Wikipedia and found no significant difference from basic zero-shot prompting.

**Interpretation**: Most advanced prompting techniques, including CoT, primarily aim to improve logic and reasoning. We hypothesise that the performance in OL is more dependent on the model’s understanding of natural language than its ability to perform multi-step reasoning, hence we do not observe any significant improvement from CoT.
## LLMs4OL

**Motivation**: Reviewer cxnM and 3YVJ both suggested that a LLM-based subtask composition baseline (as studied in LLMs4OL [2]) would be useful for evaluating whether the improvement by OLLM is due to the improved methodology (end-to-end modelling) or simply just because we used LLMs.

**Method**: The subtasks studied in LLMs4OL are concept discovery (given a document, predict its relevant concepts), and link prediction (given two concepts, predict whether they are taxonomically related). Unfortunately, constructing a baseline from such two subtasks is non-trivial. We encountered significant scalability issues in the link prediction stage as it required $O(n^2)$ inferences. We make two modifications to overcome such limitation:
1. After the concept discovery stage, we only discard all but the N most frequent concepts to limit the number of inferences required during link prediction, where N is the number of concepts in the ground truth.
2. Instead of using zero-shot Mistral 7B as the link predictor, we use a finetuned BERT as the link predictor as it runs much faster. Given that [2] demonstrated that finetuned models perform much better than zero-shot inference on link prediction, we expect the finetuned BERT to be at least as good, if not better, than zero-shot Mistral 7B on this subtask.

In summary, the method is as follows:
 1. Use zero-shot Mistral 7B for concept discovery, allowing it to tag more than one concept per document.
 2. Discard all but the top N most common concepts. Manually add the root concept.
 3. Perform link prediction with a finetuned BERT for all concept pairs.
 4. Apply post-processing, as described in section 3.2.
 
We design this baseline such that it is comparable to zero-shot end-to-end modelling: both use zero-shot Mistral 7B as the backbone, just utilised in different ways.

**Result**: We tested this method on Wikipedia and found that it is worse than zero-shot end-to-end modelling on all metrics except Motif Distance.

**Interpretation**: We take this as evidence that our end-to-end modelling approach is a clear improvement over traditional subtask-based OL. Not only does the link prediction-based method suffer from significant scalability bottlenecks for large ontologies, its performance is also worse. The results suggest that we can more effectively and efficiently leverage the capabilities of LLMs beyond just solving subtasks, such as by predicting subgraphs. 

# Final remark
We ask the reviewers to evaluate our contribution in the context of existing methods for using LLMs in OL. Our experiments and ablations (above) suggest that our method is more effective and scalable than traditional LLM-based subtask composition methods (e.g., LLMs4OL). Additionally, in contrast to prior attempts to use LLMs in a more end-to-end manner that relies on qualitative evaluation [3, 4], our evaluation framework is more systematic and quantitative, laying the foundation for more rigorous research in the future. We hope the reviewers will consider raising the score if we have addressed any of their concerns.

[1]: Kojima, Takeshi, et al. "Large language models are zero-shot reasoners."

[2]: Babaei Giglou, Hamed, Jennifer D’Souza, and Sören Auer. "LLMs4OL: Large language models for ontology learning."

[3]: Funk, Maurice, et al. "Towards ontology construction with language models."

[4]: Trajanoska, Milena, Riste Stojanov, and Dimitar Trajanov. "Enhancing knowledge graph construction using large language models."

Meta-review: The paper provides a novel approach for ontology learning using LLMs with a novel regularizer and a novel suite of evaluation metrics. Those were acknowledged as strengths by the reviewers. Concerns in the reviews are the limited expressivity of the learned ontology and the breadth of the evaluation. Some clarifications demanded by the reviewers have been added in the rebuttal stage increasing my confidence in the paper. Overall, the paper presents a novel and interesting approach advancing the state-of-the-art in ontology learning. Therefore, I recommend acceptance.
Decision: Accept (poster)