ORAG: Ontology-Guided Retrieval-Augmented Generation for Theme-Specific Entity Typing
Authors: Jinfeng Xiao
Linyi Ding
James Barry
Mohab Elkaref
Geeth De Mel
Jiawei Han
Venue: COLM (2024)
Keywords: Fine-grained entity typing, theme-specific entity typing, retrieval-augmented generation, ontology enrichment, ontology-based information retrieval
TL;DR: We propose ontology-guided retrieval-augmented generation (ORAG), and show it mitigates LLM hallucinations for theme-specific entity typing.

Abstract: Large language models (LLMs) incorporated with retrieval-augmented generation (RAG) have shown great power in many NLP tasks, including fine-grained entity typing (FET). However, we observe that recent LLMs can easily suffer from hallucinations on highly specialized and fast-evolving themes (e.g., redox-active organic electrode materials), especially in the following cases: (1) unseen entities: an entity never appears in the pre-training corpora of LLMs; and (2) misleading semantics: the context of an entity can potentially mislead an entity typing algorithm if the relevant knowledge is not correctly retrieved and utilized. To address these challenges, this paper proposes an Ontology-Guided Retrieval-Augmented Generation (ORAG) approach that incorporates ontology structures with RAG for the theme-specific entity typing task. ORAG first enriches the label ontology with external knowledge and constructs a structured knowledge unit for each node. Then, it retrieves the relevant nodes by dense passage retrieval and expands the retrieved results based on the ontological structure. In this way, more supporting knowledge will be retrieved within the limited input of LLMs for entity typing. In the evaluation, we construct a dataset with two themes for theme-specific entity typing with a focus on unseen entities and misleading semantics. We observe notable cases of hallucination when vanilla RAG is applied to Llama-3, GPT-3.5, and GPT-4, while ORAG can effectively mitigate such hallucinations and improve the results.

Published as a conference paper at COLM 2024
ORAG: Ontology-Guided Retrieval-Augmenter Generation for Theme-Specific Entity Typing
Jinfeng Xiao* & Linyi Ding"
James Barry, Mohab Elkaref Geeth De Mel IBM Research
University of Illinois at Urbana-Champaige Urbana, IL 61801, USA
Daresbury, England, UK
(jxiao13, 1inn edu
{j sames.barze ,ehrab,moab.... mohab. elkaref @iim geeth. demel@uk ibm. com
Jiawei Han
University of Illinois at Urbana-Chama Chaama-Chaampign Urbana, IL 61801, USA hanj@illinois.co
Abstract
Large language models (LLMs) incorporated with retrieval-augmented generation (RAG) have shown great power many NLP tasks, including fine-grained entity typing (FET). However, we observe that recent LLMs can easily suffer from hallucinations on highly specialized and fast-evolving themes (e.g., redox-active organic electrode materials), especially in the fol- lowing cases: (1) unseen entities: an entity never appears in the pre- traiiraining corpora of LLMs; and (2) misleading semantics: the context of an entity can potentially mislead an entity typing algorithm if the relevant knowledge is not correctly retrieved and utilized. To address these challenges, this paper proposes an Ontology- approach that incorporates ontology structures with RAG for the theme- specific entity typing task. ORAG first enriches the label ontology with external knowledge and constructs a structured knowledge unit for each node. Then, it retrieves the relevant nodes by dense passage retrieval and expands the retrieved results based on the ontological structure. In this way, more supporting knowledge will be retrieved within the limited input of LLMs for entity typing. In the evaluation, we construct a dataset with two themes for theme-specific entity typing with focus on unseen entities and misleading semantics. We observe notable cases of hallucination when vanilla RAG is applied to Llama-3, GPT-3.5, and GPT-4, while ORAG can effectively mitigate such hallucinations and improve the results.
prrrivall-Augmennten Generation (ORAG)
1 Introduction
Eetrieval-augmennte generation (RAG) (Lewis et al. 2020 is effective solution to deal with the limited input text length of LLMs when task heavily relies on an external knowl- edge base like long documents or a large database, referred to retrieval memory. Given an LLM query, RAG first retrieves the relevant text chunks from the retrieval memory and then appends them the query generate answers. This RAG framework has shown its success on various NLP tasks.
Fine-grained entity typing (FET) is the task of classifying entities into detailed types, where RAG can take the large label ontology as the retrieval memory. Existing datasets of FET such as OntoNotes Giillick et al. 22014) and FIGER (Ling & Weld (021) mainly focus on common, general topics in the real world such company, city, and actor, which are easy to understand by LLMs since the context and entity types frequently appear in the pre- tre-training corpora.
Equul contribution

---

Published as a conference paper at COLM 2024
However, observe that directly performing RAG with recent LLMs to zero-shot entity typing can easily lead to hallucinations on highly specialized or fast-evolving themes. This is observed with Llama-3 and GPT-4, which are among the most powerful open-source LLMs and commercial LLMs respectively.
We take passage under a highly specialized theme "redox-active organic electrode ma- terials" as example. The passage discusses the effectiveness of a novel electrode material Cu-TCA compared to a bare electrode. The FET task is to find the type of the "bare electrode" in a given fine-grained label ontology for the theme. The correct type is "working electrode".
Example Passage 1. CV data of Cu-TCA were recorded using Li foil as counter electrode. ... ...(Oiitted)..... The CV curve of a bare electrode (only containing the Al foil) proves that the above-mentioned redox areas originate from the active sites of the Cu-TCA structure.
To reason the correct answer "working electrode", the model should (i) understand each type in the ontology to identify the correct type; and (2) understand the terms in the context such as counter electrode. RAG with GPT-4 gets the wrong answer "aluminum insertion electrode" because it semantically matches the misleading contextual information of the Al foil (which is aluminum electrode) and Cu-TCA (which is an insertion electrode).
Inspired by such cases, we propose the task of theme-specific entity typing. Compared to FET, our task focuses on narrow themes, especially highly specialized or fast-evolving themes, which require expert knowledge to reason the entity types. In this task, the hallu- cinations of LLMs are more likely to happen in: (i) unseen entities: new entities that never appear in the pre-training corpora of LLMs (e.g,, the entity in Figure I); and (ii) misleading semantics: the context can potentially mislead an entity typing algorithm if the relevant knowledge is not correctly retrieved and utilized. Challenges in RAG include (i) lack of expert knowledge to understand the theme-specific label ontology; and (ii) even with large collection of expert knowledge, the ineffective retrieval based on text chunks may miss evidence for reasoning with the limited input length of LLMs.
To tackle the challenges and mitigate LLMs' hallucinations for theme-specific entity typing, we propose the Ontology-guided Retrieval-Augmenter Generation (ORAG I The overall design is outlined in Figure 2 For a better understanding of the types in the ontology, ORAG enriches the theme ontology with expert knowledge from multiple external sources. After the enrichment, each node in the theme ontology becomes a structured knowledge unit containing knowledge-dense sub-unit and semantic-rich sub-unit, which constitutes the structured retrieval memory. For more knowledge-deng information into fixed-sized and incoherent text chunks, we retrieve the knowledge units with dense passage retrieval (DPR) (Karpukhin et al. 2020) and selectively add their sub- units into the LLM's query. For more accurate reasoning, we expand the evidence set of retrieved nodes with their neighbors on ontology. For example, we add the definitions of child nodes to discriminate between the more fine-grained entity types. Finally, ORAG combines the evidence and the context to predict the entity type. In the above Example Passage 1, ORAG successfully predicts the correct entity type "working electrode". A more in-depth explanation of ORÁG's success in this case is provided in Section 5.2
retrieval, instead of dividing
As no publicly available datasets focus on our settings, we construct a dataset of two themes: (i) a battery theme "redox-active organic electrode materials", and (ii) game theme "online posts for Genshin Impact". The former represents highly specialized themes, which usually feature complicated ontologies and terminologies. The latter represents fast-evolving themes, where new entities and knowledge emerge so quickly (i.e., a significant update every six weeks) that the pre-training corpora of publicly available LLMs always lag behind the most recent version of the game.
We compare the entity typing performance of ORAG versus vanilla RAG when applied to current state-of-the-art LLMs on the two themes. We observe that ORAG consistently provides a higher boost to Llama-3, GPT-3.5, and GPT-4. The effectiveness of ORAG is especially prominent in cases of unseen entities and misleading semantics.
Additional resources about this work are available at httpppps//ggtaaa.cccm////a/aaaap
2

---

Published as a conference paper at COLM 2024
advantages non-aqueous electrolytes electroactive organic compounds, novel "all-organic redox flow battery" (AORFB) proposed.....
context
Retrieval Memory
Related Information
Redox flow hattery
Ontology guided Retrieval
Redox flow battery
Hybrid flow battery
Full flow battery
Full flow battery
Záne Flow
Aqueous organic flow battery
Battery
Aqueous organic flow battery
Answer:
Non-aqueous organic flow battery
Non Von-aqueous organic flow battery
Non-aquee Non-aqueous organic flow battery
constructed from theme ontology
Figure 1: Ontology-guided RAG for theme-specific entity typing.
We summarize our contributions follows:
We propose the theme-specific entity typing task, discuss two types of hard cases for LLMs, and construct a dataset of two themes focusing on such hard cases.
We propose ORAG, which performs structured retrieval on knowledge-enrichedd on- tologies, to achieve more knowledge-dens, retrieval and more accurate reasoning.
We experimentally demonstrate that applying ORAG to the theme-specific entity typing task can generate more accurate results and mitigate the hallucinations of LLMs compared to vanilla RAG, without requiring additional labeling efforts.
2 Related Work
2.1 Fine-Grained Entity Typing
Compared to conventional entity typing, the label space of FET usually entails dozens or even hundreds of entity types, which is organized into hierarchical ontology structure. Thus, FET faces the challenges of the large label space and limited accurate annotations. To deal with the data scarcity issue, ontologies and external knowledge are exploited and utilized in previous works. AFET (Ren et al. 2016) leverages the defined type hierarchy to formulate loss functions and integrates them into a unified optimization problem. ALIG- NIE (Huang al. 2022) learns to relate entity types with vocabulary using the ontology and then uses a prompt-based method to enrich the few-shot training samples. SEType (Zhang) et al. 2024 finds more instances for each seen type from an unlabeled corpus using pre- trained language models. OnEFET (Ouyang et al. (023)) enriches the original ontology structure with instances and topics. These methods are need of annotated instances and thus not developed under a zero-shot setting.
For the zero-shot setting, line of work utilizes linking from Wikipedia entries to the entities or types (Choi et al.) 20088). For example, ZOE (Zhou et al.) 2018) uses the taxonomy of Freebase and types the enfity by linking it Wikipedia entries. The assumption that such links exist is a limitation of this line of work. Several LLM-based methods are also proposed for zero-shot FET. OntoType (Komarlu et al. 2023) proposes an aatology-guider framework that leverages pre-trained language models and head words to match the fine fine-grained types to a type ontology. QQn et al. 20023) and Hu et al. (024) apply LLMs to zero-shot entity typing by prompt engineering. These studies focus common entity types, and the assumption that LLMs know the types well limits their applicability to highly specialized or rapidly evolving themes.
3

---

Published as a conference paper at COLM 2024
2.2 Knowledge-Grounded Retrieval-Augmennte Generation
RAG (Lewis et al. 2020) retrieves text chunks from text corpora to augment LLM queries. This technique has shown its power on various NLP tasks (Gao et al. 2023). Besides unstructured text, structured knowledge like knowledge graphs (KGs) and ontologies can also be used as retrieval memory in the RAG framework. KG-RAG (Soman al.) 2023 applies RAG to biomedical KGs to facilitate domain-speciff generation tasks. SURGE Kang et 2023) trains graph neural networks to retrieve knowledge triples from a KG, which answers. KGLM (Logan et al. 2019) selects relevant facts from a KG with a KG embedding in each step of generation.
encourages the model to generate knowledge-grouue
KGQA (Sen et al. 2023) retrieves knowledge triples from KG related to query entities
by Hadamard product to augment the LLM input. REALM (Guu et al. 2020) retrieves
documents from Wikipedia during the pretraining stage. EMAT (Wu et al. 2022) encodes external knowledge into a key-value memory to be retrieved and introduces a memory- augmented transformer for QÅ tasks. While these works attempt to retrieve evidence from KGs or ontologies, most of them ignore or do not fully utilize structured information. How to better utilize structured and unstructured knowledge for FET remains less studied.
3 Method
3.1 Problem Formulation
The conventional FET task mainly focuses on frequent and common entity types in the real world. Examples include company, city, and doctor in the FIGER dataset (Ling & Weld 2021), as well as date, money, and time in the OntoNotes dataset (Gillick et al. 2014). Despite the large label space, the types are easy to understand for LLMs without requiring much additional expert knowledge (Ding et al. 2022). In contrast, our targeted theme-specific entity typing task involves several less-studied challenges. (1) It focuses on narrow themes that need in-depth exploration for specific purposes. (2) It often deals with entities, types, or contexts that are unseen or long-tailed in the pre-training corpora of publicly available pre-trained LLMs. (3) It requires expert knowledge and reasoning on the themes to figure out the answers.
For each theme, we assume its theme ontology (i.e., label hierarchy) is available. As reviewed Section 2.1 the availability of such an ontology is part of the problem definition of works on FET. Our task is to type the entities with the most proper fine-grained labels. Once a fine-grained type is identified, all parent types on the path from the root of the ontology to the fine-grained type are also considered as identified. Therefore, our task belongs to the family of hierarchical multi-label classification. Take the case in Figure I as an example. The theme ontology contains entity types like non-aqueous organic flow baftery, which is the most proper fine-grained type of the underlined entity in the given context. Once this type is identified, all highlighted types in the ontology are considered identified.
3.2 Structured Retrieval Memory Construction
3.2.1 Theme-Specific Ontology Enrichment
For a highly-specialized theme, such as redox-active organic electrode materials, the raw label ontology may not provide sufficient information for reasoning the entity types by LLMs. Directly applying LLMs RAG on the label hierarchy can easily lead to hallucination, especially for knowledge-intensivee contexts and unseen entities since the reasoning would require specialized expert knowledge.
To address the issue related to the lack of specialized information, we first enrich the label ontology with expert knowledge from external sources (Panel 1, Figure 2). For each entity type in the label ontology, we collect the following knowledge from various sources:
i. The concise and formal definition from theme-specific dictionaries encyclopedias can help establish a common understanding of highly specialized terms and avoid ambiguity.
4

---

Published as a conference paper at COLM 2024
Vanilla RAG:
relevant chunks
Redox flow battery a upp of rechargadble battery Unrrecennt internztion)
TE
Chunk
tye rechorpebbee battery
Redoa Bow buttery as
Brndessnt Informaion))
Top-K chunks
Hybröd flow battery rodox Boo battery.
Hylr Dow battery is type of redox fiowe battery. Grndessnt inforastion)
T
Chunk
Irmdesnn Information)
Inpur Langth
Ontology-Guided RAG:
GPT-4
Theme Ontology
Structured Knowledge
Knowledge denae sub umit
Battery
Buttery
Panett: luttery
Alia: 83%
Ineascoee Full (low buttory buttery bar groperty desettyl..
Redox battery
Defiaitio::
battryy
tooking hair buttery
Ico traade inskte the cell (acoompenied flow of clectric carrent thoogh un cstersal dimae).. redox battery (RFN) yout foture hrgg-sodle applicaioos. The membrane, especially....
Semantic-rich cab-unit
Top-K units
O Botteyy etll
Fall flwe bettery
Itybrid flow battery
Sodiumion
Full
battery
Ontology Enrichment & Unit Construction
Evidence Retrieval Expansion
Figure Overall framework (ORAG vs. RAG). Conventional RAG retrieves text chunks while ORAG retrieves informative structured knowledge units. ORAG first enriches the ontology with external sources and constructs a knowledge unit for each node. It then retrieves the relevant nodes and further expands evidence based on the ontology structure.
ii Alias like abbreviations can help to recognize entities that may appear under different names and improve the recall rate of retrieval.
iii. KG triples can provide possibly related entities that may be included in the context. The KGs usually contain many noisy relations. We filter the noisy relations with the similarity between the triples and the theme, calculated by pre-trained encoder
iv. If some entity instances are available, also collect them to help LLMs better under- stand long-tailed types. rich information is available about such instances, we further add them into the ontology leaf nodes under the corresponding type.
To include comprehensive knowledge for better reasoning, collect related descrip- tions from large corpora such as Wikipedia, which can provide multi-view information about the type. A description is usually longer than definition.
Note that we do not assume complete knowledge is available. In other words, we allow substantial misses in the above fields in most types, which is true in real-world applications. For example, in Figure 2, the real available data miss some information for all types except the battery type and the redox flow battery type (e-g., the rocking chair battery type misses knowledge triples, the full flow battery type misses everything except a definition). Our algorithm handles partial information by inheriting information between parents and children in the ontology, as described in later sections.
3.2.2 Structured Knowledge Unit Construction
As shown in Figure 2 vanilla RAG retrieves relevant text from its retrieval memory, which are directly split into fixed-size text chunks. However, the evidence for reasoning may be scattered in different chunks, and thus the retrieved chunks could include considerable percentage of irrelevant context. With the limited input length of LLMs, including irrele- vant context in one chunk means dropping other chunks that also contain some relevant information. Therefore, this type of naive retrieval can be inefficient in terms of the density and coverage of the relevant knowledge in the retrieved results. To effectively retrieve the

---

Published as a conference paper at COLM 2024
related information, construct Structured Knowledge Unit for each entity type in the theme ontology (Panel 1, Figure
The structured knowledge unit of an entity type includes two parts:
i. Knowledge-dense sub-unit: It includes the concise definition, alias, KG triples, and instances.
ii. Semantic-rich sub-unit: It includes the descriptions about the entity type.
We separate the two sub-units to balance knowledge density and semantic richness when we augment LLM queries in later steps. The knowledge-dense sub-units contain the most essential and concise information about the entity types. Although the semantic-rich sub- unit may be generally less important and informative, it may also provide subtle signals for reasoning if it is aligned with an LLM query. Thus, the handling of the two sub-units is different.
We also observe that RAG has better performance understanding and reasoning when the retrieval memory consists of complete sentences instead of key-value pairs. Therefore, design patterns for each type of information. For example, for the alias of redox flow battery, we store the sentence Redox flow battery is also known as RFB in the retrieval memory instead of (alias, RFB). Similarly, we restate other information with complete sentences and finally construct the retrieval memory.
3.3
Ontdlogy-Guided RAG
3.3.1 Evidence Retrieval Based on Structured Units
Different from vanilla RAG that searches text chunks, ORAG searches structured knowledge units. The motivation of ontology-guidee RAG can be two-fold: (1) Structured knowledge units are more knowlegge-dense and informative when compared to fixed-sized text chunks. With knowledge-dens units, ORAG can append more relevant evidence to query LLMs within the same input length to facilitate entity typing. (2) The retrieved information is self-coherent and effective and could contain less irrelevant information (which may mislead the LLMs to generate erroneous results (Wang et al. 2023)).
We use the embedding model from OpenAl to encode the sub-units and cluster the embeddings based on FAISS index (Johnson et al. 2219).. For dense passage retrieval (DPR) (Karpukhin et al. 2020), given the LLM input query for entity typing, we rank the structured knowledge sub-units in bi-encoder architecture.
We retrieve the top-k entity types on the theme ontology regarding the rank of similarities of sub-units. at least one sub-unit is retrieved, we consider the corresponding entity type to be a relevant node in the ontology. The entity types will be used as starting entities for further ontology-guides evidence expansion and selection.
3.3.2 Ontdlogy-Guided Evidence Expansion
In this section, further expand and select the information based on ontology and starting entities (i.e. redox flow battery and battery cell in Figure 2).
The starting entity types are retrieved because they can be directly semantically matched by entity mentions or context. However, the one-hop semantically matching evidence may still be insufficient when the typing process requires reasoning, especially for unseen entities.
Take the example in Figure 1 the entity mention all-organic redox flow battery is newly proposed entity unseen for LLMs. The mention is semantically matched the type redox flow battery. However, it can be reasoned from the context that it belongs to a more fine- grained type non-aqueous organic flow battery. In the vanilla RAG, the descriptions of the target type are not retrieved, and LLMs generate an inaccurate answer.

---

Published as a conference paper at COLM 2024
Table 1: Dataset statistics. Battery represents "redox-active organic electrode materials" and Game represents "online game posts for Genshin Impact".
Theme
Documents
Entities
Hard Cases
Types
Battery
20
100
20
484
Game
20
200
20
73
The goal of this step is to include multi-hop information for more accurate reasoning entity types. Given starting entity, to discriminate the finer-grained entity type, we retrieve its top-N relevant child nodes. In the case above, the LLMs can give the right answer with the knowledge units of non-aqueous organic flow battery and aqueous organic flow battery. We also retrieve the knowledge-intensive sub-unit of its parent node to provide more information to discriminate the starting type and its sibling types. In this way, even if some knowledge about a type is missing, the information inherited from parent and child nodes can help.
Due to the limited input length of LLMs, for most retrieved entity types, we only include the concise knowledge-dense sub-units in the augmented LLM query. The long semantic-rich sub-units are included only when their similarity to context is higher than the knowledge- dense sub-unit, which means the descriptions can help understand the context. The evidence is gathered, and identical items are merged. Finally, the evidence and the context are fed into LLMs to perform typing. The query prompts are as follows, where (entity) and (passage) are placeholders for the corresponding entity and passage.
LLM query template. What are the types of the entity (entity) in this passage? If there are multiple types, return them all. Format the output as typel, type2, (passage)
Experiments
4.1 Dataset Construction
We construct dataset with two themes: one for redox-active organic electrode materials and the other for online game posts for Genshin Impact. The dataset have a focus on unseen entities and misleading semantics, two situations where LLMs are more likely to suffer from hallucinations. The former features highly specialized and complicated terminologies and ontology structures, whereas the latter features fast-evolving entities and knowledge that pre-training corpora of commercial LLM services almost always lag behind. The data on each theme include the following. (1) Corpus: a set of relevant documents that contain unseen entities and misleading semantics; (2) Ontology: set of entity types arranged into a hierarchy describing the fine and coarse granularities; (3) Entity mentions: entity spans are located in the corpus by human annotators; (4) Entity types: the ground-trutie entity types of the entity spans are identified from the given ontology by human annotators;
and (5) Hard cases: A hard case is manually selected from each document, representing entity mention that can hardly be typed correctly with its context only but correct typing becomes logically possible when theme-specific knowledge is included in the reasoning. In our experiments, such a hard case always features either unseen entities or misleading semantics. The statistics of the dataset are given in Table 1
4.1.1 Highly Specialized Theme: Redox-Active Organic Electrode Materials
The first theme is a highly specialized battery theme of redox-active organic electrode materials. We use the Battery Interface Ontology (BattINFO) (Clark et al., 2022) which contains around 20,000 entity types. Entity annotations are performed on a set of research papers in this field with a focus on batteries and electrodes. All annotated entities have types under the two sub-ontologies rooted at the Battery type (with 298 nodes in the sub-ontology) and the Electrode type (with 186 nodes in the sub-ontology). These two sub-ontologies add up to 484 types in total. The structured retrieval memory is constructed from BattNNFO,
7

---

Published as a conference paper at COLM 2024
Wikipedia, Wikidata, and Electropedia which contains all the terms and definitions in the International Electrotechnical Vocabulary.
4.1.2 Fast-Evolving Theme: Online Game Posts for Genshin Impact
The second theme is fast-evolving game theme of online posts for Genshin Impact popular game winning the TGA best mobile game in 2021 and keeping fast-evolving on new entities. We collect bilingual passages from Chinese and English forums. We invited senior players with at least three years of gaming experience Genshin Impact to construct the ontology (with 73 nodes) and perform annotation. The structured retrieval memory is constructed from Genshin Impact Wik S
4.2 Experimented Methods
We report the results of the following methods: (1) RAG-Llama-3: the open-source Llama-3-
8B-Instruct mode combined with vanilla RAG, (2) RAG-GPT-35; the GPT--355 with vanilla RAG, (3) RAG-G the GPT-4 with vanilla RAG, (4) ORAG-Llama-3: the same Llama-3 with ORAG, (5) ORAG-GPT-355 the same GPT-3.5 with ORAG, and (6) ORAG-GPT-4: the same GPT-4 with ORAG. The pre-training data cutoff dates of both GPT-3.5 and GPT-4 are September 2021. All methods share the same prompts in Section 3.3.2
4.3 Experiment Results
Table shows the entity typing results on the two themes. We report the macro-average of precision, recall, and Fl scores. Given an LLM (Llama-3, GPT-335, or GPT-4), our ORAG improves the Fl scores over the vanilla RAG by 10% 24%. Remarkably, ORAG improves the precision of GPT-3.5 by 24% and 29% respectively, demonstrating the substantial efficacy of ORAG in reducing LLM hallucinations. ORAG also boosts the precision of GPT-4 nearly perfect level, providing hope for the complete elimination of hallucinations when the correct knowledge is used together with more advanced LLMs in the future. ORAG also boosts the performance of Llama-3 compared to vanilla RAG, demonstrating the applicability of our method to open-source LLMs.
A notable improvement in recall is also consistently observed, while this improvement has a different indication than that in precision. Without ORAG, LLMs tend to predict coarse-grained types, which is easier to get right. Stopping at correct but coarse-grained types without going deeper into the ontology only hurts the recall but not the precision. That why the precision scores consistently higher than recalls. The rise of recall brought by ORAG demonstrates the usefulness of theme-specific knowledge in guiding LLMs into more fine-grained types.
We also examined the performance of all models on the set of hard cases. Results are shown in Table 3 While vanilla RAG performs much worse on hard cases compared to easy cases, ORAG brings the typing quality on hard cases much closer to the overall performance. In the battery theme, the performance of LLMs on hard cases is almost brought back to that on the average cases, indicating that ORAG can effectively handle unseen cases and misleading semantics. On the game theme, the performance of GPT-3.5 on hard cases after ORAG is applied still has a gap from its counterpart on the average cases. As this theme is evolving so fast (i.e., a significant update every six weeks) that a considerable portion of the up-to-date knowledge is missing from LLM pre-training corpora, the strong reasoning capability of very large models seems to play an essential role in handling hard cases.
https://ww.bbyolabb.co////
Impact_ Wiki
htgp://ugggungneeecccmmaaaaaaaap Instruct
8

---

Published as a conference paper at COLM 2024
Table Theme-specific entity typing results for all entities. Themes are abbreviated. Battery: Redox-active organic electrode materials. Game: Online game posts for Genshin Impact.
Battery Theme
Game Theme
LLM
Method
Precision
Recall
Micro-F1
Precision
Recall
Micro-F1
RAG
0.86
0.56
0.68
0.76
0.61
0.67
LLAMA-3
ORAG
0.97
0.80
0.83
0.85
0.80
0.83
RAG
0.72
0.64
0.64
0.68
0.59
0.61
GPT-3.5
ORAG
0,96
0.79
0.83
0.97
0.73
0.80
RAG
0.87
0.65
0.70
0,92
0.71
0.78
GPT-4
ORAG
0.99
0.90
0,94
0.98
0.82
0.88
Table 3: Theme-specific entity typing results for hard cases with unseen entities or mislead- ing semantics. Themes are abbreviated. Battery: Redox-active organic electrode materials. Game: Online game posts for Genshin Impact.
Battery Theme
Game Theme
LLM
Method
Precision
Recall
Micro-F1
Precision
Recall
Micro-F1
RAG ORAG
0.74
0.48
0.53
0.60
0.56
0.51
LLAMA-3
0.95
0.76
0.83
0.87
0.82
0.83
RAG
0.67
0.43
0.50
0.42
0.41
0.39
GPT-3.5
ORAG
0.99
0.71
0.79
0.94
0.50
0.62
RAG ORAG
0.77
0.41
0.48
0.83
0.53
0.64
GPT-4
0.99
0.87
0.92
0.96
0.78
0.85
5
Case Studies
5.1 A Case of An Unseen Entity
In the following, we discuss case of an unseen entity from the game theme. In Example Passage 2, the ground-truth type of the entity "fruit platter" is Weapon Catalyst (from coarse-grained type to fine-grained type). This weapon was first online in February 2022, so the GPT models with training corpora cut-off date in September 2021 never saw this entity before. The entity "fruit platter" is nickname of the weapon Kagura's Verity originating from the community, and is also not included databases. Therefore, the correct entity type should be logically inferred from theme-specifis knowledge and the context.
In this case, the context discusses some characters and the weapons they can use. can be semantically inferred from the context that the "fruit platter" refers to a Weapon. However, with expert knowledge about characters, we can further infer that "fruit platter" Cafalyst.
Example Passage 2. (Original post in Chinese) & - K #
f / A
(Translated post in English) Now that the Polar is coming, Childe will have his own weapon, and Yoimiya can take back her Thunder. Unfortunately, T don't have Miko or Nahida. turn out to get the fruit platter instead, who make use of it?
RAG-GPT-4 only gives the coarse-grained entity type Weapon. The Weapon is generated because the term can be directly semantically matched in the context. Also, the RAG-GPT4 lacks essential knowledge about context to infer the more fine-grained entity type.
ORAG-GPT-4 correctly predicts the fine fine-grained type Catalyst. Although the Catalyst does not appear in the context, we still get its structured knowledge units by evidence expansion from its parent node Weapon. Also, ORAG retrieves the related information of two characters

---

Published as a conference paper at COLM 2024
in the context, Miko Yae and Nahida, which reveals that both Miko and Nahida use Catalysts. With such information added to the query, GPT-4 becomes able to tell the correct type.
5.2 A Case of Misleading Semantics
Example Passage (Repeated). CV data of Cu-TCA were recorded using a Li foil as counter electrode. (Omitted) The CV curve of bare electrode (only containing the Al foil) proves that the above-mentioned redox areas originate from the active sites of the Cu-TCA structure.
In this section, we dive deeper into the "bare electrode" case in Example Passage in Section to analyze the output of experimented methods. We repeat the passage here for readers' convenience. The "bare electrode" has two ground-truth fine-grained types following the ontological paths: Electrode + Working Electrode, and Electrode + Active Electrode + Metal Electrode + Aluminum Electrode. (The Aluminun Electrode type is not this case, the correct typing requires full understanding of the context and the types to avoid being misled by irrelevant context.
discussed in Section [] to avoid over-complication)
RAG-GPT-4 predicts Aluminium Electrode and Aluminum Insertion Electrode, where the former is correct fine-grained type but the latter is a hallucination. While an Insertion Electrode type can be inferred from the first half of Example Passage and an Aluminum Electrode type can be inferred from the second half, combining the two pieces of semantics into an Aluminum Insertion Electrode type is scientifically wrong.
ORAG-GPT-4 predicts Aluminium Electrode and Working Electrode, which are correct and fine-grained. The algorithm retrieves the structured knowledge unit of the Working Electrode through its evidence expansion process, the Working Electrode is child type of Electrode. also retrieves the unit of Insertion Electrode through its parent Active Electrode. With the relevant knowledge added to the query, GPT-4 becomes able to identify that the Working Electrode is match while the Insertion Electrode is not.
Conclusion
In this work, we introduce ORAG, theme-specific fine-grained entity typing framework that works well on cases of unseen entities and misleading semantícs that are hard for LLMs. Compared to vanilla RAG, our proposed ORAG gives more precise entity types with structured retrieval from knowledge-enriched theme-specific ontology. During the evaluation of highly specialized theme on electrode materials and rapidly evolving theme on game posts, we find that our ORAG yields more accurate theme-specific entity typing results compared to the vanilla RAG method.
Acknowledgement
The research was supported in part by the IBM-IIlinois Discovery Accelerator Institute (IDAII, the Molecule Maker Lab Institute: An Al Research Institutes program supported by NSF under Award No. 2019897, INCAS Program No. HR001121C0165, National Science Foundation IIS-19-5615, and the Institute for Geospatial Understanding through an Integra- tive Discovery Environment (I-GUIDE) by NSF under Award No. 2118329. Any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and do not necessarily represent the views, either expressed or implied, of IBM or the U.S. Government.
References
Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettlemoyer. Ultra-fine entity typing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 87-96, July 2018.
10

---

Published as a conference paper at COLM 2024
Simon Clark, Francesca L Bleken, Simon Stier, Eibar Flores, Casper Welzel Andersen, Marek Marcinek, Anna Exczesna-Chrzan Miran Gaberscek, M Rosa Palacin, Martin Uhrin, et al. Toward unified description of battery data. Advanced Energy Materials, 12(17):210270,, 2022.
Ning Ding, Yulin Chen, Xu Han, Guangwei Xu, Xiaobin Wang, Pengjun Xie, Haitao Zheng, Zhiyuan Liu, Juanzi Li, and Hong-Ged Gee Kim. Prompt-learnng for fine-grained entity typing. In Findings of the Association for Computational Linguisties: EMNLP 2022, PP. 1888-6901 2022.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arviv preprint ar xxXiv:2312.10997 2023.
Dan Gillick, Nevena Lazic, Kuzman Ganchev, Jesse Kirchner, and David Huynh. Context- dependent fine-grained entity type tagging. arXiv preprint ar arv: Xii::112..1800 2014.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wai Chang. Realm: retrieval-augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, pp. 3929-3938, 2020.
Yan Hu, Qingyu Chen, Jingcheng Du, Xueqing Peng, Vipina Kuttichi Keloth, Xu Zuo, Yujia Zhou, Zehan Li, Xiaoqian Jiang, Zhiyong Lu, et al. Improving large language models for clinical named entity recognition via prompt engineering, Journal of the American Medical Informatics Association, pp. ocad259, 2024.
Jiaxin Huang, Yu Meng, and Jiawei Han. Few-shot fine-grained entity typing with automatic label interpretation and instance generation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 605-614, 2022.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions Big Data, 733)5535-547 2019.
Minki Kang, Jin Myung Kwak, Jinheon Back, and Sung Ju Hwang. Knowledge graph- augmented language models for wwwlledg--rrundde dialogue generation. Xrv preprint axXiv:2305.18846 2023.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answer- ing. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online, November 2020. Association for Computational Linguistics. doi: 11883//vl org/2020..../.
2020.emnlp-main.55 5500. /
org/ 2020. emnlp-main. 550
Tanay Komarlu, Minhao Jiang, Xuan Wang, and Jiawei Han. OntoTypp: Ontology-guider zero-shot fine-grained entity typing with weak supervision from pre-trained language models. arxiv preprint ar Xiv:2333 12307, 2023.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for mnwwledge-iteensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459 9474, 2020.
Xiao Ling and Daniel Weld. Fine-grained entity recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, pp. 94-100. Association for the Advancement of Artificial Intelligence (AAAI), 2021.
Robert Logan, Nelson F Liu, Matthew E Peters, Matt Gardner, and Sameer Singh. Barack's wife hillary: Using knowledge graphs for fact-aware language modeling. In Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
Siru Ouyang, Jiaxin Huang, Pranav Pillai, Yunyi Zhang, Yu Zhang, and Jiawei Han. Ontology enrichment for effective fine-grained entity typing. arxiv preprint arXin:2310.07795, 2023.
11

---

Published as a conference paper at COLM 2024
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt a general-purpose natural language processing task solver? In Proceedings of the 2023 Conference on Empirical Methods in Natural Languing Processing, pp. 1339-1384, 2023.
Xiang Ren, Wenqi He, Meng Qu, Lifu Huang, Heng Ji, and Jiawei Han. AFET: Automatic fine-grained entity typing by hierarchical partial-label embedding. In EMNLP'66, pp. 1369-1378, 2016.
Priyanka Sen, Sandeep Mavadia, and Amir Saffari. Knowledge graph-augmented language models for complex question answering. In Bhavana Dalvi Mishra, Greg Durrett, Peter Jansen, Danilo Neves Ribeiro, and Jason Wei (eds.), Proceedings of the 1sf Workshop Natural Language Reasoning and Structured Explanations (NLRSE), pp. 1-8, Toronto, Canada, June 2023. Association for Computational Linguistics. doi: 10. / /v///v1///// v1//223.nnrsee URL https::/aclanttologg..o org/ 2023. 21w.3enrreee..1
Karthik Soman, Peter W Rose, John H Morris, Rabia E Akbas, Brett Smith, Braian Pee- toom, Catalina Villouta-Reyes, Gabriel Cerono, Yongmei Shi, Angela Rizk-Jackson, al. Biomedical knowledge graph-enhanced prompt generation for large language models. ar Xiv preprint ar xrX:::3311.1733) 2023.
Yile Wang, Peng Li, Maosong Sun, and Yang Liu. Self-knowledge guided retrieval augmenta- tion for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Find- ings of the Association for Computational Linguistics: EMNLP 2023, pp. 10303- Singa- pore, December 2023. Association for Computational Linguistics. doi: (0.18653/v1/2000 findings-emmlp.699 URL http:///aaaaaaoooo.....oo org. wtt.g//22...nnnnnoe.. findings- emnlp mannp.699) 691
Yuxiang Wu, Yu Zhao, Baotian Hu, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. An efficient memory-augmentedd transformer for knowledge-intenssve NLP tasks. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp- 5184-5196 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. org/ 2022. emnlp-main
18653/v1/20BB 2tt/5//1/2222.emmpmmammmg 346
URL
Yu Zhang, Yunyi Zhang, Yanzhen Shen, Yu Deng, Lucian Popa, Larisa Shwartz, ChengXiang Zhai, and Jiawei Han. Seed-guided fine-grained entity typing in science and engineering domains. AAAI, 2024.
Ben Zhou, Daniel Khashabi, Chen-Tse Tsai, and Dan Roth. Zero-shot open entity typing as type-compatible grounding. In EMNLP'18, pp. 2065-2076, 2018.
12


Reviews:
[{"review_id": "4D0LDcteSM", "reviewer": "Reviewer_28vN", "summary": "The paper proposes an Ontology guided RAG for the task of theme specific fine-grained entity typing. The paper argues that traditional datasets for entity typing, does not capture scenario with unseen entities or situations with misleading semantics, and constructs two datasets for evaluation under this scenario. Further, the paper proposes an ontology guided RAG algorithm, where the labels are enriched with expert knowledge and RAG is performed on top of these enriched ontology. Experimental evaluation shows ORAG leads to 10-24% improved F1 score as compared to traditional RAG.\n\nThe paper has good quality, clarity and originality. However, evaluation is done on a very small dataset of 20 documents, raising questions on the generalizability of the results.\n\nBased on new results presented by the author in the rebutal phase, where they show generalizability of their approach to an existing CHEMNER dataset, I have updated my rating to \"6: Marginally above acceptance threshold\"", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our paper's quality, clarity, and originality. We understand your concern about generalizability, and we would like to respond to your concern as follows.\n\n1. **Regarding the small dataset.** Our work focuses on the entity typing task on very narrow themes (e.g. redox-active organic electrode materials), which feature long-tailed knowledge with limited data. The number of documents per theme (20) is set to be small to align our experiments with such scenarios.\n\n2. **Regarding the choice of themes and the generalizability.** The two themes are very different, demonstrating the **generalizability across themes**. The battery theme shows **generalizability in unseen types** because the types in the ontology are very hard to understand even for GPT-4 (e.g., the “Aluminum Insertion Electrode” type in Section 5.2). The game theme shows **generalizability in unseen entities** because the game is evolving very fast, and about half of the entities emerged very recently (e.g., four entities in Example Passage 2 are non-existent in the GPT pre-training corpus).\n\n3. **To further demonstrate generalizability**, we add an additional experiment on a **chemistry theme “organic reactions”** using the ontology from a previous work called ChemNER. Results are shown in the following table. The conclusions are consistent with the two other themes in our submission.  If you have a particular theme of interest, you are welcome to let us know and we may try it out.\n| Method | Precision | Recall | F1 |\n| --- | --- | --- | --- |\n| RAG-GPT-4 | 0.89 | 0.64 | 0.75 |\n| ORAG-GPT-4 (Ours) | **0.91** | **0.83** | **0.87** |\n\n4. Our methodology is designed to be **theme-independent** and generalizable. We would like to gratefully refer to Reviewer ShE4’s “Reasons To Accept” section for this point.\n\n5. Despite all the above discussions, the scope of our experiments is still limited. Thus, we agree with Reviewer ShE4’s evaluation of our work as a **successful pilot study** that clearly suggests the approach is worth large-scale evaluation. Creating annotations on more themes is challenging as it requires expertise and can hardly be crowdsourced. We would greatly appreciate it if you kindly consider this pilot study to be accepted for publication to attract attention and resources for constructing large-scale theme-specific datasets.\n\nWe welcome further comments and discussions.\n\nUpdate: Thank you for raising your rating!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper argues that traditional datasets for entity typing, does not capture scenario with unseen entities or situations with misleading semantics, and constructs two datasets for evaluation under this scenario.\n2. The paper proposes an ontology guided algorithm for RAG, including enriching the labels in the ontology. The proposed approach has novel contributions.\n3. Empirical evaluation shows that the proposed ORAG approach has significantly improved performance over traditional RAG for the task of theme based entity typing.", "reasons_to_reject": "1. Evaluation is done on a very small dataset of 20 documents, raising questions on the generalizability of the results.\n2. Even the choice of themes - a) redox-active organic electrode materials, and b) online game posts for Genshin Impact, seems arbitrary without proper justification on how learnings on those datasets showcase generalizability.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R9s3Jvwe4Q", "reviewer": "Reviewer_ShE4", "summary": "This manuscript discusses an approach for improving fine-grained entity typing in a zero/few shot setting with LLMs, specifically GPT-4, by combining retrieval-augmented generation (RAG) with a knowledge base generated from a domain-specific ontology. The application area is chosen to be \"highly specialized or fast-evolving themes,\" with the idea being that general models would not have been trained sufficiently in these areas. This is because the relevant distinctions would have appeared after the training cutoff for the fast-evolving themes, and because the relevant distinctions for highly specialized domain would be too specialized to appear in the training data.\n\nThe basic approach is to use the ontology to construct a knowledge base consisting of a \"knowledge unit\" for each node. The system then uses a RAG approach with a untuned LLM to perform the task, but in addition to returning the knowledge unit that best matches the query, the search component also returns the knowledge units for closely related nodes. The evaluation selects one highly specialized domain (redox-active organic electrode materials, a type of battery) and one fast-evolving domain (Genshin Impact, an online game). The battery domain uses the Battery Interface Ontology, which contains approximately 20,000 entity types. The online game does not have an ontology, so the authors work with \"senior players\" to create one, with 73 entity types, and annotate documents. The evaluation corpus for the battery domain consists of 20 \"research papers in this field with a focus on batteries and electrodes.\" The corpus contains 100 annotated entities (mentions?) and 484 types, note this implies multiple types per mention. The evaluation corpus for the online game domain also contains 20 documents, in English and Chinese, with 200 entities (mentions?) and 73 types. Both evaluation datasets list one \"hard case\" for each document, essentially a mention that cannot be typed correctly without using knowledge from the ontology. The evaluation uses a straightforward prompt and considers GPT3.5 and GPT-4 with and without ORAG, demonstrating substantial improvements for precision, recall and f1-score in all cases. The discussion focuses on two case studies, one \"unseen entity\" (a nickname for a weapon in the online game that would only make sense in the context of the game) and one \"misleading semantics\" (a case where the specific discussion in context would seem to suggest a different, incorrect, type).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful feedback and thoughtful queries. We aim to respond concisely within the length limit.\n\n**Themes vs. Domains**. Compared to domains (e.g., chemical and biological research), we use “themes” to refer to much narrower topics. Themes feature long-tailed, in-depth, or fast-evolving knowledge (e.g., “redox-active organic electrode materials” is a theme under the chemistry domain)\n\n**Error bars**. On the battery theme, for ORAG-GPT-3.5, SD(P) =0.11, SD(R) =0.28, SD(F1) =0.21. The paired t-test p-value over RAG is 0.008, showing our improvement is statistically significant. We omit other statistics to save space. \n\n**Experiment processes**. Each theme has two annotators. The primary annotator selects documents and performs the first round of annotation. Then, the predicted results from all models are pooled together and delivered to the primary annotator for a second round of annotation. After this, the secondary annotator checks the results and resolves disagreements with the primary annotator. The ontology is created and checked similarly. \n\n**Concrete applications**. The battery theme is inspired by a request from our industry partner to discover novel battery materials from text. The game theme originated from our observation that AI tools often make mistakes on game-related topics.\n\n**Evaluation metrics**. You are right that evaluation is done at the mention level. When multiple mentions in a document share the same surface name, we evaluate the first occurrence. The evaluation is multi-type, and parent types are auto-filled when a child type is predicted. Precision evaluates the percentage of predicted types that are ground-truth types. Recall evaluates the percentage of ground-truth types that are predicted.\n\n**Battery ontology**. We report the result using 484 relevant types instead of the full ontology.\n\n**Hallucination**. We use the word's broader meaning to refer to cases where LLM generates answers that look reasonable to ordinary people but are wrong based on expert judgment.\n\n**Scalability**. You may refer to the cost of running GPT-4. We add the experiments on the open-source model LLAMA3-8b-instruct and observe similar results. Please refer to our response to Reviewer YRhV for the performance table.\n\n**Generalizability**. Please refer to Points 1-3 of our response to Reviewer 28vN. There we also include results for an additional theme.\n\nWe highly appreciate your detailed, constructive review and welcome further comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This approach neatly addresses the need for domain-specific logic when using general LLMs in a specialized domain. The approach requires no modifications or fine-tuning for the LLM, and thus can be used in zero-shot settings for any LLM that already supports RAG. The approach is straightforward and seems relatively domain-independent, and thus should be widely implementable. The evaluation supports that the idea works, at least for the domains chosen. Overall, this seems like a viable way to add specialized semantics to an LLM without fine-tuning. It would also be orthogonal to approaches that would fine-tune the LLM, though this is not evaluated/tested in this manuscript.", "reasons_to_reject": "There are several issues with the manuscript that reduce my enthusiasm for it. \n\nFirst, the evaluation corpora are quite small: each is only 20 documents and the total number of entities (I think they mean mentions) is only 300. Thus, even with the substantial performance improvements described, these evaluations should not be considered definitive, even for the two domains described. The results for \"hard cases\" are even more uncertain, since they are based on only 20 mentions for each domain. I believe error bars on the evaluation (Tables 2 & 3) would help substantially, but even so, I think this work is best described as a pilot study - a successful one that clearly suggests the approach is worth large-scale evaluation, but still a pilot study. \n\nSecond, only two domains are considered. There are many others which this method should help, such as chemical, biological, and clinical research, which have substantial datasets that should be adaptable to FET. It is possible, however, that the authors do not consider these to be sufficiently specialized or fast moving. In any case, with only two domains considered, it is likely that there are substantial issues in some domains that would complicate its use or cause it to fail. Again, I think this work is best described as a successful pilot study. \n\nIt is also somewhat unclear in what applications this approach would be most useful. It requires a fairly comprehensive ontology, which must be created if it does not already exist, which implies a substantial effort. This effort itself would require substantial time and expense to evaluate examples from documents, which suggests that at least in the case of domains that do not already have an ontology, the approach should ideally be compared to creating a small training corpus and fine-tuning an LLM. The approach also requires one LLM prompt per document, mention pair, so likely cannot be applied to large-scale datasets with GPT-4: if we assume the prompt and response are 1000 tokens on average (750 words) and 100 mentions per document, then a dataset of only 10,000 documents would cost $45,000 (this analysis ignores the RAG component, which I assume would also cost tokens). Finally, fine-grained entity typing itself is probably best understood as an intermediate natural language processing task, one that would be useful in a pipeline but one that users are not likely to care about directly. The most straightforward application users might care about, building or expanding an existing knowledge base, would likely run into problems scaling. At the very least, it would be helpful to suggest some user-facing applications.\n\nIn addition to the evaluation being small, the manuscript provides only a very superficial description of the process for creating the evaluation datasets. Not described are: how the documents were selected, the annotation process (the number of annotators per document, how disagreements were handled, etc.), the annotation guidelines (what constitutes a mention and what does not), inter-annotator agreement, how the online game ontology was created (e.g. the number of entity types in the ontology is the same as the number in the corpus, suggesting that the ontology was created from the corpus), whether the knowledge units for the battery domain included all 20,000 entity types or only the 484 in the dataset. All of these are relevant for understanding the quality of the evaluation datasets.\n\nThe manuscript does not clarify how performance was measured. Is the performance for each mention (i.e., two mentions of \"fruit platter\" in a single document would count twice), or for each (document, mention text) pair, or something else? How are true positives calculated, given that each mention may be annotated with multiple entity types, each of which is hierarchical? The authors allude to a hierarchical evaluation in section 4 (\"Stopping at correct but coarse grained...\") but do not provide details. \n\nSome may argue that the straightforward approach described is too incremental or obvious to be useful. While I know of related methods, I am not aware of any that combine ontologies with RAG so neatly and in a way that would be useful in a zero-shot setting. Thus, I specifically argue that any such argument against this manuscript should be accompanied by a citation of one or more works that allow an LLM to use RAG with an ontology in a zero-shot setting. \n\nThere are several clarifications or terminology changes that would be helpful:\n\nFine-grained entity typing (FET) is not as well-known as named entity recognition (NER) and entity linking (EL). A brief contrast between them would help the reader better understand the needs of the task. In this case, NER typically involves predicting span and rough entity type, while EL involves predicting a specific identifier from a controlled terminology or lexicon that is potentially very large (millions of entities). Fine-grained entity typing does not involve predicting the span, but does involve predicting a type from a set of types which are intermediate in size between the rough entity types of NER and the identifiers in EL. FET may also have a context dependency such that a particular aspect or role of the entity is emphasized, such as classifying a mention of a person that is a politician, business person and media personality as referring in context specifically to \"person/politician\"; in this sense FET may be more specific than even EL (see, e.g., Dai et al., EMNLP-IJCNLP 2019, Improving Fine-grained Entity Typing with Entity Linking https://aclanthology.org/D19-1643)\n\nThe authors use the word \"entity\" ambiguously, to refer to the type from the ontology and also to the text from the document that should be typed. The latter is more typically known as a \"mention,\" and I strongly suggest updating this terminology since it makes the manuscript difficult to read and makes some of the descriptions unclear (e.g., doe the \"entities\" in Table 1 refer to mentions?)\n\nThe authors use the word \"theme\" where I believe that the field more typically uses the word \"domain.\" This is not critical, since I don't think there is any ambiguity, but it would align the manuscript better with the field.\n\nFinally, while the LLM literature uses the word \"hallucination\" rather liberally, this manuscript not only does not define it but uses it to describe almost any sort of error made by the LLM. This sort of imprecise language makes understanding the problems being addressed and analyzing the results more difficult. It also fails to make one of the strongest points in favor of this approach: Subbarao Kambhampati (e.g. https://twitter.com/rao2z/status/1772636256184520981) talks about LLMs being able to capture a distribution, but correctness is a property of an instance; by extension, \"hallucination\" is where the LLM returns something that sounds correct because it follows the distribution of the domain, but is not correct because the LLM has no knowledge of the entities within the domain. Thus, the approach described by the authors neatly allows the ontology to inform the LLM of the entities, which should dramatically reduce hallucinations, according to this definition, but the terminology used in the manuscript is not precise enough to support this point. (I'm also unsure whether the task would even support this sort of analysis.)", "questions_to_authors": "Could you describe some concrete applications where this approach would be useful to users?\n\nPlease clarify the evaluation metrics\n\nWhat are the error bars on the evaluation metrics from Table 2 & 3?\n\nPlease clarify the document-selection process, annotation guidelines, annotation process, inter-annotator agreement and ontology creation process. \n\nPlease clarify whether the full battery ontology was used during the evaluation or only the part relevant to the 484 types in the dataset.\n\nHow do you define \"hallucination\"?\n\nHow well would this approach scale?\n\nHow applicable would this approach be to other domains? How do you know?", "extra_scores": {}}, {"review_id": "OTKjbxYkzN", "reviewer": "Reviewer_YRhV", "summary": "The paper proposes a new task called theme-specific entity typing which is related to the typical entity typing task problems studied in NL, but for a specific theme or domain such as online gaming. The paper aims to guide the LLM generation with information-rich ontologies. It organizes the knowledge in a hierarchical ontology with each node representing an ontology type along with knowledge-dense sub-units and semantic-rich sub-units. The retrieval happens at the ontology’s node level as opposed to the chunk of text used in regular RAG.  The paper curated two datasets with human annotations and the corresponding ontologies. Experimental results show that the ontology-guided RAG makes GPT-3.5 and 4 models perform better.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for recognizing our work’s uniqueness, effectiveness, and usefulness. We aim to respond concisely within the length limit.\n\n**The Availability of Knowledge**\n\nThank you for raising the question on the availability of knowledge for fast-evolving themes. We would like to clarify the following points.\n\n- **We do not require a complete set of knowledge.**  In fact, most types in our ontology are described by partial information, and our reported results are based on such partial information. In Figure 2, all types miss some information except the Battery type and the Redox Flow Battery type (e.g., the Rocking Chair Battery type misses knowledge triples, the Full Flow Battery type misses everything except a definition)\n- **Our algorithm handles partial information by inheriting information between parents and children in the ontology**. For example, although Full Flow Battery misses everything except a definition, by inheriting the information from its parent type, Redox Flow Battery, LLMs become better at tackling the Full Flow Battery type.\n- **We assume partial knowledge is available. This is true even for fast-evolving themes.** On one hand, entity types evolve much slower than entities, and information for existing entity types is largely available for fast-evolving themes. On the other hand, new knowledge regarding emerging types may be extracted from texts (e.g., [Klink-2](https://skm.kmi.open.ac.uk/klink-2/)), which is out of the scope of this study, or updated by the users to customize the results.\n\n**The Scope of Experiments**\n\nOur response to Reviewer 28vN is dedicated to this concern. To save space, please refer to that response. There, we provided **new results on an additional theme** and reached the same conclusions.\n\n**Open-Source LLMs**\n\nFollowing your suggestion, we added an experiment with **LLAMA3**-8b-instruct, which is open-source and easy to reproduce. The **results are consistent** with the LLMs reported in our submission.\n\n| Battery Theme | Precision | Recall | F1 |\n| --- | --- | --- | --- |\n| RAG | 0.86 | 0.56 | 0.68 |\n| ORAG (Ours) | **0.97** | **0.80** | **0.87** |\n\n| Game Theme | Precision | Recall | F1 |\n| --- | --- | --- | --- |\n| RAG | 0.76 | 0.61 | 0.67 |\n| ORAG (Ours) | **0.85** | **0.80** | **0.83** |\n\n**Ablation Study**\n\nTo save space, we provide the **results of an ablation study** in our response to Reviewer pCEi. \n\nWe welcome further comments and discussions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The problem statement is unique. It will be useful for entity typing in highly specialized domains where information-rich domain-specific ontology is available.\n2. The paper shows the effectiveness of ontology-guided RAG as opposed to traditional RAG with strong results.\n3. The paper contributes 2 datasets which (once made public) will be useful for theme-specific entity typing research in future research.", "reasons_to_reject": "1. The  experiment setup does not quite reflect the problem statement. The problem statement aims to solve fine-grained entity typing under “”highly-specialized and fast-evolving themes” and yet, the experiment setup requires an information-rich ontology with type description from specific dictionaries and encyclopedias, aliases, KG triples, etc. as outlined in section 3.2.1. How is it possible to have this complete set of knowledge for a “fast-evolving theme”? It seems the paper makes a strong assumption that such information is readily available for emerging entity and entity types. In practice, it will be hard to satisfy this assumption.\n2. The experiment section is not extensive.  \n- I understand that it is time consuming to curate annotated data, I’d still expect evaluation to be done on more datasets.\n- Lack of experiments with open-sourced LLMs. The paper only used GPT-3.5 and GPT-4 for experiments which will be hard to reproduce. It’s worth exploring similar techniques for other open-sourced LLM so that experimental results can be reproduced.\n- Lack of ablation studies. For example, what happens if the “Semantic-rich sub-unit” is removed from the ontology, or the descriptions and the aliases are removed? Having an ablation study would help to better understand the contribution of each piece of information in RAG.\n\n*After rebuttal*\nThe authors have addressed my questions. I have updated my scores accordingly.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "16UTU2hcpE", "reviewer": "Reviewer_pCEi", "summary": "- This paper focuses on theme-specific entity typing. Compared to typical fine-grained entity typing, theme-specific entity typing requires expert knowledge to reason the entity types, which causes LLMs to be in trouble of hallucinations in two cases. First, unseen entities: an entity never appears in the pre-training corpora of LLMs. Second, misleading semantics: the context of an entity can potentially mislead an entity typing algorithm if the relevant knowledge is not correctly retrieved and utilized.\n- This paper clearly describes two technical challenges to address: (i) lack of expert knowledge to understand the theme ontology; and (ii) even with a large collection of expert knowledge, the ineffective retrieval based on text chunks may miss evidence for reasoning with the limited input length of LLMs.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback, especially on our technical contributions and experimental success.\n\nFor your concern about the **effectiveness of each technique**, we provide **ablation study** results on the battery theme in the following tables. The experiment analyzes the effectiveness of (i) the text unit, (ii) the knowledge unit, and (iii) the evidence expansion process.\n\n| Model          | Precision | Recall | F1   |\n|----------------|-----------|--------|------|\n| RAG-GPT-3.5    | 0.72      | 0.64   | 0.64 |\n| ORAG-GPT-3.5 (No text unit) | 0.79 | 0.78 | 0.78 |\n| ORAG-GPT-3.5 (No knowledge unit) | 0.75 | 0.58 | 0.53 |\n| ORAG-GPT-3.5 (No expansion) | 0.85 | 0.70 | 0.74 |\n| **ORAG-GPT-3.5** | **0.96** | **0.79** | **0.83** |\n\n| Model          | Precision | Recall | F1   |\n|----------------|-----------|--------|------|\n| RAG-GPT-4      | 0.89      | 0.64   | 0.75 |\n| ORAG-GPT-4 (No text unit) | 0.83 | 0.81 | 0.78 |\n| ORAG-GPT-4 (No knowledge unit) | 0.85 | 0.77 | 0.74 |\n| ORAG-GPT-4 (No expansion) | 0.87 | 0.85 | 0.81 |\n| **ORAG-GPT-4** | **0.99** | **0.90** | **0.94** |\n\nWe welcome further comments and discussions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Overall, the proposed framework achieves remarkable performance improvement.\n- Also, the proposed framework successfully deals with the problematic cases: (i) unseen entities and (ii) misleading semantics.", "reasons_to_reject": "- The proposed framework consists of two main techniques: (i) Structured Retrieval Memory Construction (Section 3.2) and (ii) Ontology-Guided RAG (Section 3.3). Unfortunately, this paper does not investigate the effectiveness of each technique. Not a few readers would like to know to what extent each technique contributes to performance improvement.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel approach, Ontology-guided Retrieval-Augmented Generation (ORAG), for theme-specific entity typing using large language models (LLMs). It addresses critical challenges in fine-grained entity typing, particularly with unseen entities and misleading semantics, demonstrating significant performance gains over traditional methods. The inclusion of two curated datasets and empirical validation on specific domains (e.g., battery materials, online gaming) highlights the effectiveness of the proposed approach. Since all reviewers have recommended accepting this paper, I recommend acceptance.
Decision: Accept