ORAG: Ontology-Guided Retrieval-Augmented Generation for Theme-Specific Entity Typing

Authors: jinfeng  xiao
linyi  ding
james  barry
mohab  elkaref
geeth  de mel
jiawei  han
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 20:17:05

Abstract: Large language models (LLMs) incorporated with retrieval-augmented generation (RAG) have shown great power in many NLP tasks, including fine-grained entity typing (FET). However, we observe that recent LLMs can easily suffer from hallucinations on highly specialized and fast-evolving themes (e.g., redox-active organic electrode materials), especially in the following cases: (1) unseen entities: an entity never appears in the pre-training corpora of LLMs; and (2) misleading semantics: the context of an entity can potentially mislead an entity typing algorithm if the relevant knowledge is not correctly retrieved and utilized. To address these challenges, this paper proposes an Ontology-Guided Retrieval-Augmented Generation (ORAG) approach that incorporates ontology structures with RAG for the theme-specific entity typing task. ORAG first enriches the label ontology with external knowledge and constructs a structured knowledge unit for each node. Then, it retrieves the relevant nodes by dense passage retrieval and expands the retrieved results based on the ontological structure. In this way, more supporting knowledge will be retrieved within the limited input of LLMs for entity typing. In the evaluation, we construct a dataset with two themes for theme-specific entity typing with a focus on unseen entities and misleading semantics. We observe notable cases of hallucination when vanilla RAG is applied to Llama-3, GPT-3.5, and GPT-4, while ORAG can effectively mitigate such hallucinations and improve the results.

Main paper: 
 introduction: 
 Retrieval-augmented generation (RAG) (Lewis et al., 2020) is an effective solution to deal with the limited input text length of LLMs when a task heavily relies on an external knowledge base like long documents or a large database, referred to as a retrieval memory. Given an LLM query, RAG first retrieves the relevant text chunks from the retrieval memory and then appends them to the query to generate answers. This RAG framework has shown its success on various NLP tasks.Fine-grained entity typing (FET) is the task of classifying entities into detailed types, where RAG can take the large label ontology as the retrieval memory. Existing datasets of FET such as OntoNotes (Gillick et al., 2014) and FIGER (Ling & Weld, 2021) mainly focus on common, general topics in the real world such as company, city, and actor, which are easy to understand by LLMs since the context and entity types frequently appear in the pre-training corpora.However, we observe that directly performing RAG with recent LLMs to zero-shot entity typing can easily lead to hallucinations on highly specialized or fast-evolving themes. This is observed with Llama-3 and GPT-4, which are among the most powerful open-source LLMs and commercial LLMs respectively.We take a passage under a highly specialized theme "redox-active organic electrode materials" as example. The passage discusses the effectiveness of a novel electrode material Cu-TCA compared to a bare electrode. The FET task is to find the type of the "bare electrode" in a given fine-grained label ontology for the theme. The correct type is "working electrode".Example Passage 1. CV data of Cu-TCA were recorded using a Li foil as counter electrode. . . . (Omitted) . . . The CV curve of a bare electrode (only containing the Al foil) proves that the above-mentioned redox areas originate from the active sites of the Cu-TCA structure.To reason the correct answer "working electrode", the model should (i) understand each type in the ontology to identify the correct type; and (2) understand the terms in the context such as counter electrode. RAG with GPT-4 gets the wrong answer "aluminum insertion electrode" because it semantically matches the misleading contextual information of the Al foil (which is an aluminum electrode) and Cu-TCA (which is an insertion electrode).Inspired by such cases, we propose the task of theme-specific entity typing. Compared to FET, our task focuses on narrow themes, especially highly specialized or fast-evolving themes, which require expert knowledge to reason the entity types. In this task, the hallucinations of LLMs are more likely to happen in: (i) unseen entities: new entities that never appear in the pre-training corpora of LLMs (e.g., the entity in Figure 1 ); and (ii) misleading semantics: the context can potentially mislead an entity typing algorithm if the relevant knowledge is not correctly retrieved and utilized. Challenges in RAG include (i) lack of expert knowledge to understand the theme-specific label ontology; and (ii) even with a large collection of expert knowledge, the ineffective retrieval based on text chunks may miss evidence for reasoning with the limited input length of LLMs.To tackle the challenges and mitigate LLMs' hallucinations for theme-specific entity typing, we propose the Ontology-guided Retrieval-Augmented Generation (ORAG)1 . The overall design is outlined in Figure 2 . For a better understanding of the types in the ontology, ORAG enriches the theme ontology with expert knowledge from multiple external sources. After the enrichment, each node in the theme ontology becomes a structured knowledge unit containing a knowledge-dense sub-unit and a semantic-rich sub-unit, which constitutes the structured retrieval memory. For more knowledge-dense retrieval, instead of dividing information into fixed-sized and incoherent text chunks, we retrieve the knowledge units with dense passage retrieval (DPR) (Karpukhin et al., 2020) and selectively add their subunits into the LLM's query. For more accurate reasoning, we expand the evidence set of retrieved nodes with their neighbors on ontology. For example, we add the definitions of child nodes to discriminate between the more fine-grained entity types. Finally, ORAG combines the evidence and the context to predict the entity type. In the above Example Passage 1, ORAG successfully predicts the correct entity type "working electrode". A more in-depth explanation of ORAG's success in this case is provided in Section 5.2.As no publicly available datasets focus on our settings, we construct a dataset of two themes: (i) a battery theme "redox-active organic electrode materials", and (ii) a game theme "online posts for Genshin Impact". The former represents highly specialized themes, which usually feature complicated ontologies and terminologies. The latter represents fast-evolving themes, where new entities and knowledge emerge so quickly (i.e., a significant update every six weeks) that the pre-training corpora of publicly available LLMs always lag behind the most recent version of the game.We compare the entity typing performance of ORAG versus vanilla RAG when applied to current state-of-the-art LLMs on the two themes. We observe that ORAG consistently provides a higher boost to Llama-3, GPT-3.5, and GPT-4. The effectiveness of ORAG is especially prominent in cases of unseen entities and misleading semantics. We summarize our contributions as follows:• We propose the theme-specific entity typing task, discuss two types of hard cases for LLMs, and construct a dataset of two themes focusing on such hard cases.• We propose ORAG, which performs structured retrieval on knowledge-enriched ontologies, to achieve more knowledge-dense retrieval and more accurate reasoning.• We experimentally demonstrate that applying ORAG to the theme-specific entity typing task can generate more accurate results and mitigate the hallucinations of LLMs compared to vanilla RAG, without requiring additional labeling efforts. 
fine-grained entity typing: 
 Compared to conventional entity typing, the label space of FET usually entails dozens or even hundreds of entity types, which is organized into a hierarchical ontology structure. Thus, FET faces the challenges of the large label space and limited accurate annotations.To deal with the data scarcity issue, ontologies and external knowledge are exploited and utilized in previous works. AFET (Ren et al., 2016) leverages the defined type hierarchy to formulate loss functions and integrates them into a unified optimization problem. ALIG-NIE (Huang et al., 2022) learns to relate entity types with vocabulary using the ontology and then uses a prompt-based method to enrich the few-shot training samples. SEType (Zhang et al., 2024) finds more instances for each seen type from an unlabeled corpus using pretrained language models. OnEFET (Ouyang et al., 2023) enriches the original ontology structure with instances and topics. These methods are in need of annotated instances and thus not developed under a zero-shot setting.For the zero-shot setting, a line of work utilizes linking from Wikipedia entries to the entities or types (Choi et al., 2018) . For example, ZOE (Zhou et al., 2018) uses the taxonomy of Freebase and types the entity by linking it to Wikipedia entries. The assumption that such links exist is a limitation of this line of work. Several LLM-based methods are also proposed for zero-shot FET. OntoType (Komarlu et al., 2023) proposes an ontology-guided framework that leverages pre-trained language models and head words to match the fine-grained types to a type ontology. Qin et al. (2023) and Hu et al. (2024) apply LLMs to zero-shot entity typing by prompt engineering. These studies focus on common entity types, and the assumption that LLMs know the types well limits their applicability to highly specialized or rapidly evolving themes.2.2 Knowledge-Grounded Retrieval-Augmented Generation RAG (Lewis et al., 2020 ) retrieves text chunks from text corpora to augment LLM queries. This technique has shown its power on various NLP tasks (Gao et al., 2023) . Besides unstructured text, structured knowledge like knowledge graphs (KGs) and ontologies can also be used as retrieval memory in the RAG framework. KG-RAG (Soman et al., 2023) applies RAG to biomedical KGs to facilitate domain-specific generation tasks. SURGE (Kang et al., 2023) trains graph neural networks to retrieve knowledge triples from a KG, which encourages the model to generate knowledge-grounded answers. KGLM (Logan et al., 2019) selects relevant facts from a KG with a KG embedding in each step of generation. KGQA (Sen et al., 2023) retrieves knowledge triples from a KG related to query entities by Hadamard product to augment the LLM input. REALM (Guu et al., 2020) retrieves documents from Wikipedia during the pretraining stage. EMAT (Wu et al., 2022) encodes external knowledge into a key-value memory to be retrieved and introduces a memoryaugmented transformer for QA tasks. While these works attempt to retrieve evidence from KGs or ontologies, most of them ignore or do not fully utilize structured information. How to better utilize structured and unstructured knowledge for FET remains less studied. 
problem formulation: 
 The conventional FET task mainly focuses on frequent and common entity types in the real world. Examples include company, city, and doctor in the FIGER dataset (Ling & Weld, 2021) , as well as date, money, and time in the OntoNotes dataset (Gillick et al., 2014) . Despite the large label space, the types are easy to understand for LLMs without requiring much additional expert knowledge (Ding et al., 2022) . In contrast, our targeted theme-specific entity typing task involves several less-studied challenges.(1) It focuses on narrow themes that need in-depth exploration for specific purposes.(2) It often deals with entities, types, or contexts that are unseen or long-tailed in the pre-training corpora of publicly available pre-trained LLMs.(3) It requires expert knowledge and reasoning on the themes to figure out the answers.For each theme, we assume its theme ontology (i.e., a label hierarchy) is available. As reviewed in Section 2.1, the availability of such an ontology is part of the problem definition of works on FET. Our task is to type the entities with the most proper fine-grained labels. Once a fine-grained type is identified, all parent types on the path from the root of the ontology to the fine-grained type are also considered as identified. Therefore, our task belongs to the family of hierarchical multi-label classification. Take the case in Figure 1 as an example. The theme ontology contains entity types like non-aqueous organic flow battery, which is the most proper fine-grained type of the underlined entity in the given context. Once this type is identified, all highlighted types in the ontology are considered as identified.For a highly-specialized theme, such as redox-active organic electrode materials, the raw label ontology may not provide sufficient information for reasoning the entity types by LLMs.Directly applying LLMs or RAG on the label hierarchy can easily lead to hallucination, especially for knowledge-intensive contexts and unseen entities since the reasoning would require specialized expert knowledge.To address the issue related to the lack of specialized information, we first enrich the label ontology with expert knowledge from external sources (Panel 1, Figure 2 ). For each entity type in the label ontology, we collect the following knowledge from various sources:i. The concise and formal definition from theme-specific dictionaries or encyclopedias can help establish a common understanding of highly specialized terms and avoid ambiguity. ii. Alias like abbreviations can help to recognize entities that may appear under different names and improve the recall rate of retrieval.iii. KG triples can provide possibly related entities that may be included in the context. The KGs usually contain many noisy relations. We filter the noisy relations with the similarity between the triples and the theme, calculated by a pre-trained encoder.2 iv. If some entity instances are available, we also collect them to help LLMs better understand long-tailed types. If rich information is available about such instances, we further add them into the ontology as leaf nodes under the corresponding type.v. To include comprehensive knowledge for better reasoning, we collect related descriptions from large corpora such as Wikipedia, which can provide multi-view information about the type. A description is usually longer than a definition.Note that we do not assume complete knowledge is available. In other words, we allow substantial misses in the above fields in most types, which is true in real-world applications. For example, in Figure 2 , the real available data miss some information for all types except the battery type and the redox flow battery type (e.g., the rocking chair battery type misses knowledge triples, the full flow battery type misses everything except a definition). Our algorithm handles partial information by inheriting information between parents and children in the ontology, as described in later sections.As shown in Figure 2 , vanilla RAG retrieves relevant text from its retrieval memory, which are directly split into fixed-size text chunks. However, the evidence for reasoning may be scattered in different chunks, and thus the retrieved chunks could include a considerable percentage of irrelevant context. With the limited input length of LLMs, including irrelevant context in one chunk means dropping other chunks that also contain some relevant information. Therefore, this type of naive retrieval can be inefficient in terms of the density and coverage of the relevant knowledge in the retrieved results. To effectively retrieve the related information, we construct a Structured Knowledge Unit for each entity type in the theme ontology (Panel 1, Figure 2 ).The structured knowledge unit of an entity type includes two parts:i. Knowledge-dense sub-unit: It includes the concise definition, alias, KG triples, and instances.ii. Semantic-rich sub-unit: It includes the descriptions about the entity type.We separate the two sub-units to balance knowledge density and semantic richness when we augment LLM queries in later steps. The knowledge-dense sub-units contain the most essential and concise information about the entity types. Although the semantic-rich subunit may be generally less important and informative, it may also provide subtle signals for reasoning if it is aligned with an LLM query. Thus, the handling of the two sub-units is different.We also observe that RAG has better performance on understanding and reasoning when the retrieval memory consists of complete sentences instead of key-value pairs. Therefore, we design patterns for each type of information. For example, for the alias of redox flow battery, we store the sentence Redox flow battery is also known as RFB in the retrieval memory instead of (alias, RFB). Similarly, we restate other information with complete sentences and finally construct the retrieval memory.3.3 Ontology-Guided RAGDifferent from vanilla RAG that searches text chunks, ORAG searches structured knowledge units. The motivation of ontology-guided RAG can be two-fold: (1) Structured knowledge units are more knowledge-dense and informative when compared to fixed-sized text chunks.With knowledge-dense units, ORAG can append more relevant evidence to query LLMs within the same input length to facilitate entity typing.(2) The retrieved information is self-coherent and effective and could contain less irrelevant information (which may mislead the LLMs to generate erroneous results (Wang et al., 2023) ).We use the embedding model3 from OpenAI to encode the sub-units and cluster the embeddings based on FAISS index (Johnson et al., 2019) . For dense passage retrieval (DPR) (Karpukhin et al., 2020) , given the LLM input query for entity typing, we rank the structured knowledge sub-units in a bi-encoder architecture.We retrieve the top-k entity types on the theme ontology regarding the rank of similarities of sub-units. If at least one sub-unit is retrieved, we consider the corresponding entity type to be a relevant node in the ontology. The entity types will be used as starting entities for further ontology-guided evidence expansion and selection.In this section, we further expand and select the information based on ontology and starting entities (i.e. redox flow battery and battery cell in Figure 2 ).The starting entity types are retrieved because they can be directly semantically matched by entity mentions or context. However, the one-hop semantically matching evidence may still be insufficient when the typing process requires reasoning, especially for unseen entities.Take the example in Figure 1 , the entity mention all-organic redox flow battery is a newly proposed entity unseen for LLMs. The mention is semantically matched to the type redox flow battery. However, it can be reasoned from the context that it belongs to a more finegrained type non-aqueous organic flow battery. In the vanilla RAG, the descriptions of the target type are not retrieved, and LLMs generate an inaccurate answer. The goal of this step is to include multi-hop information for more accurate reasoning on entity types. Given a starting entity, to discriminate the finer-grained entity type, we retrieve its top-N relevant child nodes. In the case above, the LLMs can give the right answer with the knowledge units of non-aqueous organic flow battery and aqueous organic flow battery. We also retrieve the knowledge-intensive sub-unit of its parent node to provide more information to discriminate the starting type and its sibling types. In this way, even if some knowledge about a type is missing, the information inherited from parent and child nodes can help.Due to the limited input length of LLMs, for most retrieved entity types, we only include the concise knowledge-dense sub-units in the augmented LLM query. The long semantic-rich sub-units are included only when their similarity to context is higher than the knowledgedense sub-unit, which means the descriptions can help understand the context. The evidence is gathered, and identical items are merged. Finally, the evidence and the context are fed into LLMs to perform typing. The query prompts are as follows, where ⟨entity⟩ and ⟨passage⟩ are placeholders for the corresponding entity and passage.What are the types of the entity ⟨entity⟩ in this passage? If there are multiple types, return them all. Format the output as type1, type2, . . . ⟨passage⟩ 
dataset construction: 
 We construct a dataset with two themes: one for redox-active organic electrode materials and the other for online game posts for Genshin Impact. The dataset have a focus on unseen entities and misleading semantics, two situations where LLMs are more likely to suffer from hallucinations. The former features highly specialized and complicated terminologies and ontology structures, whereas the latter features fast-evolving entities and knowledge that pre-training corpora of commercial LLM services almost always lag behind. The data on each theme include the following. (1) Corpus: a set of relevant documents that contain unseen entities and/or misleading semantics; (2) Ontology: a set of entity types arranged into a hierarchy describing the fine and coarse granularities; (3) Entity mentions: entity spans are located in the corpus by human annotators; (4) Entity types: the ground-truth entity types of the entity spans are identified from the given ontology by human annotators; and (5) Hard cases: A hard case is manually selected from each document, representing an entity mention that can hardly be typed correctly with its context only but correct typing becomes logically possible when theme-specific knowledge is included in the reasoning.In our experiments, such a hard case always features either unseen entities or misleading semantics. The statistics of the dataset are given in Table 1 .The first theme is a highly specialized battery theme of redox-active organic electrode materials.We use the Battery Interface Ontology (BattINFO) (Clark et al., 2022) which contains around 20,000 entity types. Entity annotations are performed on a set of research papers in this field with a focus on batteries and electrodes. All annotated entities have types under the two sub-ontologies rooted at the Battery type (with 298 nodes in the sub-ontology) and the Electrode type (with 186 nodes in the sub-ontology). These two sub-ontologies add up to 484 types in total. The structured retrieval memory is constructed from BattINFO, Wikipedia, Wikidata, and Electropedia4 which contains all the terms and definitions in the International Electrotechnical Vocabulary.The second theme is a fast-evolving game theme of online posts for Genshin Impact5 , a popular game winning the TGA best mobile game in 2021 and keeping fast-evolving on new entities. We collect bilingual passages from Chinese6 and English7 forums. We invited senior players with at least three years of gaming experience on Genshin Impact to construct the ontology (with 73 nodes) and perform annotation. The structured retrieval memory is constructed from Genshin Impact Wiki8 .We report the results of the following methods: (1) RAG-Llama-3: the open-source Llama-3-8B-Instruct model9 combined with vanilla RAG, (2) RAG-GPT-3.5: the GPT-3.5 with vanilla RAG, (3) RAG-GPT-4: the GPT-4 with vanilla RAG, (4) ORAG-Llama-3: the same Llama-3 with ORAG, ( 5) ORAG-GPT-3.5: the same GPT-3.5 with ORAG, and ( 6) ORAG-GPT-4: the same GPT-4 with ORAG. The pre-training data cutoff dates of both GPT-3.5 and GPT-4 are September 2021. All methods share the same prompts in Section 3.3.2.Table 2 shows the entity typing results on the two themes. We report the macro-average of precision, recall, and F1 scores. Given an LLM (Llama-3, GPT-3.5, or GPT-4), our ORAG improves the F1 scores over the vanilla RAG by 10% ∼ 24%. Remarkably, ORAG improves the precision of GPT-3.5 by 24% and 29% respectively, demonstrating the substantial efficacy of ORAG in reducing LLM hallucinations. ORAG also boosts the precision of GPT-4 to a nearly perfect level, providing hope for the complete elimination of hallucinations when the correct knowledge is used together with more advanced LLMs in the future. ORAG also boosts the performance of Llama-3 compared to vanilla RAG, demonstrating the applicability of our method to open-source LLMs.A notable improvement in recall is also consistently observed, while this improvement has a different indication than that in precision. Without ORAG, LLMs tend to predict coarse-grained types, which is easier to get right. Stopping at correct but coarse-grained types without going deeper into the ontology only hurts the recall but not the precision. That is why the precision scores are consistently higher than recalls. The rise of recall brought by ORAG demonstrates the usefulness of theme-specific knowledge in guiding LLMs into more fine-grained types.We also examined the performance of all models on the set of hard cases. Results are shown in Table 3 . While vanilla RAG performs much worse on hard cases compared to easy cases, ORAG brings the typing quality on hard cases much closer to the overall performance.In the battery theme, the performance of LLMs on hard cases is almost brought back to that on the average cases, indicating that ORAG can effectively handle unseen cases and misleading semantics. On the game theme, the performance of GPT-3.5 on hard cases after ORAG is applied still has a gap from its counterpart on the average cases. As this theme is evolving so fast (i.e., a significant update every six weeks) that a considerable portion of the up-to-date knowledge is missing from LLM pre-training corpora, the strong reasoning capability of very large models seems to play an essential role in handling hard cases. In the following, we discuss a case of an unseen entity from the game theme. In Example Passage 2, the ground-truth type of the entity "fruit platter" is Weapon → Catalyst (from coarse-grained type to fine-grained type). This weapon was first online in February 2022, so the GPT models with a training corpora cut-off date in September 2021 never saw this entity before. The entity "fruit platter" is a nickname of the weapon Kagura's Verity originating from the community, and is also not included in databases. Therefore, the correct entity type should be logically inferred from theme-specific knowledge and the context.In this case, the context discusses some characters and the weapons they can use. It can be semantically inferred from the context that the "fruit platter" refers to a Weapon. However, with expert knowledge about characters, we can further infer that "fruit platter" is Catalyst. RAG-GPT-4 only gives the coarse-grained entity type Weapon. The Weapon is generated because the term can be directly semantically matched in the context. Also, the RAG-GPT-4 lacks essential knowledge about context to infer the more fine-grained entity type.ORAG-GPT-4 correctly predicts the fine-grained type Catalyst. Although the Catalyst does not appear in the context, we still get its structured knowledge units by evidence expansion from its parent node Weapon. Also, ORAG retrieves the related information of two characters in the context, Miko Yae and Nahida, which reveals that both Miko and Nahida use Catalysts.With such information added to the query, GPT-4 becomes able to tell the correct type. 
a case of misleading semantics: 
 Example Passage 1 (Repeated). CV data of Cu-TCA were recorded using a Li foil as counter electrode. . . . (Omitted) . . . The CV curve of a bare electrode (only containing the Al foil) proves that the above-mentioned redox areas originate from the active sites of the Cu-TCA structure.In this section, we dive deeper into the "bare electrode" case in Example Passage 1 in Section 1 to analyze the output of experimented methods. We repeat the passage here for readers' convenience. The "bare electrode" has two ground-truth fine-grained types following the ontological paths: Electrode → Working Electrode, and Electrode → Active Electrode → Metal Electrode → Aluminum Electrode. (The Aluminum Electrode type is not discussed in Section 1 to avoid over-complication.) In this case, the correct typing requires a full understanding of the context and the types to avoid being misled by irrelevant context.RAG-GPT-4 predicts Aluminium Electrode and Aluminum Insertion Electrode, where the former is a correct fine-grained type but the latter is a hallucination. While an Insertion Electrode type can be inferred from the first half of Example Passage 1 and an Aluminum Electrode type can be inferred from the second half, combining the two pieces of semantics into an Aluminum Insertion Electrode type is scientifically wrong.ORAG-GPT-4 predicts Aluminium Electrode and Working Electrode, which are correct and fine-grained. The algorithm retrieves the structured knowledge unit of the Working Electrode through its evidence expansion process, as the Working Electrode is a child type of Electrode. It also retrieves the unit of Insertion Electrode through its parent Active Electrode.With the relevant knowledge added to the query, GPT-4 becomes able to identify that the Working Electrode is a match while the Insertion Electrode is not. 
conclusion: 
 In this work, we introduce ORAG, a theme-specific fine-grained entity typing framework that works well on cases of unseen entities and misleading semantics that are hard for LLMs. Compared to vanilla RAG, our proposed ORAG gives more precise entity types with structured retrieval from knowledge-enriched theme-specific ontology. During the evaluation of a highly specialized theme on electrode materials and a rapidly evolving theme on game posts, we find that our ORAG yields more accurate theme-specific entity typing results compared to the vanilla RAG method. 
 
 Appendix: Additional resources about this work are available at https://github.com/JinfengXiao/ORAGhttps://huggingface.co/sentence-transformers/all-MiniLM-L6-v2https://platform.openai.com/docs/guides/embeddingshttps://www.electropedia.org/https://en.wikipedia.org/wiki/Genshin_Impacthttps://ngabbs.com/https://www.hoyolab.com/homehttps://genshin-impact.fandom.com/wiki/Genshin_Impact_Wikihttps://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct 


Decision: The paper introduces a novel approach, Ontology-guided Retrieval-Augmented Generation (ORAG), for theme-specific entity typing using large language models (LLMs). It addresses critical challenges in fine-grained entity typing, particularly with unseen entities and misleading semantics, demonstrating significant performance gains over traditional methods. The inclusion of two curated datasets and empirical validation on specific domains (e.g., battery materials, online gaming) highlights the effectiveness of the proposed approach. Since all reviewers have recommended accepting this paper, I recommend acceptance.

Reviews: review: paper_summary: paper_summary: The paper proposes an Ontology guided RAG for the task of theme specific fine-grained entity typing. The paper argues that traditional datasets for entity typing, does not capture scenario with unseen entities or situations with misleading semantics, and constructs two datasets for evaluation under this scenario. Further, the paper proposes an ontology guided RAG algorithm, where the labels are enriched with expert knowledge and RAG is performed on top of these enriched ontology. Experimental evaluation shows ORAG leads to 10-24% improved F1 score as compared to traditional RAG.

The paper has good quality, clarity and originality. However, evaluation is done on a very small dataset of 20 documents, raising questions on the generalizability of the results.

Based on new results presented by the author in the rebutal phase, where they show generalizability of their approach to an existing CHEMNER dataset, I have updated my rating to "6: Marginally above acceptance threshold"
strength_weakness: strength_weakness: 1. Evaluation is done on a very small dataset of 20 documents, raising questions on the generalizability of the results.
2. Even the choice of themes - a) redox-active organic electrode materials, and b) online game posts for Genshin Impact, seems arbitrary without proper justification on how learnings on those datasets showcase generalizability.
paper_summary: paper_summary: This manuscript discusses an approach for improving fine-grained entity typing in a zero/few shot setting with LLMs, specifically GPT-4, by combining retrieval-augmented generation (RAG) with a knowledge base generated from a domain-specific ontology. The application area is chosen to be "highly specialized or fast-evolving themes," with the idea being that general models would not have been trained sufficiently in these areas. This is because the relevant distinctions would have appeared after the training cutoff for the fast-evolving themes, and because the relevant distinctions for highly specialized domain would be too specialized to appear in the training data.

The basic approach is to use the ontology to construct a knowledge base consisting of a "knowledge unit" for each node. The system then uses a RAG approach with a untuned LLM to perform the task, but in addition to returning the knowledge unit that best matches the query, the search component also returns the knowledge units for closely related nodes. The evaluation selects one highly specialized domain (redox-active organic electrode materials, a type of battery) and one fast-evolving domain (Genshin Impact, an online game). The battery domain uses the Battery Interface Ontology, which contains approximately 20,000 entity types. The online game does not have an ontology, so the authors work with "senior players" to create one, with 73 entity types, and annotate documents. The evaluation corpus for the battery domain consists of 20 "research papers in this field with a focus on batteries and electrodes." The corpus contains 100 annotated entities (mentions?) and 484 types, note this implies multiple types per mention. The evaluation corpus for the online game domain also contains 20 documents, in English and Chinese, with 200 entities (mentions?) and 73 types. Both evaluation datasets list one "hard case" for each document, essentially a mention that cannot be typed correctly without using knowledge from the ontology. The evaluation uses a straightforward prompt and considers GPT3.5 and GPT-4 with and without ORAG, demonstrating substantial improvements for precision, recall and f1-score in all cases. The discussion focuses on two case studies, one "unseen entity" (a nickname for a weapon in the online game that would only make sense in the context of the game) and one "misleading semantics" (a case where the specific discussion in context would seem to suggest a different, incorrect, type).
strength_weakness: strength_weakness: There are several issues with the manuscript that reduce my enthusiasm for it. 

First, the evaluation corpora are quite small: each is only 20 documents and the total number of entities (I think they mean mentions) is only 300. Thus, even with the substantial performance improvements described, these evaluations should not be considered definitive, even for the two domains described. The results for "hard cases" are even more uncertain, since they are based on only 20 mentions for each domain. I believe error bars on the evaluation (Tables 2 & 3) would help substantially, but even so, I think this work is best described as a pilot study - a successful one that clearly suggests the approach is worth large-scale evaluation, but still a pilot study. 

Second, only two domains are considered. There are many others which this method should help, such as chemical, biological, and clinical research, which have substantial datasets that should be adaptable to FET. It is possible, however, that the authors do not consider these to be sufficiently specialized or fast moving. In any case, with only two domains considered, it is likely that there are substantial issues in some domains that would complicate its use or cause it to fail. Again, I think this work is best described as a successful pilot study. 

It is also somewhat unclear in what applications this approach would be most useful. It requires a fairly comprehensive ontology, which must be created if it does not already exist, which implies a substantial effort. This effort itself would require substantial time and expense to evaluate examples from documents, which suggests that at least in the case of domains that do not already have an ontology, the approach should ideally be compared to creating a small training corpus and fine-tuning an LLM. The approach also requires one LLM prompt per document, mention pair, so likely cannot be applied to large-scale datasets with GPT-4: if we assume the prompt and response are 1000 tokens on average (750 words) and 100 mentions per document, then a dataset of only 10,000 documents would cost $45,000 (this analysis ignores the RAG component, which I assume would also cost tokens). Finally, fine-grained entity typing itself is probably best understood as an intermediate natural language processing task, one that would be useful in a pipeline but one that users are not likely to care about directly. The most straightforward application users might care about, building or expanding an existing knowledge base, would likely run into problems scaling. At the very least, it would be helpful to suggest some user-facing applications.

In addition to the evaluation being small, the manuscript provides only a very superficial description of the process for creating the evaluation datasets. Not described are: how the documents were selected, the annotation process (the number of annotators per document, how disagreements were handled, etc.), the annotation guidelines (what constitutes a mention and what does not), inter-annotator agreement, how the online game ontology was created (e.g. the number of entity types in the ontology is the same as the number in the corpus, suggesting that the ontology was created from the corpus), whether the knowledge units for the battery domain included all 20,000 entity types or only the 484 in the dataset. All of these are relevant for understanding the quality of the evaluation datasets.

The manuscript does not clarify how performance was measured. Is the performance for each mention (i.e., two mentions of "fruit platter" in a single document would count twice), or for each (document, mention text) pair, or something else? How are true positives calculated, given that each mention may be annotated with multiple entity types, each of which is hierarchical? The authors allude to a hierarchical evaluation in section 4 ("Stopping at correct but coarse grained...") but do not provide details. 

Some may argue that the straightforward approach described is too incremental or obvious to be useful. While I know of related methods, I am not aware of any that combine ontologies with RAG so neatly and in a way that would be useful in a zero-shot setting. Thus, I specifically argue that any such argument against this manuscript should be accompanied by a citation of one or more works that allow an LLM to use RAG with an ontology in a zero-shot setting. 

There are several clarifications or terminology changes that would be helpful:

Fine-grained entity typing (FET) is not as well-known as named entity recognition (NER) and entity linking (EL). A brief contrast between them would help the reader better understand the needs of the task. In this case, NER typically involves predicting span and rough entity type, while EL involves predicting a specific identifier from a controlled terminology or lexicon that is potentially very large (millions of entities). Fine-grained entity typing does not involve predicting the span, but does involve predicting a type from a set of types which are intermediate in size between the rough entity types of NER and the identifiers in EL. FET may also have a context dependency such that a particular aspect or role of the entity is emphasized, such as classifying a mention of a person that is a politician, business person and media personality as referring in context specifically to "person/politician"; in this sense FET may be more specific than even EL (see, e.g., Dai et al., EMNLP-IJCNLP 2019, Improving Fine-grained Entity Typing with Entity Linking https://aclanthology.org/D19-1643)

The authors use the word "entity" ambiguously, to refer to the type from the ontology and also to the text from the document that should be typed. The latter is more typically known as a "mention," and I strongly suggest updating this terminology since it makes the manuscript difficult to read and makes some of the descriptions unclear (e.g., doe the "entities" in Table 1 refer to mentions?)

The authors use the word "theme" where I believe that the field more typically uses the word "domain." This is not critical, since I don't think there is any ambiguity, but it would align the manuscript better with the field.

Finally, while the LLM literature uses the word "hallucination" rather liberally, this manuscript not only does not define it but uses it to describe almost any sort of error made by the LLM. This sort of imprecise language makes understanding the problems being addressed and analyzing the results more difficult. It also fails to make one of the strongest points in favor of this approach: Subbarao Kambhampati (e.g. https://twitter.com/rao2z/status/1772636256184520981) talks about LLMs being able to capture a distribution, but correctness is a property of an instance; by extension, "hallucination" is where the LLM returns something that sounds correct because it follows the distribution of the domain, but is not correct because the LLM has no knowledge of the entities within the domain. Thus, the approach described by the authors neatly allows the ontology to inform the LLM of the entities, which should dramatically reduce hallucinations, according to this definition, but the terminology used in the manuscript is not precise enough to support this point. (I'm also unsure whether the task would even support this sort of analysis.)
questions: questions: Could you describe some concrete applications where this approach would be useful to users?

Please clarify the evaluation metrics

What are the error bars on the evaluation metrics from Table 2 & 3?

Please clarify the document-selection process, annotation guidelines, annotation process, inter-annotator agreement and ontology creation process. 

Please clarify whether the full battery ontology was used during the evaluation or only the part relevant to the 484 types in the dataset.

How do you define "hallucination"?

How well would this approach scale?

How applicable would this approach be to other domains? How do you know?
paper_summary: paper_summary: The paper proposes a new task called theme-specific entity typing which is related to the typical entity typing task problems studied in NL, but for a specific theme or domain such as online gaming. The paper aims to guide the LLM generation with information-rich ontologies. It organizes the knowledge in a hierarchical ontology with each node representing an ontology type along with knowledge-dense sub-units and semantic-rich sub-units. The retrieval happens at the ontology’s node level as opposed to the chunk of text used in regular RAG.  The paper curated two datasets with human annotations and the corresponding ontologies. Experimental results show that the ontology-guided RAG makes GPT-3.5 and 4 models perform better.
strength_weakness: strength_weakness: 1. The  experiment setup does not quite reflect the problem statement. The problem statement aims to solve fine-grained entity typing under “”highly-specialized and fast-evolving themes” and yet, the experiment setup requires an information-rich ontology with type description from specific dictionaries and encyclopedias, aliases, KG triples, etc. as outlined in section 3.2.1. How is it possible to have this complete set of knowledge for a “fast-evolving theme”? It seems the paper makes a strong assumption that such information is readily available for emerging entity and entity types. In practice, it will be hard to satisfy this assumption.
2. The experiment section is not extensive.  
- I understand that it is time consuming to curate annotated data, I’d still expect evaluation to be done on more datasets.
- Lack of experiments with open-sourced LLMs. The paper only used GPT-3.5 and GPT-4 for experiments which will be hard to reproduce. It’s worth exploring similar techniques for other open-sourced LLM so that experimental results can be reproduced.
- Lack of ablation studies. For example, what happens if the “Semantic-rich sub-unit” is removed from the ontology, or the descriptions and the aliases are removed? Having an ablation study would help to better understand the contribution of each piece of information in RAG.

*After rebuttal*
The authors have addressed my questions. I have updated my scores accordingly.
paper_summary: paper_summary: - This paper focuses on theme-specific entity typing. Compared to typical fine-grained entity typing, theme-specific entity typing requires expert knowledge to reason the entity types, which causes LLMs to be in trouble of hallucinations in two cases. First, unseen entities: an entity never appears in the pre-training corpora of LLMs. Second, misleading semantics: the context of an entity can potentially mislead an entity typing algorithm if the relevant knowledge is not correctly retrieved and utilized.
- This paper clearly describes two technical challenges to address: (i) lack of expert knowledge to understand the theme ontology; and (ii) even with a large collection of expert knowledge, the ineffective retrieval based on text chunks may miss evidence for reasoning with the limited input length of LLMs.
strength_weakness: strength_weakness: - The proposed framework consists of two main techniques: (i) Structured Retrieval Memory Construction (Section 3.2) and (ii) Ontology-Guided RAG (Section 3.3). Unfortunately, this paper does not investigate the effectiveness of each technique. Not a few readers would like to know to what extent each technique contributes to performance improvement.

Comments: title: Rebuttal acknowlegement
comment: Thank you for your positive and constructive feedback!
Thank you for considering our response and raising your score!
Thank you for addressing my concerns. I appreciate the additional effort of running experiments on LLaMa 3 and doing the ablation study. I'll update my score.
Thank you for the response.
I think that the ablation study makes clearer how much each technique contributes to the final results.
Thanks again.
Thank you for your constructive feedback and for raising your rating!
Thank you for the response. Many of these make good sense, and I assume that there will be (minor) updates to the manuscript. I remain concerned about the size of the evaluation. I really do not believe it would be a good thing for the field to encourage additional work with such a small evaluation. And I am now concerned about the annotation process, which seems prone to bias. However, the ChemNER evaluation helps, and this work has my interest despite continued concerns. I also see that you adopted the "succcessful pilot study" phase, which I think does help frame the work. I will update my recommendation slightly.