KnowGPT: Knowledge Graph based Prompting for Large Language Models

Authors: qinggang  zhang
junnan  dong
hao  chen
daochen  zha
zailiang  yu
xiao  huang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 04:57:45

Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in many real-world applications. Nonetheless, LLMs are often criticized for their tendency to produce hallucinations, wherein the models fabricate incorrect statements on tasks beyond their knowledge and perception. To alleviate this issue, graph retrieval-augmented generation (GraphRAG) has been extensively explored which leverages the factual knowledge in knowledge graphs (KGs) to ground the LLM's responses in established facts and principles. However, most state-of-the-art LLMs are closed-source, making it challenging to develop a prompting framework that can efficiently and effectively integrate KGs into LLMs with hard prompts only. Generally, existing KG-enhanced LLMs usually suffer from three critical issues, including huge search space, high API costs, and laborious prompt engineering, that impede their widespread application in practice. To this end, we introduce a novel **Know**ledge **Gr**aph based **P**romp**T**ing framework, namely **KnowGPT**, to enhance LLMs with domain knowledge. KnowGPT contains a knowledge extraction module to extract the most informative knowledge from KGs, and a context-aware prompt construction module to automatically convert extracted knowledge into effective prompts. Experiments on three benchmarks demonstrate that KnowGPT significantly outperforms all competitors. Notably, KnowGPT achieves a 92.6% accuracy on OpenbookQA leaderboard, comparable to human-level performance.

Main paper: 
 introduction: 
 Large Language Models (LLMs), such as GPT-4 [54] and Claude 3 [3] , have surprised the world with superior performance in a wide range of real-world applications [14, 38, 82, 83] . Despite their impressive performance, LLMs are frequently criticized for their limited ability to handle factual information accurately and their tendency to generate hallucinations [18] , especially when faced with questions requiring domain-specific or professional knowledge not covered in their training corpus [2, 59] . For example, when queried about nutrient composition, an LLM might erroneously associate it with "energy", as depicted in Figure 1 . This error stems from the model's insufficient biological knowledge. Therefore, integrating domain knowledge into LLMs is crucial for reducing hallucinations and unlocking their full potential in diverse industry applications [24] . reports [42] . These textual documents may have varying levels of quality, accuracy, and completeness, leading to potential inconsistencies or errors in the retrieved knowledge [92] . A promising avenue for addressing the above issue entails the integration of Knowledge Graphs (KGs) into LLMs. KGs provide a structured representation of domain knowledge, as they are constructed based on rigid ontologies that clearly define the jargon, acronyms, specialized terminologies and their relationships in specific domains [43, 60, 84, 85, 86] . The enormous factual knowledge stored in KGs holds the potential to ground the model's responses in established facts and principles [29, 77, 56] . For instance, in Figure 1 , an LLM can correct itself by leveraging the related background knowledge in ConceptNet [62] . Earlier studies adopted a heuristic way to inject knowledge from KGs into the LLMs during pre-training or fine-tuning. For example, ERNIE [63] incorporates entity embeddings and aligns them with word embeddings in the pre-training phase, encouraging the model to better understand and reason over entities. Similarly, KnowBERT [57] integrates entity linkers with BERT to infuse knowledge about entities during fine-tuning for knowledge-intensive applications. Another line of work focuses on retrieving relevant knowledge from KGs at inference time to augment the language model's context. Typically, K-BERT [47] uses an attention mechanism to select relevant triples from KGs based on the query context, which are then appended to the input sequence. Similarly, KEPLER [73] learns joint embeddings of text and KG entities to enhance the model's predictions. Subsequent works have further integrated graph neural networks alongside LLMs for joint reasoning [80, 87, 15] and introduced interactions between text tokens and KG entities within the intermediate layers of LLMs [64, 67] .However, as LLMs have been keeping evolving, most SOTA LLMs remain closed-source in practice. For instance, GPT-4 [54] and Claude 3 [3] exclusively grant access through their APIs, which means we can only retrieve model responses by submitting textual inputs, with model specifics inaccessible. As such, the research focus has recently shifted towards KG prompting that enhances fixed LLMs with KG-based hard prompts [46, 56] . KG Prompting for LLMs has been a new learning paradigm in natural language processing. Specifically, CoK [71] introduces Chain-of-Knowledge prompting to decompose LLM-generated reasoning chains into evidence triples, verifying their factuality and faithfulness using an external KG. Mindmap [74] provides more transparency on LLMs' decisionmaking by enabling comprehension and reasoning over structured KG inputs. RoG [50] presents a planning-retrieval-reasoning framework that synergizes LLMs and KGs for more transparent and interpretable reasoning. KGR [25] proposes an autonomous approach to retrofit LLM-generated responses by leveraging KGs to extract, verify, and refine factual information throughout the reasoning process, effectively mitigating hallucination for knowledge-intensive applications.Despite the promising performance of existing KG prompting methods, three critical issues hinder their widespread application in practice. ❶ Huge search space. Real-world KGs often consist of millions of triples, resulting in a vast search space when retrieving relevant knowledge for prompting. ❷ High API cost. Closed-source LLMs, like GPT-4 and Claude 3, are accessible through proprietary APIs, which can incur significant costs when performing KG prompting at scale [17] . Thus, careful selection of the most informative knowledge from KGs is essential to minimize costs. ❸ Laborious prompt design. LLMs are highly sensitive to prompts, with even minor variations in prompts conveying the same semantic meaning potentially yielding drastically different responses. However, existing methods rely on manually designed or rule-based prompts to present factual knowledge from KGs. These hard prompts are inherently inflexible and rigid, lacking the adaptability to accommodate variations in question semantics and KG structures.To this end, we propose a novel Knowledge Graph-based PrompTing framework, namely KnowGPT, which leverages the factual knowledge in KGs to ground the model's responses in established facts and principles. In this paper, we aim to answer two key research questions. ❶ Given a query and a large-scale KG, how could we effectively and efficiently retrieve factual knowledge from KG that is Extracted paths1 1 1 2 … 2 4𝒫/0 relevant to the query? ❷ Given the raw knowledge extracted from KGs, how could we convert the extracted knowledge into an effective prompt that is easily understandable for LLM?We shed light on the the above questions with a novel prompt learning method that is rather effective, generalizable, and cost-efficient. Specifically, to address question ❶, we leverage deep reinforcement learning (RL) to extract the most informative knowledge from KGs. To encourage the agent to discover more informative knowledge chains, we devise a tailored reward scheme that promotes the reachability, context-relatedness, and conciseness of the extracted paths. Then, a policy network is trained to maximize the reward using training questions and applied to unseen questions. To tackle question ❷, we introduce a prompt construction strategy based on Multi-Armed Bandit (MAB).Given several knowledge extraction strategies and prompt templates, an MAB is learned to select the most effective combination for each question by balancing exploration and exploitation. The learned MAB is then applied to new questions to select knowledge extraction strategies and prompt templates automatically. Our main contributions are summarized as follows:• Formally define the problem of KG-based prompting, which leverages the structured knowledge in KGs to ground the LLM's responses in established facts and principles. • Propose KnowGPT, a novel prompting framework that leverages deep reinforcement learning (RL) and Multi-Armed Bandit (MAB) to generate effective prompts for domain-specific queries. • Implement KnowGPT upon GPT-3.5. Experiments on three QA datasets shows KnowGPT outperforms SOTA baseline models by a large margin. Notably, KnowGPT achieves an average improvement of 23.7% over GPT-3.5 and an average improvement of 2.9% over GPT-4. Additionally, it attains a 92.6% accuracy on the OpenbookQA leaderboard, which is comparable to human performance.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We have clearly made the main claims in both the abstract and introduction. Specifically, we have summarized and itemized the contributions at the end of the introduction section.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
problem statement: 
 We formally define the problem of knowledge graph based prompting for LLMs in question answering. We represent each question as a question contextQ = {Q s , Q t }, where Q s = {e 1 , e 2 , ..., e m } is a set of m source entities, and Q t = {e 1 , e 2 , ..., e n } is a set of n target entities. Following prior work [20, 79] , Q s is extracted by concept recognition, and we assume it is given in our problem. Similarly, each target entity in Q t is extracted from a corresponding candidate answer.We denote an LLM as f , a real-world KG as G, which consists of triples (head entity, relation, tail entity), denoted as (h, r, t). In our setting, we only have access to the APIs of f . However, we can employ open-source lightweight language models (not f ), like Bert-Base [35] , to initialize question embeddings. Using the above notations, we describe our problem below. Given a question Q, an LLM f , and a domain KG G, we aim to learn a prompting function f prompt (Q, G), which generates a prompt x that incorporates the context of Q and the factual knowledge in G, such that the prediction of the LLM f (x) can output the correct answers for Q.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Please see page 9, Section 5 for the Discussion and Limitations section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
knowgpt framework: 
 Learning the prompting function f prompt (Q, G) involves two challenges, i.e., what knowledge should be used in G, and how to construct the prompt. To address these challenges, we present KnowGPT, which extracts raw knowledge with deep RL and then constructs the prompt with MAB. An overview of our framework is shown in Figure 2 .Intuitively, the relevant reasoning background lies in a question-specific subgraph G sub that contains all the source entities Q s , target entities Q t , and their neighbors. An ideal subgraph G sub is expected to have the following properties: (i) G sub encompasses as many source and target entities as possible, (ii) the entities and relations within G sub exhibit a strong relevance to question context, and (iii) G sub is concise with little redundant information such that it can be fed into LLMs with limited lengths.However, it is challenging to find such a G sub since extracting a subgraph is NP-hard. • State: A state indicates the current location in KG, i.e., one of the entities in KG. Specifically, it represents the spatial change from entity h to t. Inspired by the prior study [76] , we define the state vector s as:EQUATIONwhere e t and e target are the embedding vectors of the current entity and the target entity. To get the initial node embeddings for entities extracted from the background KG, we adopt the approach proposed by the previous study [20] . Specifically, we transform knowledge triples from the KG into sentences and feed them into pre-trained LM to get node embeddings.• Action: The action space encompasses all the neighboring entities of the current entity, enabling the agent to explore the KG flexibly. By taking an action, the agent will move from the current entity to the chosen neighboring entity.• Transition: The transition model P measures the probability of moving to a new state (s ′ ) given existing state (s) and the undertaken action (a). In KGs, the transition model takes on the form P(s ′ |s, a) = 1 if s is directed to s ′ through action a; Otherwise, P(s ′ |s, a) = 0.• Reward: To determine the quality of the formed path, we define the reward based on reachability:EQUATIONwhich represents whether the path eventually reaches the target within limited steps. Specifically, the agent receives a reward of +1 if it can attain the target within K actions. Otherwise, it will receive -1 as the reward.Reaching a target entity is not our sole focus. To avoid overlong and rigmarole reasoning chains, we also design two auxiliary rewards to promote context-relatedness and path conciseness.The key motivation is to encourage paths closely related to the given question context. Specifically, we evaluate the semantic relevance of a path P i to the context Q. Inspired by the prevailing study [80] , a fixed but well-trained matrix W is applied to map the path embedding P to the same semantic space with context embedding c. To this end, this auxiliary reward is formulated as:EQUATIONwhere c is the embedding of context Q we obtained from a pre-trained LM [35] and the embedding of path P i is the average of the embeddings of all the entities and relations we have walked through till i, i.e., Avg(e source + re 1 ... + e i ), where i ≤ length(P target ). This step-by-step reward scheme provides rewards before the target is reached.There are two additional significant challenges for the candidate reasoning background. (i) The natural limitation of LLMs for over-long context understanding gives constrained budgets for prompts, where the extracted knowledge chain is expected to be concise enough to ensure the full understanding by closed-source LLMs. (ii) The prohibitive cost of calling LLMs' API guides the prompt to be more concise. By limiting the step size, we encourage the policy to find as much valuable information as possible within the shortest path length.Considering the inevitable homogeneity in the large-scale real-world KG constructed from the online corpus, each step in the final path is ideally a necessity. Specifically, we evaluate the conciseness of a path to reduce twists and turns on redundant entities, e.g., synonyms. Thus, the reward for the conciseness of a path P i is formulated as follows.EQUATION)To this end, our overall reward modeling consists of three major criteria that comprehensively incentivize the entire policy learning for an effective knowledge extraction.To solve the MDP defined above, a tailored policy network π θ (s, a) = p(a|s; θ) is trained to extract a reasoning chain in the KG. We optimize the network with policy gradient [76] . The optimal policy navigates the agent from the source entity to the target entity while maximizing the accumulated rewards. We provide more training details in the Appendix.In this subsection, we design a tailored prompt construction strategy based on Multi-Armed Bandit (MAB). The key idea is to learn to select the best knowledge extraction and prompt templates at a meta-level. We will begin by outlining the overall strategy, followed by detailing its instantiation with two knowledge extraction methodologies and three templates.Suppose we have several knowledge extraction strategies {P 1 , P 2 , ..., P m } and several candidate prompt formats F = {F 1 , F 2 , ..., F n }. Each knowledge extraction strategy P i is a method for selecting reasoning background given a question context, such as the RL-based strategy discussed above. Every prompt template F j represents a mechanism to transform the triples within the subgraph into a prompt for an LLM prediction.The prompt construction problem is to identify the best combination of P and F for a given question. We define the overall process of selection as a reward maximization problem, max r pf , where r pf is obtained as:EQUATIONSpecifically,PF (i) , i ∈ {0, 1, • • • , m× n} is one of the combination, and r pf ∈ {0, 1} indicates the performance of the output of LLM in answering the current question.To capture the context-aware correlation between questions and different combinations of knowledge and prompt formats, we formulate the selection mechanism of MAB with an expectation function E(•). It adaptively measures the potential expectation of a combination for different questions.EQUATIONHere, c represents the embedding of Q. The vector α(i) corresponds to a set of non-negative parameters associated with PF(i), which have been learned during the previous k-1 iterations. Additionally, β (i) stands for a balancing factor introducing noise according to a Gaussian distribution.Empirically maximizing c × α i could encourage exploitation [12, 16] for the best combination, we could effectively update α (i) via modeling the correlations between the context embedding of the anchor question c i and all the previously selected contexts C (i) for particular combination PF (i) in former k steps, and the rewards r(i)pf obtained from the selection of the current combination. Concretely, the β (b) is updated as:EQUATIONHere, J denotes the OLS training loss. I ∈ R d×d is an identity matrix and λ i is a regularization factor that controls the complexity of the model.Similarly, in order to encourage exploration within less frequently selected pairings, we employ an upper confidence bound approach to balance exploration and exploitation. This is achieved through the introduction of the parameter β (i) . Inspired by prevailing studies [70, 16] , we can derive the following exploration term β (i) :EQUATION)where γ is a fixed constant, i.e., γ = 1 + ln(2/δ)/2.When the model picks a combination with a large c×α i , it signifies an exploitation process. Likewise, when the model selects a combination with larger β (i) , this variance indicates an exploration process due to the model making fewer selections of the current combination. Thus, jointly maximizing c × α i + β (i) could help us get rid of the dilemma of exploration and exploitation.Consequently, our MAB design can leverage the feedback from the LLM to optimize the selection policy. By maximizing the expectation function E(•), it learns to balance the exploitation and exploration to prioritize the most promising prompts for specific question contexts.We implement the above MAB strategies with two knowledge extraction strategies and three templates. Note that our MAB design is general and can be implemented with more knowledge extraction strategies and prompt templates for better performance. The knowledge extraction strategies include:• P RL : The RL-based knowledge extraction strategy presented in the previous subsection.• P sub : A heuristic sub-graph extraction strategy that extracts a 2-hop subgraph around both the source and target entities. Detailed implementation can be found in Section B.1 of Appendix. Since RL is notoriously unstable [65] , we introduce P sub as an alternative candidate strategy for the MAB selection, ensuring a fallback option if the RL-based approach does not perform well.The prompt templates include:• Triples, denoted as F t , are indeed the originally extracted knowledge and empirically tested that could be understood by the black-box LLMs, e.g., (Sergey_Brin, founder_of, Google),(Sundar_Pichai, ceo_of, Google), (Google, is_a, High-tech Company). • Sentences is a following solution to transform the knowledge into a colloquial F s , e.g., 'Sergey Brin, who is a founder of Google, a high-tech company, has now passed the reigns to Sundar Pichai, who is currently serving as the CEO of the company.' • Graph Description, F g prompts the LLM by treating the knowledge as a structured graph. We preprocess the extracted knowledge with black-box LLM itself to generate the description by highlighting the center entity, e.g., 'Google, a high-tech company, stands central in the network.The entity is strongly associated with significant individuals in the tech industry. Sergey Brin, one of the founders, established Google, underscoring its historical beginnings. In the present graph context, Sundar Pichai is recognized as the CEO of Google, symbolizing the company's current leadership. Thus, Google serves as a vital link between these key figures.'Considering two knowledge extraction methods: P sub and P RL , as well as three prompt translation methods: F t , F s and F g , the MAB is trained to learn from the feedback from LLMs to prioritize the most appropriate combination among two extraction methods and three predefined prompt formats for different real-world question contexts, i.e., PF = {(P sub F t ), (P sub F s ), (P sub F g ), (P RL F t ), (P RL F s ), (P RL F g )}.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Code and datasets are accessible through this anonymous URL: https: //anonymous.4open.science/status/KnowGPT-DD64. In this repository, we included detailed instructions for running the code in the readme.md file.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 We conduct extensive experiments to evaluate KnowGPT on three benchmark question-answering datasets, covering both commonsense and domain-specific QA. We implement KnowGPT upon GPT-3.5. Our experiments are designed to answer the following research questions:• RQ1 (Main results): How does KnowGPT perform when compared with the state-of-the-art LLMs and KG-enhanced QA baselines?• RQ2 (Ablation Study): How does each key component of KnowGPT contribute to the performance?• RQ3 (Case study): How could KG help solve complex reasoning tasks? See Appendix 4.4.Datasets. We evaluate KnowGPT on three QA datasets spanning two fields: CommonsenseQA [66] and OpenBookQA [52] serve as benchmarks for commonsense reasoning, while MedQA-USMLE [34] acts as a domain-specific QA benchmark. The statistics of these three datasets can be found in Table 5 in the Appendix.Baselines. We carefully select baseline models from four categories for a comprehensive evaluation.LM + Fine-tuning. We compare our method with vanilla fine-tuned LMs. Specifically, we choose Bert-base, Bert-large [35] , and RoBerta-large [49] as representative fine-tune LM methods. To conduct commonsense and biomedical QA, we fine-tune these three LMs via additional linear layers.KG-enhanced LM. We have also implemented several recently released models for integrating KGs into question answering, which encompass MHGRN [20] , QA-GNN [80] , HamQA [15] , JointLK [64] , GreaseLM [88] and GrapeQA [67] . To ensure a fair comparison, we implement these baselines with advanced language models that are optimized for particular datasets. Specifically, RoBerta-large [49] is used for CommenseQA, while AristoRoBERTa [13] is designated for OpenBookQA. For MedQA, we opt for the top-tier biomedical language model, SapBERT [45] . Note that due to the white-box nature of these methods and their high computation overheads, it is infeasible to apply them to state-of-the-art LLMs, like GPT-3.5 and GPT-4.LLM + Zero-shot. We include several representative generative LLMs, including ChatGLM, Chat-GLM2, Baichuan-7B, InternLM, GPT-3, GPT-3.5 and GPT-4 as knowledge-agnostic alternatives. Specifically, we used the model 'text-davinci-002' provided by OpenAI as the implementation of GPT-3, and 'gpt-3.5-turbo' and 'gpt-4' as the implementations of GPT-3.5 and GPT-4, respectively (we have provided more implementation details of all LLMs in Appendix A.4). The question-answering task is conducted under the zero-shot setting with the question query from the test set as input.LLM + KG Prompting. To verify the effectiveness of our prompting strategy, we also add the state-of-the-art KG prompting methods, i.e., CoK [71] , RoG [50] , and Mindmap [74] as baselines. Notably, we did include KGR [25] in our main results, since the authors have not released their codes.To address RQ1, we evaluate KnowGPT by comparing it to state-of-the-art baselines on the three benchmark datasets. KnowGPT is based on the original GPT-3.5. We measure the performance using accuracy, which calculates the percentage of questions correctly predicted by the model out of the total questions in the test set. We have the following observations:• KnowGPT outperforms all categories of methods, including sixteen different baselines, across all datasets and model architectures. This suggests that KnowGPT can effectively inject the knowledge from KGs to LLMs.• KnowGPT surpasses the performance of GPT-3.5 and even GPT-4. On average, KnowGPT achieves a 23.7% higher testing accuracy than GPT-3.5. Specifically, KnowGPT outperforms GPT-3.5 by 10.8%, 32.4%, and 29.4% on the CommonsenseQA, OpenBookQA, and MedQA datasets, respectively. More importantly, despite being based on GPT-3.5, KnowGPT outperforms the state-of-the-art LLM GPT-4 by 3.3%, 1.4%, and 1.8% on the CommonsenseQA, OpenBookQA, and MedQA datasets, respectively. These results confirm that black-box knowledge injecting can effectively enhance the capabilities of LLMs.• KnowGPT outperforms all KG-enhanced LMs significantly. This implies our black-box knowledge injection method proficiently encodes knowledge into LLMs. Furthermore, it showcases the superiority of our black-box approach, given its adaptable application to GPT-3.5 using only the model API, a feat not achievable by white-box methods.We submit our results onto the official leaderboard maintained by the authors of OpenbookQA. The full records on the leaderboard are shown on the website2 , while our result can be found from here 3 .We summarize the related submissions in Table 2 , including three categories: traditional KG-enhanced LM, fine-tuning of LLMs, e.g., T5-11B used in UnifiedQA, and ensemble of multiple predictors.KnowGPT significantly outperforms traditional KG-enhanced LMs with 5.2% improvements when compared to the best baseline. The third group of methods occupies the leaderboard by leveraging ensemble learning strategies. Nevertheless, KnowGPT can still obtain competitive performance without ensembling with 0.6% above GenMC Ensemble [33] . Notably, our KnowGPT is remarkably comparable to the human performance. [20] 0.806 DRAGON [79] 0.878 QA-GNN [80] 0.828 GenMC [33] 0.898GreaseLM [88] 0.848 Human Performance 0.917 HamQA [15] 0.850 GenMC Ensemble [33] 0.920 JointLK [64] 0.856 X-Reasoner [32] 0.952 GSC [72] 0.874 KnowGPT 0.926To answer RQ2, we conduct two ablation studies. First, in Table 3 , we measure the importance of the tailored reinforcement learning-based knowledge extraction module, i.e., P RL . Specifically, we compare it with the heuristic sub-graph extraction strategy, i.e., P sub . The performance is evaluated by directly feeding the extracted knowledge with the prompt format of 'Sentence', i.e., F s , to GPT-3.5.We also include 'w/o KG' as the baseline where GPT-3.5 is asked to independently answer the given question with no reasoning background provided. The results clearly indicate the vital role of our proposed knowledge extraction strategies. Second, we compare each of the three prompt formats subject to the same extracted knowledge. The detailed results are shown in Table 4 . Though different formats perform similarly within the difference of 2.2% -3.3%, they are particularly suitable for different kinds of questions. We illustrate this observation in the following case study section. Both ablation studies support the indispensability of each module, armed with a tailored deep reinforcement learning-based knowledge extraction and a context-aware prompt translation, our KnowGPT performs best on all three benchmark datasets. For RQ3, we provide insights into how KnowGPT facilitates the prompt translation with a real case from CommonsenseQA. We visualize both the extracted knowledge and the textual inputs to GPT-3.5 in Figure 3 . In this example, given the same extracted knowledge, GPT-3.5 answers correctly based on the sentence format that we provide. In contrast, it fails to answer the question with triples and graph descriptions. They clearly indicate the superiority of KnowGPT in an automatic context-aware prompt translation. We make the following observations: (i) Triple format F t is intuitively suitable for all the simple questions by directly indicating the one-hop knowledge. (ii) Graph description may inevitably introduce noise to ensure the completeness and contextual fluency of the directed graph.In this example, since 'vacation' appears in both question and answer choices, over-emphasizing and connecting the knowledge about 'vacation' with other concepts in the graph misleads the model to make a prediction with an oblique focus. (iii) Our KnowGPT has shown superior performance in automatically constructing suitable prompts for particular questions. 
limitation: 
 Through our exploration, we realize the natural limitations of KnowGPT brought by real-world KGs.Existing KGs are automatically constructed based on online corpora. This inevitably introduces a considerable number of noisy triples into KGs. The noisy knowledge may mislead the LLMs to wrong predictions despite the effectiveness of our prompting methods. In the future, we would leverage KnowGPT with off-the-shelf KG refinement algorithms to improve the quality of KGs.David watched some nesting birds using his binoculars while on vacation. Where might David be? Nesting, an instinctual action performed by various animals, including birds, involves inhabiting and crafting intricate structures known as nests. Birds, closely related to this behavior, are often spotted at their nests, where they are capable of building and preparing them for raising their young. These feathered creatures have a natural affinity for trees and are often found in tree nests, which are nestled within lush jungles and rainforests. During a relaxing vacation, one can observe birds engaging in nesting activities, taking advantage of binoculars as a vision aid to witness their behavior in the expansive sky, a realm intertwined with the beauty of the outdoors.This graph highlights the intricate relationships between birds, their nesting behaviors, natural habitats like trees and forests, the sky, and the tools that aid in the observation of these captivating avian activities. The graph revolves around the central concept of a "bird" which is closely connected to various aspects of nesting behavior.[…] This aerial environment is also associated with outdoor settings. Adding depth to the observation of birds, binoculars are presented as a vision aid that allows closer scrutiny of their nesting behavior, both in natural locations like backpacks, sporting goods stores, and suitcases, and during the context of a vacation.  
conclusion: 
 The main objective of this paper is to tackle the hallucination issue that arises when applying LLM to specific domains. Although LLMs have strong reasoning capabilities, they still struggle to answer professional questions in specific domains, especially when the pre-training corpus lacks relevant knowledge. To address this issue, we propose a KG-augmented LLM model, named KnowGPT, which injects relevant domain knowledge from KGs into LLMs to assist the LLM in accurately answering professional questions. A novel framework, namely KnowGPT, is presented to integrate KGs into LLMs effectively with model APIs only. We first train a deep RL policy to extract informative and concise reasoning background from the KG. Then we learn an MAB to select the most effective knowledge extraction method and prompt template for each question. Extensive experiments on both general and domain-specific QA show superior performance of KnowGPT compared to all competitors.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We present the experimental setting details at the beginning of the experiment section. We also provide the dataset statistics and the preprocessing details in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks, as the work involves only querying the API of an LLM to get pseudo-labels, without releasing a generative model or datasets.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited the original papers of the datasets and EA models used in our experiments, ensuring proper credit and respect for their licenses and terms of use. For the open-source code and datasets, we have also stated the licenses they use in the readme.txt file in our code repository.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: Code of the proposed framework is released and accessible via this anonymouse URL: https://anonymous.4open.science/status/KnowGPT-DD64. It contains a readme.md file and a GPLv3 license.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: To address the hallucination issues that arise when applying LLMs to specific domains, researchers have proposed various methods, which can be broadly categorized into three main categories: Fine-Tuning (FT), Retrieval-Augmented Generation (RAG) and KG-enhanced LLMs.To help the language model generate more accurate responses in specific domains, extensive studies have been conducted to investigate the efficacy of fine-tuning LLMs with specialized data [8, 27, 26, 39, 40, 22, 28] . Fine-tuning involves training the pre-trained LLM on a smaller dataset of domain-specific text, enabling the model to adapt its knowledge to the target domain, such as in recommendation [10, 91] , and noGCNde classification [9, 30] . By exposing the LLM to domain-specific vocabulary, terminology, and patterns, fine-tuning enables the model to generate more accurate and relevant responses for domain-specific tasks [26, 31] . Fine-tuning LLMs with domain-specific data has been successfully applied across various domains. For example, in the healthcare domain, fine-tuned LLMs have been used for clinical note analysis [1] , biomedical text mining [40] , and medical dialogue [69] . Similarly, in the legal domain, fine-tuned LLMs have shown promise in tasks such as legal document classification [6] , contract analysis [7] , and legal judgment prediction [90] .However, recent research has highlighted the limitations and risks associated with fine-tuning LLMs using domain-specific data [37] . A study conducted by Google Research highlighted that using fine-tuning to update LLMs' knowledge can be problematic, particularly when the new knowledge conflicts with pre-existing information [23] . In such cases, acquiring new data through FT can lead to the model generating new hallucinations and even experiencing catastrophic forgetting [37, 78] . Besides that, many state-of-the-art LLMs are confined to a black-box role in practice. For instance, Calaude3 [3] and GPT-4 [54] exclusively grant access through their APIs, which means we can only retrieve model responses by submitting textual inputs, with model specifics inaccessible. This lack of access to model architecture and parameters prevents us from employing these fine-tuning methods.Recently, Retrieval-Augmented Generation (RAG) models have been extensively explored to enhance LLMs with external knowledge from text corpora or online sources [41, 89, 53] . Combining LLMs with external knowledge retrieval systems can help reduce hallucinations. However, these approaches face challenges in domain-specific applications: (i) Data quality. Domain knowledge is often scattered across different sources, such as textbooks, research papers, technical manuals, and industry reports [42] . These textual documents may have varying levels of quality, accuracy, and completeness, leading to potential inconsistencies or errors in the retrieved knowledge [92] . (ii) Knowledge hierarchy. Domain knowledge is always complex and hierarchical and contains a high degree of jargon, acronyms, and specialized terminology. However, textual documents typically present this information in a linear and unstructured manner. The lack of explicit relationships and structured organization limits the reasoning and inference capabilities of RAG models, as they cannot leverage the structured connections within domain knowledge to derive new insights or generate more contextually appropriate responses [58] . (iii) Huge search space. There is a lot of domain-irrelevant information in these textual documents, while domain-specific terminologies are always sparsely distributed over these documents [61] . The retrieval model can be computationally expensive and time-consuming, especially when dealing with large-scale knowledge sources, as the model needs to search through vast amounts of unstructured text to find relevant information [51] .KG-enhanced LLM aims to leverage the structured knowledge in knowledge graphs (KGs) [11, 48] to ground the model's responses in established facts and principles [29, 77, 56] . This grounding ensures that the generated outputs are based on reliable information rather than arbitrary or fabricated statements.Integrating KGs during Training. Earlier studies adopted a heuristic way to inject knowledge from KGs into the LLMs during pre-training or fine-tuning. For example, ERNIE [63] incorporates entity embeddings and aligns them with word embeddings in the pre-training phase, encouraging the model to better understand and reason over entities. KnowBERT [57] integrates entity linkers with BERT to introduce knowledge about entities during fine-tuning for downstream tasks.Integrating KGs during Inference. Another line of work focuses on retrieving relevant knowledge from KGs at inference time to augment the language model's context. Typically, K-BERT [47] uses an attention mechanism to select relevant triples from a KG based on the input context, which are then appended to the input sequence. Similarly, KEPLER [73] learns joint embeddings of text and KG entities, enabling efficient retrieval of relevant knowledge to enhance the model's predictions. Later works further integrated graph neural networks alongside LLMs for joint reasoning [80, 87, 15] and introduced interactions between text tokens and KG entities within the intermediate layers of LLMs [64, 67] .KG Prompting for LLM. The methods above all assume they know everything about LLMs, including model architectures and parameters. However, as LLMs have been keeping evolving, many SOTA LLMs are confined to a black-box role in practice. As such, the research focus has shifted towards KG prompting that enhances fixed LLMs with KG-based hard prompts. KG Prompting for LLMs has been a new learning paradigm in natural language processing [46, 56] . Specifically, CoK [71] introduces a novel Chain-of-Knowledge prompting to decompose LLM-generated reasoning chains into evidence triples. It then verifies the triples' factuality and faithfulness using an external knowledge graph, ensuring the accuracy and reliability of the LLM outputs. By allowing the models to comprehend and reason over structured KG inputs, Mindmap [74] enhances the LLMs' ability to incorporate external knowledge, reduce hallucinations, and provide more transparency into their decision-making process. RoG [50] presents a planning-retrieval-reasoning framework that synergizes the strengths of LLMs and KGs to enhance the reasoning capabilities of language models in a more transparent and interpretable manner. Instead of retrieving factual information from KGs, KGR [25] proposes to autonomously retrofit the initial draft responses generated by LLMs. This innovative technique leverages the knowledge stored within KGs to mitigate hallucination by effectively extracting, verifying, and refining factual information throughout the entire reasoning process of LLMs.Our model. Despite the promising performance of existing KG prompting methods, their extensive deployment and practical implementation in domain-specific applications are still impeded by two critical issues. ❶ The cost associated with calling the LLM API or deploying LLMs with cloud services is prohibitive. For example, GPT-4 is estimated to cost at least thousands of dollars for pilot-scale customer service, making the careful selection of the most informative triples from KGs essential to minimize costs. ❷ LLMs are highly sensitive to prompts, with even minor variations in prompts conveying the same semantic meaning potentially yielding drastically different responses. However, existing methods rely on manually designed or rule-based prompts to present factual knowledge from KGs. These hard prompts are always fixed and rigid, lacking the flexibility to adapt to variations in question semantics and KG structures. In this paper, we shed light on the whole community with a novel prompt learning method that is rather effective, generalizable and cost-efficient. There are two key components in our proposed KnowGPT including (i) knowledge retrieval, which leverages deep RL to extract relevant knowledge from KGs, and (ii) context-aware decision module to translate the structured knowledge from KGs into the appropriate prompt format.For each QA context, we adopt the methodology outlined in the prior research [44, 79] to extract the subgraph from the background knowledge graph (KG), denoted as G. We commence by executing entity linking on G, resulting in an initial collection of nodes, V topic . Next, we incorporate bridge entities that appear within a 2-hop path between any two linked entities from V topic , yielding the set V retrieval . Subsequently, we refine this set by evaluating the relevance score for each node, adhering to the method proposed [79] . From this refined set, only the top 200 nodes, based on score, are retained. We then extract all edges connecting any pair of nodes in V sub , creating the retrieved subgraph G sub . Each node within G sub is designated a type based on its association to either the topic entities Q or target entities A. Intuitively, the relevant reasoning background lies in a question-specific subgraph G sub that contains all the source entities S, target entities A, and their k-hop neighbors. Therefore, the reasoning background could be provided as the G sub , we denote this direct knowledge extraction method as P sub .To calculate the initial node embeddings for entities extracted from the background KG, we adopt the approach proposed by the previous study [20] . Specifically, we transform knowledge triples from the KG into sentences and feed them into pre-trained LMs to get node embeddings. Specifically, to ensure a fair comparison, we implement all the KG-enhanced baselines and our model with the same advanced language models that are optimized for particular datasets. Specifically, RoBert-large [49] is used for CommenseQA, while AristoRoBERTa [13] is designated for OpenBookQA. For MedQA, we opt for the top-tier biomedical language model, SapBERT [45] , to enhance comprehension of the biomedical field.We evaluate KnowGPT on three QA datasets spanning two fields: CommonsenseQA [66] and Open-BookQA [52] serve as benchmarks for commonsense reasoning, while MedQA-USMLE [34] acts as a domain-specific QA benchmark. While the official test set serves primarily for leaderboard rankings, we initially assess model efficacy using the in-house (IH) data split introduced in [44] . The official dataset is denoted as CSQA, while the IH split is represented by CSQA(IH)*. The statistics of these three datasets can be found in Table 5 . CommonsenseQA is a multiple-choice question-answering dataset, each question accompanied by five potential answers. Answering its 12,102 questions necessitates a foundation in commonsense knowledge. While the official test set serves primarily for leaderboard rankings, we initially assess model efficacy using the in-house (IH) data split introduced in [44] . The official dataset is denoted as CSQA, while the IH split is represented by CSQA(IH)*.OpenBookQA, commonly abbreviated as OBQA, comprises 5,957 multiple-choice questions, each offering four possible answers. To successfully answer these questions, one must have a comprehensive understanding of fundamental scientific facts and its applications.MedQA-USMLE, abbreviated as MedQA, is a dataset consisting of 4-option multiple-choice questions that demand a grasp of biomedical and clinical understanding. These questions are sourced from preparatory tests for the United States Medical Licensing Examinations, and the dataset encompasses 12,723 questions. We adhere to the original data divisions as outlined in [34] .Background Knowledge To facilitate common sense reasoning, we employ ConceptNet [62] , an extensive commonsense knowledge graph comprising more than 8 million interconnected entities through 34 concise relationships. For tasks specific to the medical domain, we leverage USMLE [80] as our foundational knowledge source. USMLE is a biomedical knowledge graph that amalgamates the Disease Database segment of the Unified Medical Language System (UMLS) [4] and DrugBank [75] . This repository encompasses 9,958 nodes and 44,561 edges.To verify the effectiveness of our proposed KnowGPT, we carefully selected baseline models from three aspects to ensure a comprehensive evaluation, among which Bert-base, Bert-large [35] , and RoBert-large [49] are picked for being representative fine-tune LM methods; MHGRN [20] , QA-GNN [80] , HamQA [15] , JointLK [64] , GreaseLM [88] and GrapeQA [67] represent the state-of-art KG-enhanced LMs; ChatGLM [19] , ChatGLM2 [81] , Baichuan-7B, InternLM [68] , GPT-3 [5] ,GPT-3.5 [55] and GPT-4 [54] are picked for being representative generative large language models. To verify the effectiveness of our proposed prompting strategy used KnowGPT, we also add the state-of-the-art KG prompting methods, i.e., CoK [71] , RoG [50] , and Mindmap [74] as baselines. Notably, while some LLM baselines are actually open-source, we conduct the question-answering task under the zero-shot setting with the question query from the test set as input. All baseline methods used in this paper are based on their open-source implementations or officially-released APIs. Notably, we used the model 'text-davinci-002' provided by OpenAI as the implementation of GPT-3, and 'gpt-3.5-turbo' and 'gpt-4' as the implementations of GPT-3.5 and GPT-4, respectively. All models are implemented in Pytorch and trained on an RTX 3090 with 24 RAM. We use their best configuration as the default value for other distinctive hyperparameters in the baseline methods. To reduce the randomness, we use the random seed and report the average results of three runs. provide LLMs with more detailed and structured information by highlighting the local structure of the central entity.If the background knowledge graph is complete and high-quality, our proposed RL-based knowledge extraction can ideally handle all cases. But in pratical implementation, we found that there are a few long-tail entities in KGs which only have few neighbors. In such cases, RL-based knowledge extraction can hardly retrieve reachable or effective paths due to the incompletion and sparsity of the background graph. Instead, directly extracting the whole subgraph through rule-based subgraph extraction can provide more knowledge for reasoning. The RL-based retrieval method P RL could make sure our model extracts more concise and informative knowledge in most cases, while the rule-based subgraph extraction P RL could supplement the former especially when the reasoning background is sparse. Thus, in this paper, we adopt Multi-Armed Bandit (MAB) to automatically select the most suitable retrieval methods from these two candidates, i.e., P RL and P sub . The ablation study on the knowledge retrieval method can be found in Table 3 of the main content. Specifically, we compare our model with P RL , the heuristic sub-graph extraction P sub , and we also include 'w/o KG' as the baseline where the LLM is asked to independently answer the given question with no reasoning background provided. The results shown in Table 3 of our submission clearly indicate the vital role of our proposed knowledge extraction strategies.To investigate the efficiency of our model, we compare it on MedQA with six representative baselines from three different categories, including traditional KG-enhanced LMs, i.e., JointLK and GrapQA, zero-shot LLMs, i.e., ChatGPT and GPT-4, and the SOTA KG-prompting-based LLMs, i.e., CoK and Mindmap. Notably, zero-shot LLMs, like ChatGPT and GPT-4, do not require training while traditional KG-enhanced LM methods like JointLK and GrapeQA are based on lightweight LMs and do not require interaction with LLMs, thus they do not incur API cost.From Table 8 , we can see that (i) Traditional KG-enhanced LM methods, like JointLK and GrapQA, have the shortest inference time. This is because the other models need to send requests to the Black-box LLM via API, and the extra response time of the LLM leads to long inference times for these models. Despite the efficiency, they have the worst performance. (ii) Our model outperforms the other models with comparable training time and the most economical API cost compared to models in the same category, including CoK and Mindmap. That is because we are the only model that consider the conciseness of knowledge during retrieval and prompt design. Due to page limits, we carefully described how we designed the reward function to ensure the RL model captures effective and informative knowledge in the main content of our submission. Actually, we have also employed the following techniques to stabilize the training process.Gradient clipping for policy network. The policy network is updated using gradient-based optimization to maximize the expected cumulative reward. To stabilize the training of the policy network, we applied gradient clipping [21] to prevent the gradients from becoming too large. This technique helps mitigate the problem of exploding gradients and ensures stable updates of the network parameters. To demonstrate the effectiveness of gradient clipping, we conducted additional experiments comparing the performance of our model with and without gradient clipping across five runs. The results are presented in the table below. As shown in Table 9 , the model with gradient clipping consistently achieves better performance across all datasets, demonstrating the stability and robustness of our method.Exploration-exploitation balance for MAB. In our designed MAB, we have introduced a penalty term, i.e., β, to encourage exploration on under-explored arms (i.e., prompt formats), otherwise, it will lead to a greedy selection based on c × α (i) . Table 10 shows the performance comparison between with and without this term. We are dedicated to not only focusing on the specific NLP task, i.e., knowledge-based QA, but also inspiring and benefitting a wide range of NLP communities with a novel knowledge injection framework to fast adapt LLMs to specific domains. Specifically, we would like to declare the following two contributions for a wide range of NLP communities.Exploring Effective Knowledge Injection for LLMs: Applying LLMs for various NLP-related tasks in different domains is gaining momentum. However, fine-tuning LLMs for domain knowledgebased QA is expensive. Our paper presents a remarkably strong framework that could (i) retrieve the reasoning background from domain knowledge graph and (ii) automatically find out the optimal prompt to inject the domain knowledge into the LLM.Providing Insights in Prompt Learning: While hard prompt is still dominating prompt learning for the majority of LLM-based scenarios, we shed light on the whole community with a novel prompt learning method that is rather generalizable and effective. We propose an auto-selection algorithm that can determine the best prompt format for LLMs to maximally understand the reasoning background.NeurIPS Paper ChecklistJustification: To control for the randomness introduced by the Large Language Models, we repeat each experiment three times and report the mean and standard deviation in tables.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).https://leaderboard.allenai.org/open_book_qa/submissions/public.https://leaderboard.allenai.org/open_book_qa/submission/cp743buq4uo7qe4e9750. 


Decision: Summary (taken from t5Eo's excellent recap)
==============
This work presents KnowGPT, a novel KG-based Prompting framework designed to integrate domain knowledge into LLMs effectively. KnowGPT addresses the key challenges of previous methods, such as large search spaces, high API costs, and labor-intensive prompt engineering. The framework includes a knowledge extraction module that identifies the most relevant information from KGs and a context-aware prompt construction module that automatically generates effective prompts from the extracted knowledge. Experiments demonstrate that KnowGPT significantly outperforms existing methods, achieving a notable 92.6% accuracy on the OpenbookQA leaderboard, approaching human-level performance.

Metareview
===============

The reviews are all positive, but none are overwhelmingly so.  They identify no major technical issues, and only minor presentation issues, which I'm confident can be addressed in a revision.  Thus, I am recommending the paper for acceptance.

The argument for acceptance is that the use of bandit models to select how to prompt a model with a knowledge subgraph is technically interesting and novel (although I'd like to see an explicit comparison with DSPy, which has a similar ethos).  And this novelty pays off with good empirical results.

The weaknesses mentioned by the reviewers are relatively minor, and I infer that reviewers' relatively low scores were more a product of lack of excitement than serious flaws.  I agree with the lack of integration of the case studies and the lack of explicit details weaken the text of the paper.

Reviews: review: paper_summary: paper_summary: The paper presents a novel framework, KnowGPT, for incorporating Knowledge Graphs into LLM-prompting-based question answering. It breaks down the problem into two main parts: identifying the concise, rich, and relevant subgraph for the question and using the Multi-Armed Bandits framework to select the best prompt construction method. The first part uses deep reinforcement learning with a three-part reward for training a model for subgraph identification. The second part uses the MAB framework to select the best subgraph representation and Prompt construction method. The presented framework is evaluated on three datasets and compared with a variety of baseline models. The results show that the proposed framework, KnowGPT, results in better performance than all of the baseline and is comparable to an ensemble model. The paper also presents an ablation study showcasing the impact of each of the two components.
strength_weakness: strength_weakness: W1.	The case studies are referenced in some places in the paper, indicating that they are present in the main paper and not in the appendix.
questions: questions: Q1.	The method assumes that the source and target entities are given. However, a discussion of the impact of using an off-the-shelf model to extract them will be important to determine the significance of that assumption.
limitations: limitations: The authors have clearly outlined the limitations of the work.
paper_summary: paper_summary: The paper proposes KnowGPT, a novel KG prompting enhanced LLM framework that leverages deep reinforcement learning (RL) to extract knowledge and Multi-Armed Bandit (MAB) to generate effective prompts for domain-specific queries. Empirical evidence on QA benchmarks shows KnowGPT’s superiority over other methods.
strength_weakness: strength_weakness: 1. The analysis of the efficiency of the method seems to be insufficient.
2. The Introduction part seems to be too long.
questions: questions: 1. The paper argues that a big challenge is the high cost of using the API of LLM, but the efficiency analysis does not show how much it costs to call the API in order to train the two modules of the method, especially the prompt construction module.
2. The article mentions that the shortcoming of the existing method is that it requires Laborious prompt design, but your method also manually designs the prompt and requires effort of training. In addition, are the prompts of the latter two templates generated by LLM? Is the cost of this step factored into the efficiency comparison?
3. Can you provide the proportional distribution of different knowledge extraction methods and prompt forms during testing?
limitations: limitations: The authors discussed the limitations of their method in the paper.
paper_summary: paper_summary: This work presents KnowGPT, a novel KG-based Prompting framework designed to integrate domain knowledge into LLMs effectively. KnowGPT addresses the key challenges of previous methods, such as large search spaces, high API costs, and labor-intensive prompt engineering. The framework includes a knowledge extraction module that identifies the most relevant information from KGs and a context-aware prompt construction module that automatically generates effective prompts from the extracted knowledge. Experiments demonstrate that KnowGPT significantly outperforms existing methods, achieving a notable 92.6% accuracy on the OpenbookQA leaderboard, approaching human-level performance.
strength_weakness: strength_weakness: 1.This paper lacks necessary details regarding the proposed method KnowGPT, such as: (a) how the reinforcement learning part on the knowledge graph is trained; (b) how to optimize the Knowledge Graph Agent together with the Prompt Construction with Multi-armed Bandit as proposed in Section 3.2; (c) besides the two-hop subgraph extraction, the paper does not provide implementation details for the proposed method; (d) since we know prompts can significantly affect the output quality of models, the paper does not provide specific examples of inputs given to LLMs, only a case from KG to text, lacking the complete prompt information given to LLMs. This causes confusion: for LLMs, is it a multi-class or multiple binary (as in the YES/NO from Figure 2) classification problem? With multiple binary classifications, there is the issue of handling multiple (or zero) options being YES—how should this be dealt with? (Situations with all NO or all YES responses may arise.)

2.I think that the overall contribution of this paper primarily lies in a type of search algorithm: searching for subgraph structures (subgraph structures formed by RL or standard 2-hop subgraphs), searching for prompt forms (direct triples, triples converted into sentences, graph descriptions). The RL algorithm is fundamentally similar to Deeppath[1], and the authors should further distinguish the differences between them. Additionally, the authors should also elaborate on how the search algorithm proposed in this article differs from [2].

[1]Xiong, Wenhan, Thien Hoang, and William Yang Wang. "Deeppath: A reinforcement learning method for knowledge graph reasoning." arXiv preprint arXiv:1707.06690 (2017).

[2] Dong, Junnan, et al. "Active ensemble learning for knowledge graph error detection." Proceedings of the sixteenth ACM international conference on web search and data mining. 2023.
questions: questions: 1.On line 185 of the 5-th page, where are the additional training details mentioned in the appendix? Policy gradient is merely an algorithm, but how is the Agent on the Knowledge Graph trained? How is the overall objective function defined? Given that the action space in reinforcement learning on Knowledge Graphs is huge, how have you addressed this issue? From a technical solution perspective, what are the differences between the method 3.2 Knowledge Extraction with Deep Reinforcement Learning proposed in this paper and [1]?

[1]Xiong, Wenhan, Thien Hoang, and William Yang Wang. "Deeppath: A reinforcement learning method for knowledge graph reasoning." arXiv preprint arXiv:1707.06690 (2017).

2.What is the difference between "Graph Description" and "Sentences" in the prompt? Is the extracted subgraph structure the same for both, with the difference being that the "Graph Description" contains additional noise, such as "The entity is strongly associated with significant individuals in the tech industry."? I think a more reasonable approach would be that the "sentence" component targets the path, whereas the "Graph Description" component pertains to the entire subgraph structure.

3.On line 212 of the 6-th page, what does OLS refer to? Does it mean Ordinary Least Squares? What does \delta on line 218 of the sixth page refer to, is it a constant?

4.From a technical solution perspective, what is the difference between the content of Section 3.2 and the Active Ensemble Policy Learning with MAB described in [2]?

[2]Dong, Junnan, et al. "Active ensemble learning for knowledge graph error detection." Proceedings of the sixteenth ACM international conference on web search and data mining. 2023.
limitations: limitations: Yes

Comments: title: General Response to Area Chairs and All Reviewers (2/2)
General Response to Area Chairs and All Reviewers (1/2)
Detailed response to Reviewer t5Eo (4/4)
Detailed response to Reviewer t5Eo (3/4)
Detailed response to Reviewer t5Eo (2/4)
Detailed response to Reviewer t5Eo (1/4)
Great Thanks to Reviewer 4L5K
Grateful Thanks to Reviewer vKLu
comment: * **Efficiency Analysis**. Existing KG prompting methods often suffer from the huge API cost. To investigate the efficiency of our model, we compare it on MedQA with several SOTA baselines in terms of both API cost and training efficiency. The results can be found in Table 9 of our submission and Table 1 of the Rebuttal to Reviewer vKLu. 

* **Prompt Design**. The hard prompts used in existing KG prompting models are inherently inflexible and rigid, lacking the adaptability to accommodate variations in question semantics and KG structures. To this end, we specifically designed three different prompt templates, guiding the LLM to convert the extracted raw knowledge into textual prompt with corresponding logical structure. To make it clear, we first carefully explained the rationale behind our prompt design and then introduced the implementation details (the Rebuttal to Reviewer vKLu and t5Eo) following with a detailed experimental analysis (Table 1 of the Rebuttal to Reviewer vKLu).

* **Proportional Distribution**. To verify the effect of the MAB-based prompt constriction strategy, we conducted a thorough analysis on three benchmarks (Table 3 of the Rebuttal to Reviewer vKLu) by investigating the proportional distribution of different knowledge extraction methods and prompt forms during testing.

* **Implementation Details**. In our submission, we focused on describing how we carefully designed a meaningful reward function to ensure that the RL-based extraction model  $P_{RL}$ captures effective and informative knowledge, and the implementation details are included in the Appendix. Specifically, we introduced the details of sub-graph extraction strategy $P_{sub}$ in Section B.1, while the training details of RL model are included in Section C.6 of Appendix. To make it clear, we first explained the rationales behind the design of two knowledge extraction methods and then summarized the training details together with several supporting experiments (Table 1 of Official Comment to Reviewer t5Eo and Table 3 of our submission). We'll include a new Algorithm 1 in our revision to show the overall training process. 
* **Differences Compared to Deeppath and AEKE**. We clarified the differences between our work and the cited papers in both the Rebuttal and Official Comment to Reviewer t5Eo. Specifically, our model diverges from DeepPath in motivation, reward function and training strategy, while our MAB approach differs from AEKE's significantly in training since we introduced a penalty term to encourage exploration of under-explored prompt formats. To make it more clear, we included additional experiments to verify the effect of our tailored design (Table 1&2 of Official Comment to Reviewer t5Eo).


Thank you again for your time and expertise throughout this review process. We deeply appreciate your expertise and recognition of our work. Each point raised has given us new perspectives to consider. We commit to addressing all the comments and incorporating the additional experiments and analysis in our final revision. Hope that this summary could facilitate the next stage of review and discussion.
Dear Area Chairs and Reviewers,

We sincerely thank you for your time, effort, and invaluable feedback during the author-reviewer discussion phase. Your insights have been crucial in helping us refine and improve our work. To facilitate the next stage of review and discussion, we would like to summarize the key contributions of our paper, the main points raised during the discussion and our response.

 ### **Motivation**

LLMs are often criticized for their tendency to produce hallucinations, wherein the models fabricate incorrect statements on tasks beyond their knowledge and perception. To alleviate this issue, we proposed a novel Knowledge Graph-based prompting framework, namely KnowGPT.  **KnowGPT aims to leverage the factual knowledge in knowledge graphs (KGs) to ground the LLM's responses in established facts and principles**.

 ### **Challenges**
Despite the promising performance of existing KG prompting methods, three critical issues hinder their widespread application in practice. 
* **Huge Search Space.** Real-world KGs often consist of millions of triples, resulting in a vast search space when retrieving relevant knowledge for prompting.
* **High API Cost**. Closed-source LLMs, like GPT-4 and Claude 3, are accessible through proprietary APIs, which can incur significant costs when performing KG prompting at scale. Thus, careful selection of the most informative knowledge from KGs is essential to minimize costs.
* **Laborious Prompt Design**. Existing methods rely on manually designed or rule-based prompts to present factual knowledge from KGs.  These hard prompts are inherently inflexible and rigid, lacking the adaptability to accommodate variations in question semantics and KG structures. 



 ### **Contribution**
KnowGPT contains a knowledge extraction module to extract the most informative knowledge from KGs, and a context-aware prompt construction module to automatically convert extracted knowledge into effective prompts.
* **Exploring Efficient and Effective Knowledge Extraction for KG Retrieval**. We introduced a novel RL-based retrieval method and designed three rewards to ensure that the RL model extracts correct, concise, and effective background knowledge.

* **Providing Insights in Prompt Learning**. While hard prompt is still dominating prompt learning for the majority of LLM-based scenarios, in this paper, we shed light to the whole community with a novel KG prompting method which is rather generalizable and effective. We propose an auto-selection algorithm which can determine the best prompt format for LLMs to maximally understand the given reasoning background.

* **Extensive Experiments**. To verify the effectiveness, **we compare KnowGPT with 22 baseline models including fine-tuning LMs, KG-enhanced LMs, prompt learning models and state-of-the-art LLMs augmented with KG prompting** on 3 public benchmarks and 7 fine-tuning LLMs on CSQA. We believe the results in Table 1 and 7 of our submission and the **`Top Rank on OpenBookQA Leaderboard`** can verify the effectiveness and generalizability of our model. Besides that, we also conducted sufficient ablation study, case study and efficiency analysis to study the effect of our model.

 ### **Main Discussion Points and Our Responses**
* **Impact of Entity Extraction Model**. In this paper, we focus on developing effective and efficient knowledge retrieval and injection methods, assuming entity recognition from the original question as a separate step to isolate the impact of our core contributions. Reviewer 4L5K suggested us to study the impact of entity extraction models. Following the suggestion, we conducted an additional experiment on CSQA by replacing BERT in our model with 4 advanced entity recognition models (Table 1 of the Rebuttal to Reviewer 4L5K). We believe that discussing the impact of using an off-the-shelf model for entity extraction is valuable to speed up real-world applications of our method.
Dear Reviewer t5Eo,

Thank you for your recognition of our work and for providing such thorough and insightful feedback. We greatly appreciate the time and effort you've invested in reviewing our work and providing such detailed comments. Your comments and suggestions are invaluable in helping us improve the quality and clarity of our work. 

We have carefully considered each point you raised and have made substantial analysis and explanation. If there are any aspects that you feel still require further clarification or improvement, we would be more than happy to provide additional information or explanations to assist in your review process. 

Thank you again for your time and expertise. Your feedback has been instrumental in enhancing the quality of our research.
Thanks for your responses. I have carefully read your responses, I think that my score is reasonable and fair. So I maintain the scores.
To verify the effectiveness of prompt formats, we conducted a statistical analysis on the benchmark datasets in terms of question types and then separately calculated the accuracy of different prompt formats on specific types of questions. As shown in Table 3, we can observe that triple-based prompt $\mathcal{F}_t$ performs best on simple questions, sentence-based prompt $\mathcal{F}_s$  work better for multi-hop questions  while graph description-based prompt $\mathcal{F}_g$ performs better than any other prompt formats on complex questions. This is because graph description-based prompt could provide LLMs with more detailed and structured information by highlighting the local structure of the central entity.


> **[Q3]** On line 212 of the 6-th page, what does OLS refer to? Does it mean Ordinary Least Squares? What does $\delta$ on line 218 of the sixth page refer to, is it a constant?

Thanks for your careful review. OLS is the ordinary least square loss. While $\delta$ is a constant, for any $\delta > 0$ with the probability at lease $(1-\delta)$, the reward expectation $E(r^{(i,b)} | {\bf x}_{i}^{(b)})$ is bounded by a confidence interval, as shown in h.

**Thanks for your time and effort in providing insightful suggestions that can help further polish our paper. We're more than happy to provide any additional information or explanations that may be helpful in your review process.**
> **[W2-a & Q1-c,d]** The RL algorithm is fundamentally similar to Deeppath[1], and the authors should further distinguish the differences between them. Given that the action space in reinforcement learning on Knowledge Graphs is huge, how have you addressed this issue? From a technical solution perspective, what are the differences between the method 3.2 Knowledge Extraction with Deep Reinforcement Learning proposed in this paper and Deeppath[1]?

We appreciate the reviewer's insightful comments and the opportunity to clarify the distinctions between our work and the cited papers. 

**Differences compared to Deeppath**:

* **Motivation**. DeepPath focuses on extracting a path from KG given start and end nodes. In contrast, our work addresses knowledge-based retrieval, extracting question-related knowledge while considering both KG structure and question semantics.
* **Reward function**. DeepPath's reward function is primarily based on KG structure, considering accuracy, path efficiency, and diversity. Our RL model introduces a more comprehensive reward scheme that promotes reachability, context-relatedness, and conciseness of the extracted knowledge. Especially, the **context-relatedness encourages paths closely related to the given question context, which can help the RL agent reduce the search space** since it can help filter out the irrelevant neightbors when considering the next step.
* **Traing strategy**. While DeepPath uses supervised policy learning, our approach employs Gradient Clipping to maximize the expected cumulative reward, as detailed in Section C.6 of the Appendix. This difference in training strategy allows our model to better adapt to the complexities of knowledge-based question answering.

> **[W2-b & Q4]** Additionally, the authors should also elaborate on how the search algorithm proposed in this article differs from AEKE[2]. From a technical solution perspective, what is the difference between the content of Section 3.2 and the Active Ensemble Policy Learning with MAB described in AEKE[2]?

Thanks for the insightful comments. Our MAB approach differs from AEKE's significantly in training: we've introduced a penalty term, β, to encourage exploration of under-explored prompt formats. This modification allows for a more balanced exploration-exploitation trade-off, which is crucial in our context of selecting optimal prompt forms. We show the performance comparison between with and without this term hereunder.

Table 2:  Ablation study on penalty term used in MAB-base prompt construction.
| Model Variant | CSQA | OBQA | MedQA |
| --- | --- | --- | --- |
| w. penalty term | 81.8 $\pm$ 0.51 | 92.4  $\pm$ 0.69 | 78.1 $\pm$ 0.66 |
| w/o penalty term | 79.5 $\pm$ 1.63 | 91.0 $\pm$ 1.45 | 76.4 $\pm$ 1.89 |


> **[Q2]** What is the difference between "Graph Description" and "Sentences" in the prompt? Is the extracted subgraph structure the same for both, with the difference being that the "Graph Description" contains additional noise, such as "The entity is strongly associated with significant individuals in the tech industry."? 

Thanks for raising this point. Existing methods rely on manually designed or rule-based prompts to present factual knowledge from KGs. These hard prompts are inherently inflexible and rigid, lacking the adaptability to accommodate variations in question semantics and KG structures. To this end, we designed three different prompt templates, including $\mathcal{F}_t$, $\mathcal{F}_s$, and $\mathcal{F}_g$, gudiding the LLM to convert the extracted raw knowledge into textual prompt with corresponding logical reasoning strucure. An MAB is trained to automatically select the most effective prompt template for each question.

**The rationale behind our prompt design**

Generally, based on the complexity of the reasoning background, questions can be roughly categorized into three classes: (1) Simple question, like “What could be used as an electrical conductor?”. (2) Multi-hop reasoning question, like “Which school did Bill Gates' wife graduate from?” (3) Global reasoning questions: “What do cats have in common with most mammals?”

Different types of questions correspond to different reasoning background. For example, simple questions only require basic factual triples, while multi-hop reasoning questions need a reasoning chain, and global reasoning questions require a more complex tree/graph-like reasoning background. To convert the extracted raw knowledge into textual prompt with corresponding logical reasoning structure, we designed three different prompt templates, including $\mathcal{F}_t$, $\mathcal{F}_s$, and $\mathcal{F}_g$.

**Experimental analysis**

Table 3. Accuracy of prompt templates on specific types of questions.

| Prompt Template | Simple | Multi-hop | Global reasoning |
| --- | --- | --- | --- |
| $\mathcal{F}_t$ | 94.1% | 74.4% | 42.1% |
| $\mathcal{F}_s$ | 88.2% | 82.9% | 47.3% |
| $\mathcal{F}_g$ | 85.2% | 70.2% | 78.9% |
> **[W1-c]** As for the two-hop subgraph extraction, the paper does not provide implementation details for the proposed method.

Thanks for your insightful feeadback. **We include the implementation details of the vanilla 2-hop subgraph extraction $P_{sub}$ in Section B.1 of the Appendix.** To make it more clear, we will first explain the rationales behind the design of two knowledge extraction methods before summarizing the implementation details of $P_{sub}$.

**The rationale behind knowldge extraction methods.**

If the background knowledge graph is complete and high-quality, our proposed RL-based knowledge extraction $P_{RL}$ can ideally handle all cases. But in practical implementation, we found that there are a few long-tail entities in KGs which only have few neighbors. In such cases, RL-based path extraction can hardly retrieve reachable or effective paths due to the incompletion and sparsity of the background graph. Instead, directly extracting the whole subgraph through rule-based subgraph extraction can provide more effective knowledge. RL-based retrieval method $P_{RL}$ could make sure our model extract more concise and informative knowledge in most case, while the rule-based subgraph extraction $P_{RL}$ could supplement the former especially when the reasoning background is sparse.

Thus, in this paper, we adopt Multi-Armed Bandit (MAB) to automatically select the most suitable retrieval methods from these two candidates, i.e., $P_{RL}$ and $P_{sub}$. 

**Experimental analysis of knowldge extraction methods.** 

The ablation study on knowledge retrieval method can be found in Table 3 of our submission. Specifically, we compare our model with  $P_{RL}$, the heuristic sub-graph extraction $P_{sub}$, and we also include ‘w/o KG’ as the baseline where the LLM is asked to independently answer the given question with no reasoning background provided. The results shown in Table 3 of our submission clearly indicate the vital role of our proposed path extraction strategies.

**Implementation details of $P_{sub}$.**

Due to page limits, we include the implementation details of the vanilla 2-hop subgraph extraction $P_{sub}$ in Section B.1 of the Appendix. Specifcally, we commence by executing entity linking on $\mathcal{G}$, resulting in an initial collection of nodes, $V_{topic}$. Next, we incorporate bridge entities that appear within a 2-hop path between any two linked entities from $V_{topic}$, yielding the set $V_{retrieval}$. After that, we refine this set by evaluating the relevance score for each node, following the previous study[1]. From this refined set, only the top 200 nodes, based on score, are retained. We extract all edges connecting any pair of nodes in $V_{sub}$, creating the retrieved subgraph $G_{sub}$. Each node within $G_{sub}$ is designated a type based on its association to either the topic entities $Q$ or target entities $A$. 


> **[W1-d]** For LLMs, is it a multi-class or multiple binary (as in the YES/NO from Figure 2) classification problem? With multiple binary classifications, there is the issue of handling multiple (or zero) options being YES—how should this be dealt with? (Situations with all NO or all YES responses may arise.)

Thank you for making this valuable question. In this paer, we evaluate KnowGPT on three multi-choice QA datasets spanning two fields: CSQA and OBQA serve as benchmarks for commonsense reasoning, while MedQA acts as a domain-specific QA benchmark. Given the quesion with several choices, we let LLM give the specifc answer, like A/B/C/D, by providing it with background knowledge about the source entities from the original question and the target entities from all candidate choices. This approach transforms the task into a decision-making problem since we provide the LLM with comprehensive background knowledge for all candidate choices simultaneously, allowing it to make informed decisions based on a holistic view of the available information.
Dear Reviewer t5Eo,

We really appreciate your recognition of our work, and also appreciate your time and effort in providing insightful suggestions that can help further polish our paper. To make it clear, we provide more detailed responses to your comments and suggestions as follows:

> **[W1-a & Q1-a,b]** How the reinforcement learning part on the knowledge graph is trained? On line 185 of the 5-th page, where are the additional training details mentioned in the appendix? Policy gradient is merely an algorithm, but how is the Agent on the Knowledge Graph trained?

Thanks for your question. We appreciate your interest in the training details of our model. In our submission, we focused on describing how we carefully designed a meaningful reward function to ensure that the RL model captures effective and informative knowledge, and **the training detail is introduced in Section C.6 of Appendix**. 

To be clear, the policy network is updated using gradient-based optimization to maximize the expected cumulative reward. To stabilize the training of the policy network, we applied gradient clipping [1] to prevent the gradients from becoming too large. This technique helps mitigate the problem of exploding gradients and ensures stable updates of the network parameters. To demonstrate the effectiveness of gradient clipping, we conducted additional experiments comparing the performance of our model with and without gradient clipping across five runs. The results are presented in Table 1 below (you can also find the results in Table 10 of our original submission). 

Table 1: Abliation study for gradient clipping.
| Model Variant | CSQA | OBQA | MedQA |
| --- | --- | --- | --- |
| w. gradient clipping | 81.8 $\pm$ 0.51 | 92.4  $\pm$ 0.69 | 78.1 $\pm$ 0.66 |
| w/o gradient clipping | 78.9 $\pm$ 1.92 | 90.2 $\pm$ 1.25 | 75.5 $\pm$ 2.01 |

As shown, the model with gradient clipping consistently achieves better performance across all datasets, demonstrating the stability and robustness of our method.


> **[W1-b]** How to optimize the Knowledge Graph Agent together with the Prompt Construction with Multi-armed Bandit as proposed in Section 3.2?

Thanks for the insightful comments. Ideally, we would like to design a end-to-end model: Given a LLM and a domain knowledge graph, the RL model could search from the KG to find the most concise, relevant, and informative knowledge according to the feedback from the LLM. This knowledge is then provided to the LLM to help it accurately answer professional questions. However, training such an RL model is extremely costly. Since RL-based knowledge retrieval is a Markov Decision Process, the RL agent would need feedback from the LLM at each decision step during training, requiring frequent interactions with the LLM and incurring huge API costs.

To reduce the API cost, in this paper, we proposed a two-stage model, splitting the goal into two steps. 
* Knowledge extraction, which aims to extract the most relevant and informative knowledge from the knowledge graph for the target question. We designed three rewards to ensure that the RL model extracts correct, concise, and effective background knowledge. 
* Prompt Construction. It designs three different prompt templates for different questions and convert the extracted raw knowledge into a form that the LLM can most easily understand based on the LLM's feedback. 

The two-stage model can greatly reduce the API cost because we only need to interact with the LLM in the second stage. 

Back to your concern, the training of our model consists of two steps:

- We fisrt train an RL-based knowledge extraction model $P_{RL}$ based on question-answer pairs in the training set, obtaining a well-trained policy network for knowledge extraction. After this stage, given a specific question, we can extract knowledge from the KG using two methods: RL-based policy network $P_{RL}$ and the vanilla rule-based subgraph extraction $P_{sub}$ defined in Section B.1 of the Appendix.
- We can obtain two sets of raw knowledge for each question by applying these two knowledge extraction methods ($P_{RL}$ and $P_{sub}$). To transform this structured raw knowledge into easily understandable text for LLM, we design three different prompt templates for knowledge transformation. Therefore, we have six combinations: two raw knowledge * three prompt templates. To automatically select the most suitable combination to construct the final prompt for LLMs, we trained a MAB model based on the data in the training set.

After that, given a new question in the test set, we can automatically get the most suitable prompt in the text format based on the well-trained RL-based policy network and MAB model.
Dear Reviewer 4L5K,

Great thanks for your affirmation and valuable suggestions. We are delighted that you found our responses to the questions and other reviewers' comments are detailed and helpful. Following your suggestions, we'll incorporate all additional experiments and discussion in our final revision to further strengthen our work.

In our previous response, due to time limits, we only replaced BERT in our model with two advanced entity recognition models. Since then, we have been keeping working on more comprehensive comparisons. Up to now, we have completed the comparison with 5 off-the-shelf models. As shown in Table 1, we list the entity recognition performance of these five models, their model size and the corresponding QA performance when applied to KnowGPT.

Table 1. Comparisons of entity recognition model on CSQA.
| Model |Bert|Roberta|XLnet|Llama2|Llama3|
| -------- | -------- | -------- | -------- | -------- | -------- |
|Model Size|110M | 125M| 117M|7B|8B|
|Precision|93.82%|94.31%|95.05%|95.51%|96.37%|
|KnowGPT  |81.83%|82.11%|82.27%|82.30%|82.95%|

Thank you again for your expertise and recognition of our work. Your expertise and guidance have been invaluable in shaping our revisions. We commit to presenting you with a substantially improved and comprehensive manuscript by incorporating all additional experiments and analysis in our final revision.
Thank you for your detailed responses to the questions and also to the other reviewers. I hope to see some of the analyses added to the final version of the paper. Given that there is still some work to do to strengthen the paper, I will retain my score of 6: Weak Accept.
Dear Reviewer vKLu,

We are deeply grateful for your strong support and for raising the confidence score. Your insightful suggestions are crucial in elevating the quality of our paper. We'll incorporate all the additional experiments and analysis in our final revision. Thank you for your time and expertise throughout this review process.
Thank you for the detailed feedback and additional experimental results. I have updated the evaluation accordingly. Hope these analyses will be added to the final version. good luck!