GIVE: Structured Reasoning of Large Language Models with Knowledge Graph Inspired Veracity Extrapolation
Authors: Jiashu He
Mingyu Derek Ma
Jinxuan Fan
Dan Roth
Wei Wang
Alejandro Ribeiro
Venue: ICML 2025 poster (2025)
Keywords: large language models
reasoning
agent
biomedical question answering
AI for science
knowledge graph
Primary area: deep_learning->large_language_models
TL;DR: LLMs can conduct faithful and efficient reasoning by combining its parametric knowledge and limited external information to tackle difficult scientific tasks.

Abstract: Existing approaches based on context prompting or reinforcement learning (RL) to improve the reasoning capacities of large language models (LLMs) depend on the LLMs' internal knowledge to produce reliable Chain-Of-Thought (CoT). However, no matter the size of LLMs, certain problems cannot be resolved in a single forward pass. Meanwhile, agent-based reasoning systems require access to a comprehensive nonparametric knowledge base, which is often costly or not feasible for use in scientific and niche domains. We present Graph Inspired Veracity Extrapolation (GIVE), a novel reasoning method that merges parametric and non-parametric memories to improve accurate reasoning with minimal external input. GIVE guides the LLM agent to select the most pertinent expert data ($\textbf{observe}$), engage in query-specific associative thinking ($\textbf{reflect}$), and then synthesize this information to produce the final output ($\textbf{speak}$). Extensive experiments demonstrated the following benefits of our framework: (1) GIVE increases the performance of LLMs across various sizes. (2) In some scenarios, GIVE allows smaller LLMs to surpass larger, more sophisticated ones in scientific tasks ($\textbf{GPT3.5T + GIVE > GPT4}$). (3) GIVE is effective on scientific and open-domain assessments. (4) GIVE is a training-free method that enables LLMs to tackle new problems that extend beyond their training data (up to $\textbf{43.5}$\% $\rightarrow$ $\textbf{88.2}$\% accuracy improvement). (5) GIVE allows LLM agents to reason using both restricted (very small) and noisy (very large) knowledge sources, accommodating knowledge graphs (KG) ranging from $\textbf{135}$ to more than $\textbf{840k}$ nodes. (6) The reasoning process involved in GIVE is fully interpretable. Our code is available at https://github.com/Jason-Tree/GIVE

GIVE: Structured Reasoning of Large Language Models with Knowledge-Grapb-Innprdd Veracity Extrapolation
Jiashu He
Mingyu Derek Ma
Jinxuan Fan
Dan Roth
Wei Wang? Alejandro Ribeiro
Abstract
commodating knowledge graphs (KG) ranging from 135 to more than 840k nodes. (6) The reasoning process involved in GIVE is fully in- terpretable. Our code is available at https: //github.
Existing approaches based context prompting or reinforcement learning (RL) to improve the reasoning capacities of large language models (LLMs) depend on the LLMs' internal knowl- edge to produce reliable Ofain-Of-Thought (CoT). However, no matter the size of LLMs, certain problems cannot be resolved in single forward pass. Mcanwhile, agent-based reasoning systems require access to a comprehensive nonparamet- ric knowledge base, which is often costly or not feasible for use in scientific and niche domains. We present Graph Inspired Veracity Extrapolation (GIVE), novel reasoning method that merges parametric and non-parametrie memories im- prove accurate reasoning with minimal external input. GIVE guides the LLM agent to select the most pertinent expert data (observe), engage in query-specific associative thinking (reflect), and then synthesize this information to produce the final output (speak). Extensive experiments demonstrated the following benefits of our frame-
1. Introduction
Context-based methods (Wei al., 2023; Brown al., 2020) to enhance the reasoning ability of large language models (LLMs) (Ouyang et al., 2022: Chowdhery et al., 2022; OpenAI et al., 2024b; Grattafiori et al., 2024; OpenAI et al., 2024a) incorporate examples with logic chains the prompt, allowing the model to generate analogous logical sequences for the given query. Recent investigations (Ze- likman et al., 2024; DeepSeck-Al et al., 2025; Shao et al., 2024) highlighted the substantial promise of reinforcement learning (RL) in improving the generation of high-quality logic. These methods operate on the premise that the para- metric memory of LLM is sufficient to execute accurate reasoning. They demonstrate satisfactory performance in cessoning-comllle tasks that do not require novel knowl- edge, such as mathematical and common sense. However, they fail to achieve comparable enhancements in scientific tasks due to the insufficiency of pre-trained knowledge (Cai et al., 2024; Zhang et al., 2024; Dorfner et al., 2024; Dong et al., 2023; Zhong et al., 2023). This type of information is rare in web data; in fact, regardless of the number of parameters, LLMs face unsolvable challenges using single forward pass (Dziri et al., 2024). Incorporating extemal knowledge becomes necessary to enable the model adapt to tasks beyond the training set.
work: (1) GIVE increases the performance of LLMs across various sizes. (2) In some sce- narios, GIVE allows smaller LLMs to surpass larger, more sophisticated ones in scientific tasks (GPT3.5T + GIVE > GPT4). (3) GIVE is effec- tive on scientific and open-domain assessments. (4) GIVE a training-fiee free method that enables LLMs to tackle new problems that extend beyond their training data (up to 43.5% 88.2% ac- curacy improvement). (5) GIVE allows LLM agents to reason using both restricted (very small) and noisy (very large) knowledge sources, ac-
Recent studies have highlighted the significant potential of incorporating structured knowledge bases (KGs) into LLM inference processes to enhance knowledge provision. Ad- vanced
Department of Computer and Information Science, Univer- sity of Pennsylvania, Philadelphia, United States 'Department of Computer Seience, University Of California, Los Angeles, Los Angeles. United States 'Department of Statistics, University of Califomia, Berkeley, Berkeley, United States. Correspondence to: Jiashu He <jiashuhe@ seas. upenn.eee.
generation (RAG) frameworks (Liang et al., 2024; Panda et al., 2024) identify precise KG from documents to aid accurate information retrieval for specific queries; research efforts are also proposed to pro- mote iterative exploration (Sun et al., 2024; Luo et al., 2024; Xu et al., 2024) and improved modeling of interconnections within knowledge bases to improve coherence (Mavromatis & Karypis, 2024; Edge et al., 2024).
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

GIVE: Graph Inspired Veracity Extrapolation
Question:
Traumatic aortic injury: does the anatomy of the aortic arch influence aortic trauma severity?
Chain-of-Thought
lack of internal knowledge
Prompt: : Let's think step by step.
Response: First, the anatomy of the aortic arch may influence aortic trauma severity, but it is not definitive yes answer. Second, there various factors consider, such the type of trauma. Therefore, the correct answer should be maybe. X
Retridval-Aggmenttce.eeerraaaa
semantically similar but irrelevant information
Based the retrieved knowledge triplets, there no direct relation between the anatomy of the aortic arch and aortic trauma severity. Therefore, the correct answer the question would be no.X
2ji tissuc function affects archaeon
Input Query (x)
DPR Retriever
Zz clinical attribute property of archaeon
Zz: anatomical abnormality affects erchaeon
Textual KB (D)
Pj (z/x,D)
Zq: tissue part of archaeon
Poly|x.z)
fail to retrieve sparse KG
the provided knowledge triplets, there specific information regarding the influence anatomy the aortic arch on aortic severity. Therefore, the
Think-on-Graph Graahh
should be moybe. X
'aortic injury'
injury or poisoning
event
résult
result disruutss/
prune
event entityy organisms
LIM
Relevant Entity in KG
injury or poisoning
obation of mort of
LLM
annmmll cell
anatomical structure
disrupts
part off location off
invertebratte
Crissue
'aortic arch'
anatomical structure
part of
invertebrate
Figure 1: Without gold context, Chain-of- Thoough (CoT) fails because LLM''s internal knowledge fails to form a faithful logic chain; Retrieval- Saaeraa-AAummeeed-GGeetete Graph (ToG) focuses on using LLM to prune the direction of traverse, thus fails for lack of high-quality candidates.
(RAG) retrieves semantically similar but irrelevant information, leading to hallucination: Think-on-
These strategies have proven effective in specific question answering scenarios, assuming the existence of an accessi- ble and retrievable database containing the correct reason- ing logic. However, in scientific realms, the construction of comprehensive knowledge bases is daunting, which requires progress in both domain-specific natural language process- ing (NLP) and field-wide vocabulary standardization (Badal et al., 2019; Verhagen et al., 2012). Enhancing LLM rea- soning with limited external information a more realistic approach. Although the retrieved knowledge lacks direct evidence for problem solving, it embodies the intuition and expertise of curation specialists, such feasible relation collection and potential links among similar entities.
tween query concepts, informed by factual linkages, and grounds these links using LLMs' parametric knowledge. Our method also constructs counterfactual reasoning mit- igate hallucinations and incorporates intermediate entities for multi-step reasoning. Specifically, GIVE identifies concentrated group of entities closely associated with the query. By exploring potential relations within pertinent knowledge graph (KG) concepts, we develop a reasoning framework that encompasses all conceivable concepts and connections that could enhance query resolution. We incor- porate additional intermediate concepts by selecting multi- step reasoning strategies that most effectively address the questions. GIVE comprises retrieved expert information, internal knowledge acquired through pre-training, and in- novative relations that unite analogous concepts via verac- ity extrapolation. To complete the reasoning framework, counterfactual links among nodes are integrated avert hallucinations. Ultimately, GIVE is structured reasoning framework of LLMs that (1) retrieves externl knowledge to increase the informativeness of responses; (2) conduct diver-
In this paper, we seek to overcome the constraints re- lying solely on either internal or external knowledge by introducing GIVE, graph-inspired veracity extrapolation framework. GIVE emulates the cognitive processes em- ployed by human experts, draws inspiration from related knowledge, and conducts associative thinking. It populates the limited information with provisional connections be-
2

---

GIVE: Graph Inspired Veracity Extrapolation
Question:
Traumatic aortic injury: does the anatomy of the aortic arch influence aortic trauma severity?
Deductive Reasoning on KG triplets
Based on the retrieved knowlecige, aortic arch is part of anatomical structure, which is the focation of injury poisoning. Anatomical structure also affects cell function, which further affects injury positioning, suggesting the anatomy of the aortic arch could Influence the severity of aortic trauma. Therefore, the correct answer this question should be yes.
GIVE
KG Concepts
Queried Entities
MIPS
anatomy
anatomical abnormality
anatomical structure
tissue
aortic injury
injury poisoning
affects
offeett
location of
location of
is
part of
cell function
influence
locotion of
location of
affects,
affects
affects
aortic trauma severity
injury or poisoning
clinical attribute
aortic arch part of
anatomical structure
related to
Figure 2: Reasoning process of GIVE. Solid lines indicate expert information, while dashed lines depict results from the "veracity extrapolation" procedure. GIVE first constructs entity group for each queried concept, then induce inner-group connections using its internal knowledge. The expert's cross-group connections serve as evidence, guiding the LLM to extrapolate the veracity of potential relationships among similar concepts.
gent reasoning on the limited expert information that does not directly solve the query, by extrapolating expert triplets to correlated queried concepts, termed "veracity extrapo- lation." We test our proposed approach on both scientific and open-domain tasks, including biomedical, common- sense, and open-domain question answering. GIVE con- sistently achieves superior performance across all datasets, utilizing KGs of differing sizes and densities, surpassing all internal/external knowledge reasoning benchmarks, which demonstrates the effectiveness and reliability of the pro- posed framework. Encouragingly, GIVE enables smaller LLMs such as GPT3.5T to achieve better performance than advanced models like GPT4 scientific tasks. GIVE pio- neers in elevating LLM's reasoning capabilities with mini- mal external cues activate its intrinsic problem-solving capacity.
edge base B", and employs a retrieval model 3 such that retrieve,(B"... , x) R(x).
In this work, we generalize the benefits of both methodolo- gies deliberately not presuming the existence of 0" or the comprehensive nature of an accessible knowledge base B We utilize retrieved structured evidence to template problem solving. directing LLM to leverage its intrinsic knowledge scientific tasks backed by minimal external information. GIVE is training-free framework that directs LLM to con- duct reasoning on the limited external information that does not directly solve the query:
(ationalecivE waiioaalooe((,,eetteeevee...B retrieeeavee (B.))) + R(x) (I)
2. Problem Statement
While theoretically, this offers no benefit over RAG (Lewis et al., 2021) for generating answers with retrieved context, numerous studies (Ge et al., 2023; Bang et al., 2023) have shown that LLMs lack expertise in scientific domains and cannot construct a multi-step logical chain (Wei et al., 2023; Wang et al., 2023; Jiang et al., 2024) to connect a complex query with the retrieved incomplete information that cannot directly solve it. GIVE, in this context, provides additional guidance to LLMs, helping them to populate the expert knowledge towards the key components of the query by integrating both parametric and non- parametrii knowledge.
Assume that R(x) represents the accurate logic chain to solve the query æ. Reasoning on internal knowledge tech- niques (Wei et al., 2023; Brown al., 2020; DeepSeek-AI al., 2025) make use of reinforcement learning or context- based prompts to direct the LLM to produce accurate logic chain with high probability, relying solely on the model's intrinsic knowledge. This signifies a dependence on the parameter 8", such that rationale (x) + R()). In contrast, reasoning with retrieval frameworks (Sun et al., 2024), posits the availability of comprehensive knowl-
3

---

GIVE: Graph Inspired Veracity Extrapolation
3. Method
3.3. Inner-group connections
For structured information extraction, our study employs Knowledge Graph (KG) as the external knowledge base. G {Eg, &c) where Eg
Focusing on each N in Section 3.2, we establish links between the queried entity and its semantically related con- cepts within its entity group. This aims to prompt the LLM to explore related concepts broadly, rather than focusing solely on the queried entity. The challenge of directly induc- ing relationships between two queried entities is addressed by identifying possible links between two sets of similar concepts. We employ the LLM to suggest relationships between the queried entity and each in-group concept. All such knowledge appended to an affirmative knowledge set. Consider an entity group with I queried entity and p additional KG concepts N = {e VE Vu .... Beph,, for S m $ p
A KG is defined ((urr,)) u, E Eg,rE RG}, with Eg being the set of en- tities and RG the set of relations. GIVE prompts inductive reasoning by: 1) breaking down the query into fundamental concepts and attributes; 2) forming entity groups by linking the queried entities with related concepts that exist in the KG; 3) inducing connections within groups between the queried entity and related concepts using LLM's parametric knowledge; 4) establishing affirmative and counter-factue factual inter-grod group links by probing and refining potential connec- tions, and leveraging intermediate concept groups for multi- hop reasoning in complex queries. Appendix I provides a comprehensive explanation of the GIVE algorithm, while Appendix D includes examples of each subroutine.
(e (em)) - LLM + (e (62:22m m Mem
(5)
3.1. Query Information Extraction
3.4. Inter-group connections
For a given query 2, GIVE initially utilizes the LLM to extract the entity and relation sets P.g and Re, as represented by:
3.4.1. POTENTIAL RELATIONS INDUCTION
We first determine all possible relationships that could link any pair of nodes within the two groups. We evaluate two categories of potential relations: (1) Relations specified in the question. These are the vital connections required for the ultimate QA task. (2) KG relations present between these groups. Due to the semantic similarity of concepts within each group, existing goos--roup KG connections may link other entity pairs across groups. Formally, for KG G, entity groups N; and Nj. we define the set of potential relations R as:
x - LLM + Ez, R.
(2)
Here, E. (c (g.,e) represents the top-k concepts, while R== ( (r2.,,....7" } comprises the top-m relations or attributes identified in the query. For example, the query "Is melatonian effective for insomnia?" contains entity set E,
[melatonin, insomnia] and relation set Rg = [effective
for).
3.2. Entity Group Construction
This step aims to assist LLM to induce connections between queried entities from expert connections between similar KG entities. We do this by exploring the knowledge space to form a cluster of related concepts for each significant entity identified in the query. For each et Fiz, GIVE uses a pre- trained LM encoder w encode knowledge base concepts and find most similar concepts to each entity in the latent space by comparing cosine similarities:
R =
No H Rs u R
(6)
3.4.2. INTERMEDIATE NODE GROUP DISCOVERY FOR MULTI-STEP REASONING
In scientific fields, directly establishing links between spe- cific concepts is sometimes not feasible. For instance, to address a query regarding a drug's effect on disease, log- ical approach is to associate these through (drug, compound, disease) links, forming the statement: "since a certain com- pound exists within the drug, and this compound alleviates particular diseases, can be inferred that the drug may treat the discase." " To support such multi-step reasoning process, GIVE investigates novel node groupings used interme- diate steps in reasoning. GIVE directs L.M to identify the most beneficial 2-hop paths between groups, helping with the question-anee -anheirng task. Through the intermediate node of an optimal 2-hop path, GIVE forms an intermediate entity group using the process outlined in Section 3.2.
y (V Ve Vxpp argmin, u (())), w()))}
jeEo
(3)
The set Y includes entities semantically akin to e and e is added to Y to create the entity group Nk:
No {e
(4)
Relating queried concepts with the semantically similar KG entities broadens the reasoning scope from strict information retrieval to inferring relationships over a broader range of relevant concepts.

---

GIVE: Graph Inspired Veracity Extrapolation
3.4.3. STRUCTURED EXPERT KNOWLEDGE GUIDED REASONING
where GIVE, GIVE GIVE corresponds to the an- swer generation process that uses (1) only affirmative knowl- edge; (2) both affirmative knowledge and counterfactual knowledge; (3) the whole set of knowledge, which contains affirmative, counterfactual, and ground truth KG knowledge.
Finally, GIVE introduces associative thinking from these non-parametric evidence, including concept groups and all potential relations between them.
Relation Assignment Using External Evidence. If an edge exists in the external knowledge graph between two entities, we inherit the ground truth relationship from the original KG. When verbalizing such edges, we use definite tense to convey the high certainty of this fact.
3.6. Running time
Consider m. entity groups with n concepts each and candi- date relations between two groups. Inner-group connections (Section 3.3) require (m)) LLM calls. For inter-group connections (Section 3,4.3), the LLM calls needed are equal to the candidate potential connections for "veracity extrap- olation", O(rm' ). As shown in Section 4.7, GIVE per- forms best with 1 or 2. Appendix C.I further shows: (1) m averages 3 or 4 for all datasets; (2) is bounded by 4 across datasets; (3) GIVE's running time and context length stay reasonable as KG size or sparsity increases; (4) GIVE's complexity can be further decreased by batch pruning and adding summarization agents to shorten knowledge.
Veracity Extrapolation with Internal Knowledge. Po- tential relations between node groups identified in Section 3.4.1 help to suggest possible node connections. It is vital to solidify the internal knowledge to validate the model's decision or discard incorrect answers with clear context. GIVE, therefore, instructs the L.MM to assess each poten- tial edge between node groups. Confirmed connections are added to the extrapolated affirmative knowledge set, while rejected ones offer counterfactual insights to avoid model hallucination.
4. Experiments
Identifying Open Relations for Novel Connections To avoid the limitations of the knowledge base's s scope, GIVE also offers LLM the freedom to independently generate short phrase describing the relation of each entity pair.
4.1. Research Questions
The experiments in this section aim to address the following research questions: (1) Is GIVE robust in enhancing L.LMs* reasoning with a limited (very small) or noisy (very large) external knowledge base? We explore this in Section 4.4
3.5. Progressive answer generation
and Section 4.5 using knowledge graphs ranging from 135 to over 840k entities, with statisties in Table 2. (2) Can GIVE improve reasoning across both general and specific domains? We test biomedical reasoning in Section 4.4, com- monsense reasoning in Section 4.5, and open-domain rea- soning in Section 4.6. (3) What are the impacts of GIVE's different components, parameters and implementation choices? This is examined through ablation studies in Sec- tions 4.7.1, 4.7.2 and in Appendices B.2, B. and B.3. (4) What other factors might affect the performance of GIVE? We provide a comprehensive analysis in Section 4.7.3, dis- cussing what makes effective "inspirations". (5) How does GIVE's cost compare to other methods? A detailed ex- amination of run time and context length is in Appendix C.I.
From the reasoning process presented in Section 3.4.3, GIVE obtained three types of knowledge: (1) Affirma- tive knowledge set that contains all knowledge affirmed by LLMs' parametric knowledge, including -nner-group connections; the cross-group open connections, and the ex- trapolated connections. We refer to this knowledge set as (*)x). (2) Counterfactual knowledge set that contains all rejected potential cross-group connections, which is referred to R²(2) (z), (3) Ground-truth KG knowledge set R2 (x))). To prevent hallucination, we adopt progressive manner to generate the final answer by first giving only the affirmative knowledge set. GIVE further directs LLM to refine this answer with the full context and the counter-factuse knowl- edge set. The final answer is generated by providing all previous context and the ground-truth knowledge set. Given generator py, and the retrieved knowledge sets described above, the answer generation process by GIVE is defined as:
4.2. Experiment Settings
We examine questions that require additional reasoning by disregarding any "gold" context or inclusive knowledge base. In PubmedQA (Jin et al., 2019), we challenge competing methods by supplying LLM only with the question state- ment and the retrieved facts, excluding any ground-truth context where the answer is self-contained. For BioASQ (Krithara et al., 2023), we focus on questions from Tasks 2B, 3B, and 4B, disregarding long answers to assess the
GIVE. po
(7)
GIVE (1 (y |r, (0 3 R2) R((
UIVEancec (I, 32, )
== =p R²))) 1/, (x), (x,
(x))
(9)
5

---

GIVE: Graph Inspired Veracity Extrapolation
Table Summary of Dataset statistics.
Task
KG
|
[
Datasets
QA Type
PubmedQA (Jin et al., 2019)
Yes-No
Biomedical Reasoning
UMLS (I.i et al., 2023)
135
5,877
BioASQ (Krithara et al., 2023)
Yes-No Multple-Choice
ProcessBank (Berant et al., 2014)
10% ConceptNet (Speer et al., 2018)
223,863
208,510
Commonsense Reasoning
50% ConceptNet (Speer et al., 2018)
607,483
1,042,550
CommonsenseQA (Talmor et al., 2019)
Multiple-Choice
Full ConceptNet (Speer et al., 2018)
844,158
2,085 ,099
Opendomain Reasoning
10% ConceptNet (Speer et al., 2018)
223,863
208,510
TruthfulQA (Lin et al., 2022)
Text Generation
precision of the short responses each method provides. For Processbank (Berant et al., 2014 4), ground-truth annotations are omitted, providing only the question statement and choices. All three datasets are paired with compact UMLS (Li et al., 2023) containing just 135 nodes. Regarding Com- monsenseQA (Talmor et al., 2019), built with ConceptNet (Speer et al., 2018), we randomly sample ConceptNet sub- graphs with varying edge ratios.
GIVE is adaptable for LLMs of various sizes, effectively averting hallucination. GIVE enhances the reasoning abil- ity of LLMs of different sizes < Llama3. GPT4o-mini). Notably, GIVE surpasses retrieval- based techniques that rely on limited external knowledge. Un- der our challenging experiment settings, triplets from DPR (Karpukhin et al.. 2020) and ToG (Sun al., 2024) didn't solve queries directly, the irrelevant information causes hallucinations. This occurs especially with strong models (GPT4/4o-mini). In this challenging scenario, GIVE con- sistently boosts their performance by properly alleviating hallucinations from irrelevant knowledge.
4.3. Competing baselines and backbone LLMs
We compare GIVE with I/O prompting (Brown et al., 2020), CoT prompting (Wei et al., 2023), text-based RAG (Lewis et al., 2021), ToG (Sun et al., 2024) and GraphRAG (Edge et al., 2024). Since GraphRAG is not suitable for large- scale KGs and struggles with context comprehension using smaller LLMs, it is excluded from certain comparisons. For biomedical reasoning tasks, we evaluate each method using GPT3.5-turbo, GPT-4, GPT40-mini, and Llama3. 1-70BB Instruct, aiming to demonstrate GIVE's ability bridge the knowledge gap between smaller and larger LLMs. This is crucial in scenarios where acquiring knowledge from pretraining is challenging. For commonsense reasoning, we test GIVE on GPT3.5-turbo using ConceptNet with varying edge ratios, highlighting its robustness in managing both limited and noisy information. In open-domain reasoning, showcase its effectiveness in general-domain QA tasks.
Integrating affirmative, counterfactual and expert knowledge into increases the reliability of its reasoning processes. The results by GIVE utilizes the entirety of generated knowledge, as elaborated in Section 3.5, and deliv- ers highly stable performance. This implies that the strategy for retrieving counterfactual knowledge, as discussed in Section 3.4.3, plays a crucial role in guiding the reasoning process in the specific-domalt tasks. It also underscores the importance of effectively using retrieved expert knowledge that does not immediately resolve the query but contributes valuable context for more informed answer-generation.
4.5. Commonsense Reasoning Results
GIVE demonstrates robustness and generalizability in reasoning with limited or noisy data. As shown in Ta- ble 4, on the complete ConceptNet, GIVE increases ac- curacy by 3.4% and 4.9% over RAG (Lewis et al., 2021) and ToG (Sun et al., 2024). Retrieving information from a dense, domain-specific knowledge base is challenging due to numerous semantically-sinn
4.4. Biomedical Reasoning results
GIVE enhances the effectiveness of smaller-sized LLMs beyond that of leading models. Our initial observation reveals that, with minimal external knowledge, GIVE allows GPT3.5T to consistently outperform GPT4 on biomedical reasoning tasks, which are challenging due to the difficulty in training and accessing comprehensive knowledge bases. This challenge is particularly evident when comparing k- shot prompt results in various models, especially in tasks like PubmedQA (Jin et al., 2019) and ProcessBank (Berant al., 2). In these instances, GIVE effectively integrates training and inference knowledge using a sparse KG with just 135 nodes, incurring no additional training expense.
but irrelevant entities and triplets. Incorporating them directly in context easily causes hallucination. These findings confirm GIVE's robustness in generating useful information for prompting structured reasoning in LLMs using noisy external knowledge sources.
4.6. Open-domain Reasoning Results
GIVE is effective in both domain-specific and open- domain reasoning tasks. Alongside domain-specite rea-
6

---

GIVE: Graph Inspired Veracity Extrapolation
Table 3: Performance comparison on biomedical QA. Retrieval cerieval-bbased methods are given access to small UMLS (Li et al., 2023). We highlight the best performance gain of GIVE compared to different categories of competing methods.
GPT S-turbo BioASQ
GPT4
4o-Miniiin BloASQ
Meta-Llama-3. 1-0---3.1-0---stttii
Methood/Daaod PuhMedQA
ProcessBank
PuhMediQ
BioASQ
ProcessBank PuhMediQA
ProcessBank PubMedQA
BloASQ
ProcessBank
Internal knowledge reasoning
110 prompt
46.2
43.5
67.3
42.2
88.2
64.8
23.4
88.7
79.4
48.0
91.0
85.4
2CT
48.6
63.5
37.8
80.4
79.3
81.4
50.4
91.3
External knowledge (text) reasoning
RAG
13.4
40,9
67.3
26.4
24.3
78.9
15.2
16.3
84.9
49.8
45.4
84.4
Esternal knowledge (KG) reasoning
ToG
17.6
18.0
66.8
19.1
154
81A
21.8
10.1
84.4
38.4
31,0
85.9
GraphRAG
23.4
10.3
26.5
11.2
10.1
84,9
Structured reasoming with internal and external knowledge method)
SIIVE
44.4
82.6
72.9
50.0
90.0
82.7
26.0
89.5
85,9
56.0
91.7
$6.4
6GIVE
49.8
$6.1
73.9
50.2
80.6
83.3
27.4
81.9
87.4
56.2
91.7
86.9
GIVV Fasose
53.6
88.2
73.4
87.8
81.9
86,9
56.0
92.6
Best %) 7.4/10.2/36.0 44.744..//77.9
6.6/6.6/7.1
12/4/23 8/31.1
9 9.6665.777 7/8.8 24.04A//244
(.0/12.22/56 10.2/73.2/994 8,0/2.5/3.0
8.2/6 1.647.266166 2.6/2.5/1.0
Table 4: Performance comparison on CommonsenseQA, using GPT3.5-turbo. Retrieval- based methods are given access to sub- graph of ConceptNet with different portions of randomly sampled triplets. We highlight the best performance gain of GIVE compared to different categories of competing methods.
GPT-4o Win Rate on TruthfuiQA
12.6%
GIV__*tc
GIVE_atote
RAG UO prompt
Category
prompt
14.0%
Commonsensee QA
# Method / % of triplets
10%
50%
100% (Full)
22.9%
Internal knowledge reasoning
IO prompt CoTl prompt
71.8
QIE
35.7%
8.8%
6.0%
72.2
External knowledge (text) reasoning
30
50
Rate (%)
3 RAG
70.4
70.6
71.3
External knowledge (KG) reasoning
Figure 3: GPT4o win rate TruthfulQA (Lin et al., 2022) in %.
4 ToG
69.7
71.2
69.8
Internal and external knowledge reasoning
It was observed that GIVE, yields the most accurate answers compared GIVE and GIVEasete> as the dataset focuses information veracity. For example, "What is the original meaning of SOS?" Here, counter- factual knowledge is not useful. Humans don't 't answer by considering "SOS does not mean..." statements, which could lead to hallucinations in LLMs. Therefore, in open-domain datasets, is advisable to prioritize affirmative knowledge and GIVE, answers.
GIVE,
73.3
73.6
74.2
GIVE,
73.4
73.6
74.2
GIVEs
73.8
74.7
Best Gain(+%)
1.7/3.1/3.8
2.0/3.2/2.6
2.9/3.4/4.9
soning tasks, we conduct supplementary experiments to evaluate GIVE's performance on open-domain QA dataset. We assess each method's performance on the TruthfulQA (Lin et al., 2022) answer generation task. For retrieval- based methods (RAG, ToG, GIVE), we employ a 10% Concept- Net to test their capability in generating open-domain answers with limited information. We utilize GPT4o to evaluate the win rate, as presented in Figure 3. For each question, GPT4o receives the best answer and each method's s response, scores them based on their semantic similarity to the ground truth, and selects the highest scoring answer as the winner. As indicated in Figure 3, GIVE attains the highest score in 50.3% of the TruthfulQA questions (Lin et al., 2022), demonstrating its efficacy in both scientific reasoning and open-domain question answering.
4.7. Ablation Study
4.7.1. NUMBER OF ADDITIONAL ENTITIES
GIVE demonstrates optimal performance with a mini- mal number of additional entities to conduct divergent reasoning. The pivotal parameter for GIVE is the count of supplementary entities added to each concept group. We examine its impact GIVE's performance through exper- iments in biomedical reasoning employing GPT3.5-turbo with KG entity numbers ranging from 0 to 3. As depicted in Figure 4, GIVE's performance initially enhances with an increase in KG entities per group from 0 to 2, yet declines when increased 3. This pattern is consistent across all
7

---

GIVE: Graph Inspired Veracity Extrapolation
65
PubmedOA
BloASQ
74.5
Procesbbak
50
74.0
Acouracy())
40
Accuracy
62
80
30
25
GVE_
GIV
GME_a
GIV
GIE GIV_ ute
71.5
GME
GIE utere
eNE_erote
15
72
71.0
additional entrties per group
additional entities group
additional entities per group
100
PubmedOQA
100
BioASQ
100
ProcessBank
60
Avg accuracy1
Avg accuracy(
Avg posuureeyyy
80
80
60
40
40
40
20
20
20
0.04
0.09
0.13
0.17
0.22
0
0.05
0.12
0.18
0.24
0.30
0.035
0.072
0.107
0,143
0,179
Expert knowledge ratio
Expert knowledge ratlo
Expert knowledge ratio
Figure 4: Different factors that may impact GIVE's performance: (upper) GIVE's performance VS no.additional entities per group. (lower) GIVE's performance VS expert knowledge ratio of the "veracity extrapolation" process
datasets. The noticeable improvement in performance with an increase in additional concepts from 0 to underscores the success of our "Graph Inspired Veracity Extrapolation" method, as detailed in Section 3.4.3. This suggests that LLMs' potential for divergent thinking has been underesti- mated, due to the emphasis on retrieving precise knowledge.
from the parametric knowledge, and need be guided by frameworks like GIVE.
4.7.3. WHAT MAKES GOOD "INSPIRATIONS"?
To further scrutinize the factors leading to GIVE's high performance, we classify the dataset questions by "expert knowledge ratio" in GIVE's "veracity extrapolation" pro- cess on For each question, we compute the propor- tion of expert knowledge in the extrapolated knowledge set. Specifically, for query x, the expert ratio defined [R*(())
Table 5: Performance (Accuracy %) comparison of GIVE using only inner-group connections, inter-group connections, both in- ner/inter group connections on 100 randomly selected questions from each dataset.
RR) where (x),, R*) and R R(xr) represent sets of expert, affirmative, and counter-factual knowledge retrieved from Section 3.4.3. We assess GIVE's average accuracy across questions with varying expert ra- tios and display findings in Figure 4. We observe a positive correlation between GIVE's performance and the expert ratio. This occurs because more expert information pro- vides GIVE with concrete candidate relations and entities for LLM to perform divergent thinking during the "veracity extrapolation" process. Limited expert information leaves no "inspiration" for divergent thinking, the final knowledge set consists of only the inner-group connections and the open-relation inter-group connections, detailed in Section 3.4. GIVE uses ground truth knowledge as a qualitative "supervision" " to guide the model on possible entity relation- ships, supporting our premise that both external and internal knowledge are insufficient alone for knowledge-intensive scientific tasks, prompting the design of GIVE to bridge this gap.
Datsee/Knowledge
Gnoe--rroup only
Inter-Group only
Inner Inter Group
PubmedQA
14
52
56
BioASQ
50
88
ProcessBank
76
Commonsence(AA
66
74
76
4.7.2.
AND INTER GEROP CONNECTIONS
We conduct experiments to test the importance of each type of connections introduced by GIVE. The results are pre- sented in Table We conclude that both inner-group and inter-group connections are necessary, but inter-group con- nections contribute the most the performance. This is because the inner-group connections are added bridge the query and the knowledge graph concepts, and it's purely built on the model's ş internal knowledge, so they are easier to be automatically inferred by the model. The inter- inter-group connections produce the necessary knowledge to bridge the different entity groups thus prompt a faithful reason- ing process to solve the query, which are hard to reason
8

---

GIVE: Graph Inspired Veracity Extrapolation
5. Conclusion
Harding, B., Huang, B., Clark, P., and Manning, C. D. Modeling biological processes for reading comprehen- sion. In Moschitti, A., Pang, B., and Daelemans, W. (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1499-1510, Doha, Qatar, October 2014. Association for Computational Linguisties. doi: 10.31 1/D1//1--11111 URL
We introduce Graph Inspired Veracity Extrapolation (GIVE), structured reasoning framework to enhance LLM reason- ing in scientific fields. This is achieved by leveraging diver- gent thinking guided by limited expert input. Rather than fo- cusing on explicit information retrieval or relying solely on the parametric knowledge, GIVE adopts an approach akin to human cognitive reasoning process, by combining high- level expert information with the model's internal knowl- edge through "veracity extrapolation". This bridges the lim- ited expert knowledge and queries, thus significantly boosts performance in both domain-specinf and open-domain rea- soning tasks, demonstrating robustness with limited and noisy knowledge bases. GIVE addresses hallucination is- sues in retrieval-dependent methods on incomplete knowl- edge sources, at the same time, adding information for more comprehensive answers compared to reasoning methods that solely rely on internal knowledge. Our study highlights the great potential of LLMS to perform divergent reasoning with minimal external guidance. Further research is needed on leveraging external knowledge as a catalyst to "inspire" LLMs to reason, rather than providing a comprehensive "long-answer" style context. More generalizable RI. post- training strategies are also needed, on tasks where additional information is needed to generate the correct logic chain. GIVE holds particular relevance for deploying L.LMs in scientific fields where exhaustive training and retrieval of knowledge are impractical. Intelligent agents draw inspi- ration from external cues to perform accurate reasoning, not thinking hard by itself or just echo the context.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners, 2020. URL https:// arxiv. org/ hwtg///b////// 2005 14165.
Cai, H., Cai, X., Chang, J., Li, S., Yao, L., Wang, C., Gao, Z.. Wang, H., Li, Y., Lin, M., Yang, S., Wang, J., Xu, M., Huang, J., Xi, F., Zhuang. J., Yin, Y., Li, Y., Chen, C., Cheng, Z., Zhao, Z., Zhang, L., and Ke, G. Sciassess: Benchmarking Ilm proficiency in scientific literature anal- ysis, 2024. URL https::: /wwxiv.ivvvv/vv////vv/eeeeaaiwva org/ abs/ /2003. 01976.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., and Barham, P. Palm: Scaling lan- guage modeling with pathways, 2022. URL https: //arxiv. www.oorg/b/aooo.eeoe /aa//ab/22200 /2004 02311.
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q.. Ma, S., and Wang, P. Deepscek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URL https:// a arxiv, arxiv.org/abi.. www.oooggg/b//eoo. 22501. 12948.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Dong. X.,
Wang. Y., Yu, P. S., and Caverlee. J. Probing explicit and implicit gender bias through Imm conditional text generation, 2023. URL httth::/t/aaiiiiiee orgg/ abs 02t//211.0000 00306.
References
Badal, V., Wright, D., Katsis, Y., Kim, H.-C., Swafford, A., Knight, R., and Hsu, C.-N. Challenges in the con- struction of knowledge bases for human microbiome- disease associations. Microbiome, 7, 09 2019. doi: 10.1 00.11866440018--11-44422222
Dorfner, F. J., Dada, A., Busch, F., Makowski, M. R., Han, T., Truhn, D., Kleesiek, J., Sushil, M., Lammert, J., Adams, L. C., and Bressem, K. K. Biomedical large languages models seem not to be superior to gen- eralist models on unseen medical data, 2024. URL https: http///xxxvvvrro/ abss /200..2488..0 10833.1333
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., Do, Q. V., Xu, Y., and Fung, P. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and in- teractivity, 2023. URL https://///attt org/ abs/ 2302.04022
Dziri, N., Lu, X., Sclar, M., Li, X. L., Jian, L., Lin, B. Y., West, P., Bhagavatula, C., Bras, R. L., Hwang, J. D., Sanyal, S., Welleck, S., Ren, X., Ettinger, A., Harchaoui, Z., and Choi, Y. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36, 2024.
Berant, J., Srikumar, V., Chen, P.-C., Vander Linden, A.,

---

GIVE: Graph Inspired Veracity Extrapolation
Edge, D., Trinh, H., Cheng, N., Bradley, J., Chao, A., Mody, A., Truit, S., and Larson, J. From local global: A graph rag approach to query-focused summarization, 2024. URL httttt::///aatt
Liang, L.,, Sun, M., Gui, Z., Zhu, Z.. Jiang, Z., Zhong, L.,
Qu Y., Zhao, P., Bo, Z., Yang, J., Xiong, H., Yuan, L.,, Xu, J., Wang, Z., Zhang, Z., Zhang, W., Chen, H., Chen, W., and Zhou, J. Kag: Boosting llms in professional do- mains via knowledge augmented generation, 2024, URL. https: http:://arrav..orraa///000.... 13731.
org/ abw//40//444/ 16130.
Ge, Y., Hua, W., Mci, K., ji, j.. Tan, J., Xu, S., Li, Z., and Zhang, Y. Openagi: When Ilm meets domain experts. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 5539-5568. Curran Associates, Inc., 2023.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods, 2022. URL https:
//arxiv. org /asgaa///////eeeeeeeeegge//saaa/ 2209..795..
Luo, L., Li, Y.F,, Haffari, G., and Pan, S. Reasoning on graphs: Faithful and interpretable large language model reasoning, 2024. URL https: http///axivvv//// 2310.010 01061.
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A. Al-Dahle, A., Letman, A., Mathur, A., and Schelten, A. The Ilama 3 herd of models, 2024. URL https:
//arxiv. org/ abs/ / 220224007720 00.00.2200 21783.
Mavromatis, C. and Karypis, G. Gnn-rag: Graph neural retrieval for large language model reasoning, 2024. URL petps://arxiin http://xxi...rrrrb//////// /0t.ab////20...3....... 20139.
Jiang. B., Xie, Y., Hao, Z., Wang, X., Mallick, T., Su, W. Taylor, C. J., and Roth, A peek into token bias: Large language models are not yet genuine reasoners, 2024.
OpenAI, :, Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., and Welihinda, A. Gpt-4o system card, 2024a. URL hhttps://ariiss org/ abs / 2410.220222 21276.
URL httph:ttp///om /ttpvaawvvvv//.///////.......wv. org/ abs/////0000 11050.
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., and Lu, X. Pub- medqa: A dataset for biomedical research question an- swering. In Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCE -INNLP) pp. 2567-2523 2019.
OpenAL, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., and Altman, S. Gpt-4 technical report, 2024b. URL https: http://wwwwww...a////ooooooo
08774.
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.t. Dense passage retrieval for open-domain question answering. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2200.emms main.550. URL https: acl wonthhlogy..... org/ 2200 emn p-ma 5.500
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback, 2022. URL https: //arxiv. org/ abs 2200222222222 02155.
Panda, P., Agarwal, A., Devaguptapu, C., Kaul, M., and P, P. A. Holmes: Hyper-relational knowledge graphs for multi-hop question answering using llms, 2024. URL https: http://axxxvvvo/ org/ abs /200 06027.
Krithara, A., Nentidis, A., Bougiatiotis, K., and Paliouras, G. Bioasq-qa: A manually curated corpus for biomedical question answering. Scientific Data, 10: 170, 2023. URL https: //te:///do..ooo https://ddo.org/....... 841597-023-0208
Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X.. Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models, 2024. URL https: //app:://axivv.. org abs /al../2402.033 03300.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., tau Yih, W., Rock- täschel, T., Riedel, S., and Kicla, D. Retrieval-augmented intensive generation for knowledge- knowledge-intensive nip tasks, 2021.
Speer, R., Chin, J., and Havasi, C. Conceptmet 5.5: An open multilingual graph of general knowledge, 2018. URL https: /ttp//iivvv.//////////..////aata http://bb/111..999999
Li, D., Zhu, B., Yang. S., Xu, K., M., He, Y., and Wang, H. Multi-task pre-training language model for semantic network completion. ACM Trans. Asian Low-Resown Lang. Inf. Process., (1),), nov 2023. ISSN 2375-4699. doi: 10.1145/3627704, URL https::ppp/.. htt.dddi.ooogg//////// 1145/3627704.
Sun, J., Xu, C., Tang. L., Wang, S., Lin, C., Gong, Y., Ni, L. M., Shum, H.-Y., and Guo, J. Think-on-graph Deep and responsible reasoning of large language model on knowledge graph, 2024.
10

---

GIVE: Graph Inspired Veracity Extrapolation
Talmor, A., Herzig, J., Lourie, N., and Berant, J. Com- monsenseqa: A question answering challenge target- ing commonsense knowledge, 2019. URL https:
00937.
Verhagen, W. Bermell-Garcia, P., van Dijk, R. E., and Curran, R. A critical review of knowledge- engineer- An identification of research challenges. Advanced Engineering Informatics, 26(1):5-15, 2012. ISSN 1474- 0346. doi: http:://hhiiagg//00000////222221 0...0 URL httppe//wwww..cceeeedddcccc
ing:
science/article//eeeeeee /ttaan///arricceee////////116100
Network and Supply Chain System Integration for Mass Customization and Sustainable Behavior.
Wang, X., Wei, J., Schuurmans, D., Le, Q.. Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency im- proves chain of thought reasoning in language mod- els, 2023. URL https:/ //tp:s://///vav////aa//aa org/ wtwgg//bb//02..3 11171.
Wei, J., Wang. X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, Chain-of-thought prompting elicits reasoning in large language models, 2023.
Xu, Y., He, S., Chen, J.. Wang, Z., Song. Y., Tong, H., Liu, K., and Zhao, J. Generate-on-grapp Treat Ilm as both agent and kg in incomplete knowledge graph question answering, 2024. URL httht:://aaaieiii //axivv./iiivv./iaa org/ /bs/ 1404.14744
Zelikman, E., Harik, G., Shao, Y., Jayasiri, V., Haber, N., and Goodman, N. Quiet-star: Language models can teach themselves to think before speaking, 2024. URL https: //trx/a./vvv///////////////aaa org/ org/abs/// awws/b/s/33 //..//223...222 09629.
Zhang, Y., Hou, S., Ma, M. D., Wang, W., Chen, M., and Zhao, J. Climb: benchmark of clinical bias in large language models, 2024. URL https: petetts::/ariiiee w/t.xi.vvovooo/./o.oo.........oa
abs/2407.00 052500.
Zhong, R., Du, X., Kai, S., Tang, Z., Xu, S., Zhen, H.-L., Hao, J., Xu, Q., Yuan, M., and Yan, J. Llm4eda: Emerg- ing progress in large language models for electronic de- sign automation, 2023. URL https: //tpx//rxvvvv////////. org/ abs/24011 12224.

---

GIVE: Graph Inspired Veracity Extrapolation
A. Algorithm for GIVE
offer a comprehensive outline of the entire GIVE procedure and explain its detailed algorithm in Algorithm
Algorithm 1 GIVE
Input:
Entity groups N {N Possible relations between two entity groups R₂ (R2) /ij-1> Knowledge Graph G
Output:R(xr), the approximated gold knowledge set that helps to solve query
Rx)x)) + Ø
(x) 2
(*)x) Ø
for all queried entity e'g and their relevant concepts y N2 do
1s build inner-group connections +/
(e', v LLM (e,, ris y²) R²xx) - (x) U ((e,, r2 }+)} end for
for (N) pair in N₂ Nr do
1s build -nter-group connections */
retrieve all triplets (x) &g connecting any node in N and any node in N (x) = (x) U Ri (z)
for all triplets (n',, rh m²d in (N² Ry N$) do
(n rá, n²) LLM {yes, no, maybe}
if yes then
(x) + (x) u {(n rf. mg))
end if
if no then (x) (x) ((n not r., n2)
end if
end for
end for return (())))R(())
B. Additional Ablation studies
In addition to Section 4.7, we conduct more detailed ablation studies for GIVE to study the robustness of the proposed method and other factors that may influence its performance. All experiments in this Section based on 50 randomly generated examples for each dataset.
B.1. Different ways of prompting
We perform additional experiments study how different prompting strategies influence the performance of GIVE. We verbalize the retrieved knowledge and prompt them in the form of triplets and text, the results are presented in Ta- ble 6. We notice that in most cases, prompting the knowl- edge triplets yields to higher accuracy than prompting knowledge in text. This is because the structure of triplets naturally provides an easier way for the LLM to connect the related entities and build faithful logical chain solve the question. However, for text-based information, ad- ditional analyzing step is needed to understand the text before links the useful information together, which is intensive queries where the volume of additional knowledge is high.
Table 6: Performance of GIVE using different prompting methods on 50 randomly chosen examples for each dataset. We highlight in green the betler-performed prompting method and the performance difference.
GPT3.S-turb
# Prompting Method dataset
PubmedQA
BioASQ Processbank CSQA
GIVE,
Triplet prompt
86
76
74
2 Text prompt
46
86
74
62
GIVE,,.
Triplet prompt
86
74
76
2 Text prompt
54
84
74
70
GIVEarese
a difficult task for ietnnsvv
Triplet prompt
88
70
76
2 Text prompt
54
84
72
68
12

---

GIVE: Graph Inspired Veracity Extrapolation
GIVE_a
GivE_(a+c)
GIVE_(a+c+e)
PubmedQA(50 samples)
BioAsQ samples)
60
100
50
80
40
Accuracy
Accuracy(%5)
30
40
20
no. k-shot examples
no. k-shot examples
Processbank(50 samples)
CSQA(50 samples, using 50% of ConceptNat)
70
70
60
60
Accuracy
50
Accuracy
50
40
40
10
10
10
10
examples
examples
Figure Sensitivity analysis of GIVE on different number seeded examples.
B.2. Number of seeded examples
Table 7: Performance of GIVE using GPT and encoding Centence"Transoomee of different sizes to search for relevant entities to build entity group (Section 3.2). Results based on 50 randomly generated samples each dataset. We highlight the results from the best performing model in green.
To better understand how difficult it is for LLMs to get the generalized ability to adopt the knowledge generated by GIVE to build the structured reasoning chain, we study the performance of GIVE by providing different number of examples in the prompt. For yes-no datasets PubmedQA (Jin et al., 2019) and BioASQ (Krithara al., 2023), randomly choose k of {0. 1, 2, 3, 4, 5) examples. For multiple-choice datasets Processbank (Berant et al., 2014) and CSQA (Talmor et al., 2019), we choose k of (0, 1, 5, 7, 10). The results are presented in Figure 5.
GPT3.5-turbo
Encoding madel(size) dataset
PuhmedQA
BioASQ Processbank CSQA
GIVE,
panpprase-lllerttsmalll--
44
84
74
68
2 all-MiniLM-LLG-222000
32
86
76
74
M.I-Mini....112222112000
4 all mmpeet-bee-224220M)
38
66
64
We observe that although the performance of GIVE in- creases as we give more seeded examples in the prompt, the only one large performance upgrade happens when we increase the number of examples from 0 to 1. This implies that GIVE is generalizable framework for the LLM to easily adopt. The high performance of GIVE does not rely on large number of examples, but stems from the high quality of the synthetic data generates.
CIVF,.c
1 guaghhase-allbrt-mmlll---- M.MMiii.M-L66-229MM)
54
82
76
70
3 all MiniLM L12,x2(120M)
52
62
70
4 all all-mpaat-bsee-x2222M.
52
86
62
64
GIVE
nonphrass-allerr-smlll---222
52
72
2 MiniLM v2280()
52
70
76
3 all Ml--MiniLMML11222112000
54
82
62
70
all-mparr-nseex242200
52
64
13

---

GIVE: Graph Inspired Veracity Extrapolation
Avg no. entily groups
Avg no. candidate relations
Avg % of questions requiring intermediate groups 70%
avg candidetes cross-group relations
90%
entty groups
% questions
50%
40%
30%
20%
10%
PubmedQA BioASQ Processbank CSQA
PubmedQA BiASSQ Processannk CSQA
PubmedQA BiAASQ Processbenk CSQA Dataset
Dataset
Dataset
Figure 6: Average number of entity groups (left), average number of candidate relations between two groups (middle) average pereentage of questions that requires intermediate entity group (right) for cach dataset included in Section 4.4 and 4.5 for runs. For CSQA, we report the results on 50% triplets version of ConceptNet.
B.3. Encoding model size
In Section 3,2, we employ Sentence Transformer as encoder model to measure the text similarities for entity group construction. We investigate the impacts of using different sizes on the performance of GIVE, and demonstrate the results in Table 7. We see that although larger size encoder models achieve better sentence embedding or performance semantic search performance, small to middle size encoders tend to perform more consistently on all datasets. For the sers-perroming GIVE the 80M encoder (all-MiniLM--6- results show that larger size encoders do not necessarily better measure text similarity between specific domain terms. On the other hand, the performance of GIVE does not rely the size of the models employed, which enhances the efficiency of GIVE.
achieves 8% higher accuracy than the 420M (all-mpnet-basee
The
C. Detailed analysis of GIVE
C.1. Efficiency of GIVE
In Section 3.6 we discussed the efficiency of GIVE and concluded that the key factors that influence the number of LLM calls required by GIVE is the number of entity groups detected for the query and the number of candidate relations between each pair of entity groups. To conduct more detailed study the scale of them, we GIVE 5 times 50 randomly selected questions for each of the datasets we included in Section 4.4 and 4.5, and we report the average number of entity groups, average number of candidate relations to connect two entity groups, and average percentage of questions that requires intermediate entity groups (3.4.2) for multi-step reasoning. The results are presented Figure 6.
We observe that on average, GIVE requires around entities groups for each question in the Biomedical datasets (PubmedQA, BioASQ, Processbank), between each datasets, there could be 1 to 6 candidate relations. For commonsenseQA, 4 entity groups on average are detected because the dataset has 5 candidate options, be- tween each pair of entity groups, only candidate relation detected general. We also notice that 60% the ques- tions in PubMedQA requires intermediate group. That the reason why PubmedQA tends need more entity groups than BioASQ "yes-no" QA dataset. This im- plies one of the potential method to improve efficiency of GIVE is to disable intermediate group detection. On the other hand, we can use the LLM to prune the candidate connections in batches, which means in Section 3.4.3, instead of asking LLM "yes" or "no" for each potential connection, we can prompt the LLM with a set k of relations and let select out which ones are true of false, which will devide the total number of L.M calls by the factor of for GIVE.
Table 8: Token consumption comparison between GIVE and ToG
Dataset
Avg Input Tokens
Avg Output Tokens
GIVE
ToG
GIVE,.:1
ToG
PubmedQA BioASQ
14518.5
12701.1
183.1
104.2
7970.3
7010.0
80.0
60.2
ProcessBank
19460.5
11995.6
232.5
100.5
CSQA/10% ConcepiNet
5321.1
4934.8
34.9
23.1
CSQA/50% ConceptNet
7203.0
6704.1
45.2
34.8
CSQA/100% ConceptNet
7398.7
6679.7
46.3
36.6
14

---

GIVE: Graph Inspired Veracity Extrapolation
We further conduct experiments to compare the efficiency of GIVE with RAG (Lewis al., 2021) and ToG (Sun et al., 2024), in terms of running time and context length, for every experiment setting include in Section 4, the results presented in Table 9. ALso, to conduct an in-depth analysis of token consumption, we compare the average number of input.ouiput tokens for GIVE and ToG (Sun et al., 2024), as in Table 8
The computational cost of GIVE remains reasonable as we increase the size and density of the KG. (1) In terns of running time, when we increase the density (num- ber of edges) to x5 or x 10 on ConceptNet (Speer al., 2018), we a sub-linear running time increase for GIVE. Even with n=2, GIVE achieves shorter or comparable run- ning time with ToG (Sun et al., 2024). This proves the O(rm² )?) running time of GIVE, as discussed in Sec- tion 3.4.3. When we increase the density of the KG, the only factor that will change is r, which is the number of relations between two entity groups, the increase of which is strictly upper- aoper-bounded by the increase of total number of edges in KG. Besides, the running time of GIVE is independent to the size of the KG, we see this if we compare its running time on small UMLS of 135 nodes and the ConceptNets which have hundreds of thou- sands of entities, because GIVE always selects the most important entities related to the query induce knowl- edge, and the cost of the entity selection phase is very low if we pre-compute the embeddings.
Table 9: Efficiency comparison between GIVE and RAG (Lowis ct al., 2021). ToG (Sun et al., 2024) on 100 random questions. We report the average running time in seconds, context length in no. words, and accuracy in %. For RAG, we retrieved top 10 knowledge and for ToG, we use search depth=5 to maximize their performance. For GIVE, we report the results for both n=1 and n=2, where n is the number of additional KG concepts per group. (x), 1' (x), R' are the retrieved affirmative knowledge set, counter-factual knowledge set and expert KG knowledge set.
Context Length words) T2(G) (G) T2 GIVE, GIVE GIVE,,..e
Acc(S)
Method/Datase
time(s)
PubmedQA UMLS
1 RAG
2.7
64.7
14
2 ToG
15.9
73.8
16
3 GIVE,..
33.6
192.5
105.9
16.1
35
45
45
GIVE,
103.8
45633
486.1
$7.6
41
48
48
BioASQ on UMLS
1 RAG
2.8
66.5
43
ToG
10.3
3 GIVE
15.3
83.6
33.8
9.0
79
81
83
GIVE,.:
45.3
205.5
176.5
32.1
80
84
90
(2) In terms of context length, GIVE does not suffer from overwhelming long context on large or dense KGs. In fact, the context length of GIVE is also largely decided by the number of relations between two groups. That is why we see that on PubmedQA where the pross-group KG knowledge is rich, GIVE can induce large number of affirmative and counterfactual knowledge, thus provides the biggest per- formance increase compared to RAG or ToG. It is also worth noticing the recent LLMs are making fast progress in overcoming the limitation of input length. For exam-
Pracersbank UMLS
1 RAG
2.8
61.7
68
15.6
54.2
3 GIVE,
35.2
15L.2
226.9
7.5
67
68
68
GIVE, 93.8
354.0
649.5
28.9
71
72
CSQA on 10% ConceptNee
1 RAG
0.6
30
64
2 ToG
39.3
106.6
67
3 GIVE,
26.5
41.1
38.
0.1
70
71
71
GIVE,
36.3
93.2
CSQA on 50% ConceptNer
1 RAG
1.1
66
2 TeG
102.2
217.0
67
3 GIVE
74.0
39.9
43.5
0.2
69
64
65
GIVE,
82.0
86.9
75
ple, Llama 3.1 Ol et al., 2024b) support up to 128k tokens, compared to Llama 3 which supports only 8.192 tokens. For GPT series models, the maximum context window size also grows from 4.1k tokens (which translates to around 3k words) of of GPT4 (Chowdhery et al., 2022) and GPT4o. Such progress makes scaling inference time compute techniques like GIVE much more applicable, and we expect even large progress in maximum tokens on further models. GIVE is far from reaching such context length limitations according to Table 9. There are also concrete solutions to easily further reduce both running time context length of GIVE: When building the knowledge sets (Section 3.4.3), we can apply a divide-and-conquer manner to prune the knowledge in batches. When generating answers, we can apply similar techniques in GraphRAG (Edge al., 2024), to use additional agent to summarize the retrieved knowledge sets into shorter paragraphs before feeding to the answer generator.
CSQA on full ConceptNet
RAG
1.7
30
2 ToG
125.2
213.7
63
3 GIVE,.)
124.2
41.8
45.9
0.5
67
69
68
GIVE
129.9
83.4
89.9
72
77
77
to 32k
GIVE provides a good trade-off' between accuracy and efficiency. Although the hyper- n=2 yields to the best accuracy in most scenarios, we see that even when we use only one additional KG entity per group, GIVE achieves better or at least the same accuracy, compared to ToG and RAG. These results further emphasize the importance of the proposed framework to incorporate structured information during inference time reasoning, at the same time, provide the practicer with balanced alternative to use n=1 with limited compute resource, but at the same time achieve good performance.
GIVE is able to generate high quality synthetic data using very limited external knowledge. If we compare the accuracy
15

---

GIVE: Graph Inspired Veracity Extrapolation
PubmedQA
BioASQ
only GIVE correct
only GIVE correct both correct
both correct
both incorrect
both incorrect
only ToG correct
only ToG correct
15%
12%
13%
%
31 %
69%
3%
Processbank
CommonsenseQA
only GIVE correct both correct
only correct both correct
both incorrect
both incorrect
only ToG correct
only ToG correct
13%
13%
8%
20%
17%
14%
62%
Figure 7: The proportions of questions answered correctly by GIVE and ToG, on PubmedQA, BioASQ, Processbank and Common- senseQA
increase offered by GIVE and the context length, we see a positive correlation between them. Related discussion is also included in the previous subsection that the performance of GIVE related to the expert knowledge ratio and the number of retrieved knowledge. The results further proved that the generated knowledge is of very high-quality. As a result, GIVE has great potential to serve as synthetic data generating algorithm in other fine-tuning tasks, such RLHF RL for reasoning.
C.2. Detailed comparison with retrieval-based methods
In addition Table 3, we conduct detailed performance comparison against text-based reasoning method RAG (Lewis et al., 2021) and KG based reasoning method (Sun et al., 2024), we calculate the portions of questions answered correctly by each method and present the statistics in Figure and Figure 8.
We observe that on three of the four datasets (BioASQ, Processbank, CommonsenseQAA) we included in our experiments, most of the questions answered correctly by ToG or RAG is also answered correctly by GIVE. We see this by calculating the
ratio only ToG/AG correct both correct For example, its 1 11% on CommonsenseQA for ToG, meaning that 89% of the questions answered correctly also answered correctly by GIVE. On PubmedQA, this ratio large because RAG and ToG both get
only ToG/RAG coreet
16

---

GIVE: Graph Inspired Veracity Extrapolation
PubmedQA
BioASQ
only GIVE correct
only GIVE correct correct
both correct
both incorrect
both incorrect
only RAG correct
only RAG correct
7%
5%
54%
38%
Processbank
Commonsense@A
only GIVE correct both correct
only GIVE correct both correct
both incorrect
incorrect
only RAG correct
only RAG correct
15%
13%
9%
18%
16%
59%
62%
Figure 8: The proportions of questions answered correctly by GIVE and RAG, PubmedQA, BioASQ. Processbank and Common- senseQA
17

---

GIVE: Graph Inspired Veracity Extrapolation
very poor performance, and for most questions, GIVE is the only method that derive the correct answer. The results further prove that only very few questions in these scientific-domain datasets be directly answered by the knowledge contained in the sparse KG, this further highlights the importance of the proposed "Veracity Extrapolation" process to combine internal knowledge and external knowledge to solve challenging scientific questions.
D. Prompts and Example Responses
D.1. 10 Prompt
You are helpful assistant answers given question about medical knowledge with yes, no or maybe, based on your own knowledge.
[k-shot EXAMPLES]
Q: Traumatic aortic injury: does the anatomy of the aortic arch influence aortic trauma severity? Output: no
D.2. CoT Prompt
You are a helpful assistant that answers given question about medical knowledge with yes, no or maybe, based on your own knowledge.
[k-shot EXAMPLES]
Q: Traumatic aortic injury: does the anatomy of the aortic arch influence aortic trauma severity? Let's think step by step. Output: maybe
D.3. RAG Prompt
For RAG, we provide both the correct textual knowledge and reasoning chain for each of the k-shot examples.
You are helpful assistant that answers a given question about medical knowledge with yes, no or maybe, based on retrieved textual knowledge "entity relation entity" from an expert knowledge graph.
[k-shot EXAMPLES]
Q: Traumatic aortic injury: does the anatomy of the aortic arch influence aortic trauma severity? Knowledge: [Textual knowledge]
Output: no
D.4. ToG Prompt
We follow the official implementation of ToG (Sun et al., 2024) and use the default prompts. We replace the k-shot examples to be examples randomly selected for each dataset, and we provide the correct reasoning chain. Overall, we use exact the same k-shot examples for ToG and our method to guarantee fair comparison.
Exemplar prompt for retrieving top entities:
Please retrieve the top entities (separated by semicolon) that contribute the question. [EXAMPLES]
Q: Traumatic aortic injury: anatomy of the aortic arch influence aortic trauma severity? Output: [Entities retrieved]
Exemplar prompt for pruning relations:
18

---

GIVE: Graph Inspired Veracity Extrapolation
Please retrieve relation that contributes to the question the most from the given relation list. The answer must be one of the given relations.
[EXAMPLES]
Q: Traumatic aortic injury: does the anatomy of the aortic arch influence aortic trauma severity?
Relations: [Relations list]
Output: [Relationship selected]
Exemplar prompt for pruning entities:
Please score the entities' contribution to the question on a scale from 0 to (the sum of the scores of all entities is 1). [EXAMPLES]
Q: Traumatic aortic injury: anatomy of the aortic arch influence aortic trauma severity?
Relation: [Relationship selected]
Entities: [Entities list]
Output: [Entity selected]
Exemplar prompt for evaluating knowledge sufficiency:
Given a question and the associated retrieved knowledge graph triplets (entity, relation, entity), you are asked to answer whether it's sufficient for to answer the question with these triplets and your knowledge (yes no).
[EXAMPLES]
Q: Traumatic aortic injury:
anatomy of aortic arch influence aortic trauma severity?
Knowledge triplets: [currently retrieved knowledge triplets]
Output: [yes/no]
Exemplar prompt for ToG answering the question:
Given a question and associated retrieved knowledge graph triplets (entity, relation, entity), you are asked to answer question with these triplcts and your knowledge.
[k-shot EXAMPLES
Q: Traumatic aortic injury: does the anatomy of the aortic arch influence aortic trauma severity? Knowledge triplets: [retrieved knowledge triplets]
Output: maybe
D.5. GraphRAG prompt
We follow the networkx implementation of GraphRAG (Edge et al., 2024) and use the default prompts. We replace the k-shot examples to be examples randomly selected for each dataset, and we provide the correct reasoning chain. The k-shot examples are provided during the intermediate answers generating step. Overall, we use exact the same k-shot examples for GraphRAG and our method to guarantee fair comparison.
Exemplar prompt for summarizing each detected community:
Summarize the following community of entities and relationships. [Description of communities] Output: [List of summaries of each community group]
Exemplar prompt for generating intermediate answers from community summaries:
19

---

GIVE: Graph Inspired Veracity Extrapolation
You are helpful assistant that answers given biomedical question based on the provided summary. You find some examples below: [k-shot examples]
Query: [Question]
Summary: [Summary list]
Output: [List of intermediate answers]
Exemplar prompt for combining intermediate answers into final answer:
You are helpful assistant that answers biomedical question with yes, or maybe, based on some intermediate answers. Query: [Question] Intermediate answers: [List intermediate answers]
Output: [yes/no/maybe]
D.6. GIVE Prompt
Exemplar prompt for extracting and ranking the entities in the question:
Please retrieve the top entities that contribute to the question. Answer only the top entities, separated by comma. [EXAMPLES]
Question: Traumatic aortic injury: does the anatomy of the aortic arch influence aortic trauma severity?
Output: ('traumatic aortic injury', 'anatomy', 'aortic arch', 'aortic trauma severity"]
Exemplar prompt for extracting the relationships in the question:
Please retrieve the relationships that connect the given entitics in the question. [EXAMPLES]
Question: Traumatic aortic injury: does the anatomy of the aortic arch influence aortic trauma severity?
Entities: traumatic aortic injury, anatomy, aortic arch, aortic trauma severity Output: ['influence']
Exemplar prompt for generating relationships between two given entities:
You are a helpful assistant that answers short relationship in few words between two given biomedical entitics. [EXAMPLES]
Entities: traumatic aortic injury, injury and poisoning Output: a"
Exemplar prompt for determining if relations exists between cross group entities:
You are a helpful assistant that answers yes, or maybe depending on the correctness of the given statement. Injury poisoning is the result of organism function. Is it true? Output: "No"
Exemplar prompt for selecting optimal 2-hop path for intermediate entity group construction:
20

---

GIVE: Graph Inspired Veracity Extrapolation
You are helpful assistant that selects from the given knowledge facts (entity, relation, entity, relation, entity), that is important the given question. Knowledge Facts:
(steroid, affects, organ or tissue function, affects, invertehrate),
(steroid, affects. experimental model of disease, manifestation of, injury or poisoning). (anatomical abnormality, manifestation of, organism function, affects, clinical attribute)... Question to answer: Traumatic aortic injury: does the anatomy the aortic arch influence aortic trauma severity? Output: (anatomical abnormality, manifestation of, organism function, affects, clinical attribute)
Exemplar prompt for generating GIVE,;
You helpful assistant that answers given question about medical knowledge with yes, maybe, based the retrieved knowledge triplets (entity, relation, entity) from your own knowledge. The return must be one yes, no or maybe. [k-shot EXAMPLES!
Q: Traumatic aortic injury: does the anatomy of the aortic arch influence aortic trauma severity?
[AFFIRMATIVE KNOWLEDGE TRIPLETS]
eg: ('anatomical abnormality", "affects', "organism function'), ('injury or poisoning', 'affects', "organism function ("anatomy', part of", "aortic arch"), ('injury or poisoning'. 'affects', 'organ or tissue function') )) ('aortic arch', 'location of", 'injury or poisoning')..
Output: maybe (GIVE.)
'maybe' by considering the relationship between the anatomy of the aortic arch and the severity of aortic trauma. The knowledge triplets suggest
Logic Chain: reached the
anatomy of aortic arch may influence severity of aortic trauma, as anatomical structure correlates with clinical attributes and impacts clinical attributes. Additionally, the severity of aortic trauma may correlate with clinical attributes, which can be affected by traumatic aortic injury. However, the relationship between the anatomy of the aortic arch and the severity of aortic trauma is not definitively stated in the knowledge triplets, hence the answer "maybe'.
Exemplar prompt for generating GIVE,
You helpful assistant that answers given question about medical knowledge with yes, maybe, based the retrieved knowledge triplets (entity, relation, entity) from your own knowledge. [k-shot EXAMPLESS
Q: Traumatic aortic injury: does the anatomy of the aortic arch influence aortic trauma severity? [AFFIRMATIVE ENOWLEEDGE TRIPLETS]
eg: ('anatomical abnormality', 'affects', 'organism function'), ('injury poisoning'. 'affects'. 'organism function' ("anatomy', "part of", "aortic arch'), ('injury or poisoning'. "affects", 'organ tissue function') )) ('aottic arch', 'location of", 'injury or poisoning"). )... A: maybe (GIVE,)
Additional knowledge triplets: MOOUNTER-FACTUA KNOWLEDGE TRIPLETS]
eg: ('organism', ; 'not result of", "aortic trauma severity'), ) C'inuury poisoning', 'not complicates', 'anatomical structure') ) ('aortic arch', "not influence', 'injury or poisoning".. )...
Output: yes (GIVE,,.)
Logic Chain: By utilizing the additional knowledge triplets provided, it can be inferred that the severity of aortic trauma is result of experimental model of disease, steroid, traumatic aortic injury, injury or poisoning. This suggests that the severity of aortic trauma influenced by these factors. Therefore, anatomy of the aortic arch may indeed influence the severity of aortic trauma in cases of traumatic aortic injury.
Exemplar prompt for generating Gi
21

---

GIVE: Graph Inspired Veracity Extrapolation
You are helpful assistant that answers given question about medical knowledge with yes, maybe, based the retrieved knowledge triplets (entity, relation, entity) from your own knowledge, and the knowledge triplets from expert knowledge base. The return must one yes, no or maybe. [k-shot EXAMPLES]
Q: Traumatic aortic injury: does the anatomy of the aortic arch influence aortic trauma severity?
[AFFIRMATIVE KNOWLEDGE TRIPLETS]
eg: ('anatomical abnormality', "affects', "organism function'). ('injury or poisoning', 'affects'. "organism function ("anatomy', part of", "aortic arch") ) ('injury or poisoning', 'affects', "organ or tissue function") ) ("aortic arch', 'location of", 'injury or poisoning")... )...
A: maybe (GIVE.)
Additional knowledge triplets: (COUNTER-FACTUR) TRIPLETS]
eg: ('organism', "not result of". 'aortic trauma severity"), ) C'injury or poisoning'. 'not complicates', 'anatomical structure') ("aortic arch', noot influence', 'injury or poisoning".. )...
A: yes (GIVE,t.)
Additional knowledge triplets retrieved from expert knowledge base: [EXPERT KG KNOWLEDGE TRIPLETS]
eg: ('ijuury poisoning'. 'result of", 'anatomical abnormality") ('steroid", "causes', 'injury poisoning"), Cinjuuyy or poisoning'. 'complicates', "anatomical abnormality') ) ('anatomical abnormality', 'result of", 'injury or poisoning )...
Output: yes (GIVE
Logic Chain: utilized the external knowledge from the expert knowledge base modify my previous answers by considering the additional knowledge triplets that retrieved. Aortic arch part of anatomical structure. which is the location of injury or poisoning. Anatomical structure also affects cell function, which further affects injury or positioning, suggesting anatomy of the aortic arch could influence the severity of aortic trauma. Therefore, the correct answer to this question should be yes".
22


Reviews:
[{"review_id": "QuGA0zbtz0", "reviewer": "Reviewer_JLdU", "summary": "This paper introduces GIVE to facilitate the reasoning ability of LLMs in specific domains. GIVE extracts the relevant information of a knowledge graph and bridges it with the query by using LLM’s internal knowledge to justify the veracity of the extrapolated knowledge. GIVE also incorporates counterfactual knowledge and progressive answer generation to alleviate hallucination caused by the additional knowledge. The authors conducted extensive experiments using both domain-specific and open-domain benchmarks, utilizing KGs of various sizes and sparcities. The empirical results proved the effectiveness and generalizability of GIVE.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> From Figure 4 we see that the expert knowledge set plays an important role for GIVE. However, from Table 8, GIVE achieves better performance than ToG and RAG when using n=1 and only the affirmative knowledge set, why can GIVE achieve good performance without using the expert knowledge set?\n\nWe are grateful for the insightful discussion brought up by the reviewer. It is essential to further elucidate that GIVE employs the expert knowledge extracted from the Knowledge Graph (KG) with the purpose of enabling faithful reasoning through knowledge extrapolation, rather than being directly utilized for answer generation. As demonstrated in the experiments reflected in Figure 4, the findings indicate that the efficacy of GIVE's \"veracity extrapolation\" is intricately linked to the ratio of expert knowledge. This ratio evaluates the degree to which the KG is pertinent to the questions, encompassing both the entities and their interrelations. The reason for this relationship is that expert knowledge constitutes the KG's connections among the pertinent entities, as detailed in Section 3.2 and depicted by the solid lines in Figure 2 of our initial manuscript. Although this expert knowledge might not directly facilitate the answering of questions, as it is not immediately connected to the queries, it serves as a valuable source of \"inspiration\" for the model to execute divergent reasoning. This is a primary rationale behind proposing a framework that integrates retrieval with reasoning when dealing with external knowledge that is sparse and noisy.\n\n\n\n> In Table 3, why does CoT harm the performance of GPT4, as CoT does not introduce new knowledge which causes hallucination, as clarified by the authors.\n\nWe appreciate your question. In Section 4.4, we critically evaluate various approaches by assessing their performance on scientific benchmarks that demand intensive reasoning. Large language models (LLMs) lack sufficient parametric knowledge to effectively address these complex queries. This is primarily because it is impractical to extensively train these models on highly specialized domains. Our observations, derived from comparing the outcomes of GPT-3.5-turbo and GPT-4 on datasets such as PubMedQA and ProcessBank, illustrate that simply scaling up pre-training does not necessarily enhance accuracy. In such instances, the Chain-of-Thought (CoT) approach seeks to promote a logical step-by-step reasoning process that relies exclusively on the model's internal knowledge, thus results in fluctuating performance levels due to its inherent randomness.\n\n> In Figure 3, there is a huge performance gap between GIVE_a and GIVE_a+c, GIVE_a+c+e. Can the authors clarify the cause of this performance gap\n\nThank you for raising this question. Figure 3 illustrates our experiments with TruthfulQA, an open-domain text generation task. This dataset is included to demonstrate that GIVE is effective across both domain-specific and open-domain tasks.  GIVE_a achieves superior results primarily because inquiries are directed at information accuracy. Consider the example query for TruthfulQA \"What is the original meaning of SOS?\" In this scenario, incorporating an affirmative knowledge set naturally leads to a more coherent context, whereas integrating counter-factual information tends to result in hallucinations. Additionally, as we employed the 10\\% version of ConceptNet in this experiment, the retrieved expert knowledge is not directly associated with the query. Instead, it is utilized to motivate the model towards \"veracity extrapolation,\" which significantly contributes to the excellent performance of GIVE_a. This is elaborated upon in Section 4.6 (lines 332-338), and the solid to dashed line process in Figure 2 of our original manuscript.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Previous works on LLM with external knowledge integration heavily focuses on the task of KGQA, where the gold answer or reasoning path is contained in the given KG, which is diMerent from the research problem defined in this paper. GIVE specializes in using limited information to prompt LLM reasoning in the specific domains, the authors include suMicient discussion of the previous research in the introduction, the motivation to propose GIVE for the hard tasks where retrieving from high-quality knowledge sources is clear.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Strength:\n\n1. The idea is novel. GIVE innovates in directing LLM to further populate the retrieved information to formulate a multi-step reasoning chain, differing from the reasoning methods on self-knowledge or the retrieval methods for gold context.\n\n2.  Experiments are comprehensive. In addition to various benchmarks, the author includes ablation studies to study the sensitivity of GIVE to parameters, and “expert knowledge ratio”, These findings further support the intuition of GIVE.\n\n3. Analysis is sufficient. The Supp further validates the claim on the efficiency of GIVE with detailed context length and accuracy of the proposed method with different parameters, providing insights to the community.\n\nWeakness:\n\n1. From Figure 4 we see that the expert knowledge set plays an important role for GIVE. However, from Table 8, GIVE achieves \nbetter performance than ToG and RAG when using n=1 and only the affirmative knowledge set, why can GIVE achieve good performance without using the expert knowledge set?\n\n2. In Table 3, why does CoT harm the performance of GPT4, as CoT does not introduce new knowledge which causes hallucination, as clarified by the authors.\n\n3. In Figure 3, there is a huge performance gap between GIVE_a and GIVE_a+c, GIVE_a+c+e. Can the authors clarify the cause of this performance gap", "other_comments_or_suggestions": "no", "questions_for_authors": "no", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZIaCMsDG0t", "reviewer": "Reviewer_mrWr", "summary": "This paper proposes a novel reasoning method based on LLMs and an external knowledge graph. The motivation of this paper is to address the reasoning quality under the situations that the knowledge graph is sparse. This is reasonable as CoT has no external knowledge, while RAG and ToG suffer from the scale of the external base knowledge. To address these challenges, the authors first transfer the queries into entities and relations formed in knowledge graphs, and then construct the entity groups by computing the cosine similarities in the embedding space. After this, the authors execute the inner-group to filter broader related concepts and inter-group connections to link both KG relations between groups and question related relations. The experiments show significant improvements in biomedical QA and robustness and scalability.\n\n## update after rebuttal\nI would like to maintain the current rating.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Compared to RAG that can retrieve any text information, graph-based retrievers (also for GraphRAT, etc.) naturally require an external knowledge base graph that is of high quality and is typically not yet learned by the LLMs. This might not be a `weakness’ but the common limitation of graph-based retrievers.\n\nWe appreciate the recognition of our contributions and agree with the common limitation of graph-based retrievers as mentioned by the reviewer. GIVE plays a pivotal role in addressing the challenge where high-quality Knowledge Graphs (KG) are not readily available. Knowledge Graphs possess certain distinct advantages over textual corpora. The primary goal of GIVE is to bridge the gap between the limited parametric knowledge typically available and the accurate reasoning required for scientific queries by effectively populating limited expert information towards the query. Our choice of KGs as a data source is due to their structured relational knowledge, which naturally provides a reasoning pathway. This choice simplifies the task to effectively creating connections between the expert KG knowledge with the ultimate question-answering task. In such scenarios, the process of constructing a relevant entity set and retrieving the connections between different groups proves efficient due to the inherent structure of graph data. Recent studies [1] have focused on converting text into KGs, and we are optimistic that textual corpora and KGs will become mutually interchangeable with advancements in this field of research.\n\n[1] KGGEN: EXTRACTING KNOWLEDGE GRAPHS FROM PLAIN TEXT WITH LANGUAGE MODELS.\n\n\n> GIVE requires the preprocessing on the knowledge graph, the extraction of query information, and retrieval, which take some time. Some of the processes are before inference (like preprocessing), while others (extraction and retrieval) are during inference.\n\nWe appreciate the constructive feedback from the reviewer. We agree with the reviewer's insight that the reasoning process used by GIVE demands additional inference time, particularly in the construction of entity groups and the extrapolation of veracity. To substantiate our claims, we present comprehensive experiments in Table 8 in the appendix. These experiments demonstrate that when the number of entities per group (n) is set to 1, GIVE surpasses other competing baselines in performance, while maintaining similar or even lesser execution times. Notably, this trend is more significant when applying larger and denser Knowledge Graphs (KGs). Efficiency of GIVE could be further improved by incorporating batch pruning for the veracity extrapolation process. Moreover, as elaborated in Section 5, future research could involve automating the \"veracity extrapolation\" procedure, for instance, by developing a corresponding process reward that can be incorporated into Reinforcement Learning (RL) post-training frameworks. We are confident that the contributions of GIVE offer substantial benefits for future endeavors in this area of research.\n\n> The method introduces some additional hyperparameters like m and n, increasing the difficulty of adjusting hyperparameters.\n\nWe appreciate your effort in highlighting this issue. It is important to elaborate further that GIVE's only hyperparameter is the number of entities per group, denoted as \\( n \\). In contrast, \\( m \\) represents the number of entity groups which is a characteristic inherent to the query, as explained in Sections 3.1 and 3.2. For each queried entity, an entity group is created using its analogous concepts in the Knowledge Graph (KG). The findings illustrated in Figure 4 and Table 8 substantiate that GIVE offers commendable performance even with \\( n = 1 \\). This facilitates GIVE in achieving an optimal balance between efficiency and accuracy without the necessity of hyperparameter tuning. Moreover, experiments detailed in Appendix C.1 reveal that the average value of \\( m \\) is typically 3 or 4 across all datasets.\n\n> Since I did not find the code link, I just want to ask the authors whether they would provide the corresponding repository to the public?\n\nThank you for raising this issue. We hold open-source research in high regard and are committed to including a link to the repository containing all the relevant codes and datasets in the revised manuscript.", "claims_and_evidence": "Yes, the claims that the authors made are intuitively correct. For example, the authors list the challenges of CoT, RAG, ToG and show why they are unworkable with a sparse knowledge base graph. The visualizations are clear to illustrate this issue with a typical example.", "methods_and_evaluation_criteria": "Yes, the method is reasonable. The benchmark datasets are sufficient to support the claims.", "theoretical_claims": "No theoretical claim is provided.", "experimental_designs_or_analyses": "The experiments are sufficient to answer five questions provided by the authors in Section 4.1.", "relation_to_broader_scientific_literature": "This work is in terms of the LLM reasoning field. Related works include Chain-of-Thoughts (CoT), Retrieval-Augmented-Generation (RAG), GraphRAG, etc.", "essential_references_not_discussed": "The literature is sufficient.", "other_strengths_and_weaknesses": "Other Strengths:\n- The authors provide a comprehensive method for LLM reasoning that significantly improves the LLM reasoning quality when the knowledge graph is large and the information is sparse. This setting is widely existing in the industry.\n- The paper is well-written with clear visualizations.\n- The authors have conducted lots of experiments. Particularly, the authors test the performance of GIVE on knowledge graphs with scales from 135 entities to 844K, which validates GIVE’s capability of retrieving on both small and large knowledge graphs.\n\nOther Weaknesses:  \nOverall, I do not find severe problems in this paper, the following could be some `weaknesses’ that the authors or other reviewers might concern:\n- Compared to RAG that can retrieve any text information, graph-based retrievers (also for GraphRAT, etc.) naturally require an external knowledge base graph that is of high quality and is typically not yet learned by the LLMs. This might not be a `weakness’ but the common limitation of graph-based retrievers. \n- GIVE requires the preprocessing on the knowledge graph, the extraction of query information, and retrieval, which take some time. Some of the processes are before inference (like preprocessing), while others (extraction and retrieval) are during inference. \n- The method introduces some additional hyperparameters like m and n, increasing the difficulty of adjusting hyperparameters.", "other_comments_or_suggestions": "Typos:\n- line 86, 'focuses on uses’ -> 'focuses on using’.\n- line 192, ',The set’.", "questions_for_authors": "- Since I did not find the code link, I just want to ask the authors whether they would provide the corresponding repository to the public?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Sl8XfXHpC", "reviewer": "Reviewer_6XQ9", "summary": "The paper introduces Graph Inspired Veracity Extrapolation, a reasoning framework that enhances LLMs by integrating parametric and non-parametric memories for more accurate reasoning with minimal external input. GIVE operates through three key steps to select relevant expert data, engage in query-specific divergent thinking, and synthesize information for final outputs. Extensive experiments show that GIVE improves LLM performance across different model sizes, allowing smaller models to outperform larger ones in scientific tasks. GIVE supports reasoning with both restricted and noisy knowledge sources and underscores the value of combining internal and external knowledge to enhance LLM reasoning capabilities for complex scientific tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> In-depth analysis of token consumption\n\nWe understand the reviewer's concern about the token efficiency and provide a comparison of token consumption between GIVE and ToG on 100 random questions from each dataset. For each question, we calculate the total number of input/output token in the whole problem-solving process. We use tiktoken with GPT3.5-turbo for this comparison. For GIVE we use n=1, and for ToG, we use D=5, the setting is the same as Table 8 in the Appendix. \n\n|Avg no.input tokens        | PubmedQA | BioASQ | ProcessBank | CSQA/10\\% ConceptNet | CSQA/50\\% ConceptNet | CSQA/100\\% ConceptNet |\n|--------|----------|----------|----------|----------|----------|----------|\n| GIVE  | 14518.5     | 7970.3     | 19460.5    | 5321.1     | 7203.0    | 7398.7     |\n| ToG  | 12701.1    | 7010.0     | 11995.6     | 4934.8    | 6704.1    | 6679.7    |\n\n|Avg no.output tokens        | PubmedQA | BioASQ | ProcessBank | CSQA/10\\% ConceptNet | CSQA/50\\% ConceptNet | CSQA/100\\% ConceptNet |\n|--------|----------|----------|----------|----------|----------|----------|\n| GIVE  | 183.1     | 80.0     | 232.5     | 34.9     | 45.2     | 46.3     |\n| ToG  | 104.2    | 60.2     | 100.5     | 23.1     | 34.8     | 36.6    |\n\nThe observation supports the conclusion of Table 8, where GIVE\\_n=1 effectively balances efficiency and accuracy in challenging scientific reasoning tasks. Specifically, in 5 out of 6 datasets, GIVE consumes around 10\\% more input tokens than ToG, generates just 80 and 20 more tokens for PubmedQA and BioASQ, respectively, while reaching a 3-fold and 5-fold increase in accuracy. The variance in token usage across datasets is due to the differing numbers of entity groups and numbers of candidate relations, leading to varying candidate connections, as depicted in Figure 6 of the appendix. GIVE also demonstrates strong generalizability in deployment on KGs with varying sizes and sparsities.\n\n> Discussion of KAG and HOLMES\n\nThank you for pointing out the related works; we will add a discussion of KAG and HOLMES in our revised manuscript. Although we acknowledge that KAG and HOLMES leverage KG to enhance LLM output, we respectfully contend that they are solving fundamentally different problems with GIVE.\n\nKAG and HOLMES are advanced $\\textbf{retrieval}$ systems that retrieve and integrate accurate expert knowledge in question-answering tasks. In contrast, GIVE, as a $\\textbf{reasoning}$ framework, uses irrelevant information pieces to promote divergent thinking and extrapolates knowledge to generate answers. Thus, KAG and HOLMES concentrate on extracting quality KG from corpus or documents to aid precise information matching via a refined SemanticGraph or Graph Schema. They use KG for more accurate information matching. Here, the LLM iteratively synthesizes the responses as an answer generator. GIVE, however, guides LLMs in problem-solving like a human expert, foregoing preprocessing irrelevant knowledge by using the structured nature of KGs to seamlessly connect expert information with a query. KG data (entities and their connections) support the construction of a reasoning chain. LLM solves queries by verifying the inferred links suggested by KG, as illustrated in Figure 2. Although we were committed to including them as baselines, as suggested by the reviewer, HOLMES currently lacks an open-source implementation, and the public KAG code lacks the kag-model, rendering them unusable in our experimental settings with query and KG as inputs. We believe that these methods will perform similarly to GraphRAG, emphasizing information retrieval and injecting over reasoning.\n\n> Discussion of sensitivity to retrieved knowledge\n\nWe appreciate your insightful discussion. To clarify, the experiments in Figure 4 consistently show that GIVE effectively uses expert knowledge not directly related to the question to create an accurate reasoning process. This is evidenced by an improvement in performance from n = 0 to n = 1 and improved results with a higher ratio of expert knowledge. Furthermore, with the same expert knowledge ratio, BioASQ performs better as its questions are more easily connected to expert knowledge. The veracity extrapolation process links expert knowledge to the query by the extrapolated connections; a lower expert knowledge ratio means fewer such \"bridges\" for the model to leverage the unrelated KG information. We will incorporate this discussion into our revised manuscript.", "claims_and_evidence": "The claims made in the paper are supported by experimental evidence. The authors demonstrate through extensive experiments that GIVE improves LLM performance across various sizes and domains.", "methods_and_evaluation_criteria": "The proposed methods make sense for improving LLM reasoning in scenarios where internal knowledge is insufficient and external knowledge is limited or noisy. The evaluation criteria, including accuracy on various reasoning tasks, are appropriate for assessing the effectiveness of the proposed framework. The use of both scientific and open-domain datasets provides a comprehensive evaluation of the method's applicability.", "theoretical_claims": "No theoretical claims are provided.", "experimental_designs_or_analyses": "The proposed approach is only compared with naïve RAG and GraphRAG. However, many graph-based RAG methods have recently emerged, with their code publicly available. It would be beneficial to include comparisons with these methods, such as KAG and HOLMES, to provide a more comprehensive evaluation.", "relation_to_broader_scientific_literature": "The key contribution of this paper is: This work proposes a structured reasoning framework that integrates internal and external knowledge, along with a veracity extrapolation method that enriches limited information by establishing provisional connections between query concepts and incorporating counterfactual reasoning to mitigate hallucinations.", "essential_references_not_discussed": "1. HOLMES:Hyper-Relational Knowledge Graphs for Multi-hop Question Answering using LLMs\n2. KAG:Boosting LLMs in Professional Domains via Knowledge Augmented Generation", "other_strengths_and_weaknesses": "Weaknesses\nThe token usage and time consumption are significantly higher than other methods, yet the paper provides limited discussion on token efficiency, potential optimizations, and the impact on scalability and real-world applications.", "other_comments_or_suggestions": "1. The proposed approach is compared only with naïve RAG and GraphRAG. However, several recent graph-based RAG methods, with publicly available code, have been introduced. Including comparisons with methods like KAG and HOLMES would provide a more comprehensive evaluation of the approach.\n2. It seems that the BioASQ dataset is highly sensitive to the retrieved knowledge. A more detailed discussion and analysis of this sensitivity would be valuable.\n3. While time consumption is provided in the supplementary material, a more in-depth analysis of token consumption would be helpful.", "questions_for_authors": "please refer the suggestions and weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cw6Z3tzAt0", "reviewer": "Reviewer_aT9E", "summary": "The authors propose GIVE, an innovative framework designed to enhance the performance of large language models in scientific reasoning tasks. The framework consists of three main stages: expert data observation, divergent thinking, and information synthesizing. The large language model constructs a structured reasoning path by combining its internal parameterized knowledge with external non-parametric knowledge through a knowledge graph-inspired method called veracity extrapolation. This approach reduces hallucinations by utilizing counterfactual knowledge. It significantly improves the model’s accuracy and interpretability in biomedical, commonsense, and open-domain reasoning tasks, achieving an efficient balance in integrating large language models with knowledge sources that are either limited or noisy. As a result, it delivers superior reasoning outcomes.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We understand the reviewer's concerns about the role of expert knowledge in the reasoning process, as well as in the answer-generation process, and we appreciate your suggestion on our literature discussion and experiments. We hope the following clarifications addressed all the questions and we will make sure to include these discussions in our revised manuscript.\n\n> For Claims And Evidence paragraph 1:\n\nWe apologize for the confusion. To clarify, GIVE enriches the expert knowledge in the KG by extrapolating expert knowledge towards relevant concepts akin to the queried ones, as detailed in Section 3.4.3. Direct injection of expert knowledge into the question-answering is $\\textbf{not}$ assumed, due to the uncertain quality of the accessible knowledge base. Figure 4 shows that the performance of GIVE improves with an increased ratio of expert knowledge to total knowledge (expert plus extrapolated). Higher expert knowledge ratio ensures that the neighborhood related to the query is well-connected in the KG, offering substantial evidence for veracity extrapolation. Lower ratio indicates reliance on internal knowledge,  as discussed in Section 3.3 and Open Relations in Section 3.4.3. GIVE performs well with ample expert evidence guiding reasoning, quantitatively when expert knowledge ratio reaches about 20\\%, $\\textbf{not}$ when injecting solely 20\\% of this knowledge. All ablation studies in Section 4.7 utilize GPT3.5-turbo, with other details consistent with Section 4.2.\n\n> For Claims and Evidence Paragraph 2:\n\nCommonsense reasoning experiments aim to test GIVE's generalizability in three ways: (1) on noisy, large-scale KG, (2) varying sparsity levels, and (3) tasks where pre-trained knowledge is extensive. This setting presents challenges, as the basic LLM already achieves about 70\\% accuracy; pre-training on commonsense knowledge is inherently easier than on scientific information. In such cases, misinformation can lead to hallucination. GIVE improves performance by up to 3.4\\% and 4.9\\% over RAG and ToG, and consistently outperforms base LLM, indicating its reasoning process avoids hallucinations. ConceptNet is indeed important, the challenge is how to wisely use its information to further enhance the rich pre-training knowledge without causing hallucination, especially with its sparse versions. \n\nIncluding a 0\\% scenario, as the reviewer suggested, offers more insight: GIVE_a and GIVE_a+c achieve 69.84\\% and 69.36\\%, respectively, due to the exclusion of expert information. Without expert information, the knowledge provided by GIVE is derived solely from the LLM's internal knowledge (Sections 3.3 and Open Relations in 3.4.3). This aligns with our observation in Figure 4, where the expert knowledge ratio is 0. The better performance in this case is attributable to richer pre-training in commonsense compared to biomedical knowledge. \n\n> For Methods and Evaluation Criteria:\n\nWe understand the reviewer's concern about the role of expert knowledge. GIVE is designed for situations where a high-quality KG is inaccessible in domain-specific tasks. In such scenarios, GIVE does $\\textbf{not}$ intend to utilize expert knowledge directly for question answering but rather as an \"inspiration,\" illustrated by the transition from solid to dashed lines in Figure 2. The KG links some entities unrelated to the query; GIVE encourages the model to assess whether similar connections exist among other related entity pairs. In Tables 3 and 4, GIVE_a+c+e generally demonstrates improved or equivalent performance. The minor margin does not imply that expert information is trivial, but rather reflects that they are not directly solving the query. Expert knowledge (solid lines in Figure 2) is pivotal for the success of GIVE_a and GIVE_a+c, as it contributes to knowledge extrapolated from expert connections among relevant entities.\n\n> For Essential References Not Discussed:\n\nWe will include discussion of CoN in our revised manuscript. GIVE and CoN differ fundamentally in several aspects: (1) Motivation: GIVE is a reasoning framework that formulates a faithful thinking process by populating the expert KG triplets towards the query, whereas CoN is a robust retrieval system excluding similar yet irrelevant documents. (2) Use of internal knowledge: GIVE employs LLM's internal knowledge for \"veracity extrapolation,\" depicted by the transition from solid to dashed lines in Figure 2, whereas CoN uses it to create document summaries (notes) for accurate relevance analysis. (4) Use of external knowledge: All expert knowledge in GIVE is integral to its reasoning process, as shown by the solid lines in Figure 2. Meanwhile, CoN filters out irrelevant documents for question-answering. (5) GIVE is designed to reason $\\textbf{beyond}$ the accessible KG, whereas CoN identifies and focuses on documents containing the essential context.\n\n> For Other Comments Or Suggestions:\n\nWe will include all connections in the Figure for our revised manuscript.", "claims_and_evidence": "Firstly, the authors propose that the introduction of expert knowledge improves reasoning accuracy, as demonstrated in the experimental analysis in Section 4.7.3. However, there seems to be an issue with the experimental data. Figure 4 shows that as the expert knowledge ratio increases, the average accuracy gradually improves, eventually reaching 100%. On one hand, the authors do not specify which backbone models are used for the data shown in Figure 4, and the full experimental details are not provided. On the other hand, according to the results, an injection of around 20% expert knowledge already leads to an average accuracy of 100%, which seems contrary to common sense. It would be helpful for the authors to provide further clarification on this part.\n\nSecondly, the authors propose that GIVE can handle limited external knowledge bases, as demonstrated in Sections 4.4 and 4.5. However, in Table 4 of Section 4.5, there is no significant difference between GIVE, RAG, and ToG methods when providing 10%, 50%, and 100% scale knowledge graphs. It is possible that for commonsense reasoning tasks, the knowledge graph ConceptNet does not play a major role, and this argument is hard to substantiate. It is suggested that the authors provide experimental data for the 0% scenario for further analysis.", "methods_and_evaluation_criteria": "From the main experimental results in Table 3, it can be observed that in the biomedical dataset selected by the authors, overall, the performance of GIVE_a+c is better than that of GIVE_a+c+e. This indicates that the introduction of expert knowledge may not only be ineffective but could even lead to a decrease in performance. Meanwhile, results on CommonsenseQA shown in Table 4, the difference between GIVE_a+c and GIVE_a+c+e is within the range of 0.1 to 0.5. In the proposed framework, the introduction of expert knowledge is an important part, and it would be helpful if the authors could provide further clarification on this matter.", "theoretical_claims": "The paper doesn't present complex theoretical proofs but focuses on algorithmic contributions.", "experimental_designs_or_analyses": "Firstly, Figure 4 shows that as the expert knowledge ratio increases, the average accuracy gradually improves, eventually reaching 100%. On one hand, the authors do not specify which backbone models are used for the data presented in Figure 4, and the full experimental details are not provided. On the other hand, according to the results, an injection of around 20% expert knowledge already leads to an average accuracy of 100%, which seems contrary to common sense. It would be helpful for the authors to provide further clarification on this part.\n\nSecondly, in Table 4, regardless of whether it is GIVE, RAG, or ToG methods, there is no significant difference when providing 10%, 50%, and 100% scale knowledge graphs. It is possible that for commonsense reasoning tasks, the knowledge graph ConceptNet does not play a major role, and this argument is hard to substantiate. It is suggested that the authors provide experimental data for the 0% scenario for further analysis.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "The key contribution is overcoming the limitations of relying solely on internal or external knowledge and fully utilizing both external knowledge bases and the model’s existing knowledge. This has been preliminarily explored in the EMNLP 2024 paper \"Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models,\" which uses the model's existing knowledge to assess the relevance and validity of external knowledge, thereby improving answer accuracy. It is suggested that the authors include a discussion on models of this type of RALM.\n\n[1] Yu, Wenhao, et al. \"Chain-of-note: Enhancing robustness in retrieval-augmented language models.\" arXiv preprint arXiv:2311.09210 (2023).", "other_strengths_and_weaknesses": "Please refer to Questions For Authors", "other_comments_or_suggestions": "If the example provided in Figure 2 could demonstrate connecting all four sets of entities, it might be more helpful for readers to understand. Currently, it shows cross-group connections with two sets per group.", "questions_for_authors": "1.Discussion of RALM Models in the Context of Your Framework\n\nThe key contribution of your paper involves overcoming the limitations of relying solely on internal or external knowledge, fully utilizing both external knowledge bases and the model’s existing knowledge. This concept has been explored in the EMNLP 2024 paper \"Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models.\" Could you include a discussion on models of this type of RALM and how they relate to your framework?\n\n2.Role of Expert Knowledge in Cross-Group Connections\n\nThe paper suggests that \"The expert's cross-group connections serve as evidence, guiding the LLM to extrapolate the veracity of potential relationships among similar concepts.\" However, there seems to be insufficient explanation of the direct role of expert knowledge in establishing cross-group connections. Could you clarify how expert knowledge directly contributes to this process?\n\n3.Performance of GIVE_a+c vs. GIVE_a+c+e in Biomedical Dataset\n\nIn Table 3, the performance of GIVE_a+c is better than that of GIVE_a+c+e in the biomedical dataset. Can you provide an explanation as to why the introduction of expert knowledge may lead to a decrease in performance in this case?\n\n4.Expert Knowledge Injection and Accuracy\n\nThe results in Figure 4 show that injecting around 20% expert knowledge leads to an average accuracy of 100%. This seems counterintuitive and contrary to common sense. Could you provide further clarification on this, and explain why such a small amount of expert knowledge leads to such high accuracy?\n\n5.Clarification on Backbone Models and Experimental Details\n\nCould you specify which backbone models are used for the data presented in Figure 4 and provide full experimental details? The lack of such information raises concerns about the reliability and reproducibility of the results.\n\n6.Discrepancy Between GIVE, RAG, and ToG in Table 4\n\nIn Table 4, there seems to be no significant difference between GIVE, RAG, and ToG methods when providing 10%, 50%, and 100% scale knowledge graphs. Could you explain why the knowledge graph ConceptNet does not show a significant effect, especially for commonsense reasoning tasks? It would also be helpful if you could provide experimental data for the 0% scenario for further analysis.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The GIVE framework presented in this paper is designed to improve the reasoning capabilities of large language models (LLMs) by combining their internal knowledge with external knowledge graphs. This approach facilitates more accurate and efficient reasoning with minimal external input. The framework operates in three key stages: observation of expert data, divergent thinking, and synthesis of information to produce the final output. Extensive experiments have demonstrated the benefits of GIVE, including its ability to boost the performance of LLMs across various sizes, enable smaller models to outperform larger ones in scientific tasks, and maintain robustness and scalability in both scientific and open-domain assessments. GIVE is also highlighted for its ability to reason with both restricted and noisy knowledge sources, and its fully interpretable reasoning process.

While there are many discussions in the rebuttal period, the main opposing opinions including comparison with CoN, KAG and HOLMES, the analysis of token consumption, the role of expert knowledge in the reasoning process and the answer-generation process, etc.  The author addressed these questions with clarity and explanations.

The GIVE framework as presented in the paper has the potential impact to the fields by improving the reasoning capabilities of AI systems, making them more accurate, interpretable, and efficient, particularly in specialized domains where such capabilities are in high demand.
Decision: Accept (poster)