Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sources
Authors: Xingxuan Li
Ruochen Zhao
Yew Ken Chia
Bosheng Ding
Shafiq Joty
Soujanya Poria
Lidong Bing
Venue: ICLR 2024 poster (2024)
Keywords: large language model
knowledge grounding
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We present chain-of-knowledge, a novel framework that augments large language models dynamically by incorporating grounding information from heterogeneous sources.

Abstract: We present chain-of-knowledge (CoK), a novel framework that augments large language models (LLMs) by dynamically incorporating grounding information from heterogeneous sources. It results in more factual rationales and reduced hallucination in generation. 
Specifically, CoK consists of three stages: reasoning preparation, dynamic knowledge adapting, and answer consolidation. 
Given a knowledge-intensive question, CoK first prepares several preliminary rationales and answers while identifying the relevant knowledge domains.
If there is no majority consensus among the answers from samples, CoK corrects the rationales step by step by adapting knowledge from the identified domains.
These corrected rationales can plausibly serve as a better foundation for the final answer consolidation.
Unlike prior studies that primarily use unstructured data, CoK also leverages structured knowledge sources such as Wikidata and tables that provide more reliable factual information.
To access both unstructured and structured knowledge sources in the dynamic knowledge adapting stage, we propose an adaptive query generator that allows the generation of queries for various types of query languages, including SPARQL, SQL, and natural sentences. Moreover, to minimize error propagation between rationales, CoK corrects the rationales progressively using preceding corrected rationales to generate and correct subsequent rationales.
Extensive experiments show that CoK consistently improves the performance of LLMs on knowledge-intensive tasks across different domains.

Published as a conference paper at ICLR 2024
CHAIN-OF-KNOWLEDGEE GROUNDING LARGE LAN- GUAGE MODELS VIA DYNAMIC KNOWLEDGE ADAPT- ING OVER HETEROGENEOUS SOURCES
Xingxuan Lj12-1, Ruochen Zhao²=;, Yew Ken Chia Bosheng Ding Shafiq Joty24
Soujanya Poria', Lidong Bing!s
DDMOO Academy, Alibaba Group, Singapore, "Nayyang Technological University, "Singapore University of Technology and Design, "Salesforce Research, HHuaan Lab, 310023, Hangzhou, China
(xingxuan, 1i, yewken, chia, besheng.ding. 1.bing) @a ibaba-inss (rucchenC 02, srjoty]entu anstty)antu.edu..e edu.sg sporia@sutd. edu.sg
ABSTRACT
We present chain-of-knowledge (CoK). a novel framework that augments large language models (LLMs) by dynamically incorporating grounding information from heterogeneous sources. It results in more factual rationales and reduced hallucination in generation. Specifically, CoK consists of three stages: reason- ing preparation, dynamic knowledge adapting, and answer consolidation. Given a hewwledgeinntenivve question, CoK first prepares several preliminary rationales and answers while identifying the relevant knowledge domains. If there is no ma- jority consensus among the answers from samples, CoK corrects the rationales step by step by adapting knowledge from the identified domains. These corrected rationales can plausibly serve as a better foundation for the final answer consolida- tion. Unlike prior studies that primarily structured knowledge sources such as Wikidata and tables that provide more re- liable factual information. To access both unstructured and structured knowledge sources in the dynamic knowledge adapting stage, we propose an adaptive query generator that allows the generation of queries for various types of query lan- guages, including SPARQL, SQL, and natural sentences. Moreover, to minimize error propagation between rationales, CoK corrects the rationales progressively using preceding corrected rationales to generate and correct subsequent rationales. Extensive experiments show that CoK consistently improves the performance of LLMs on knowledge-intensive tasks across different domains. Our code is avail- able at http:://gthubbccom/AAAAAOPPSSSS -oai-----iiiiiiii---ooieeeeeeg -nowlldde
unstructured data, CoK also leverages
INTRODUCTION
OOpenAl 2023) have demonstrated impressive language generation capabilities (Cheng et 2023 Ding al. 2023 Chen al. (024). However, one major challenge of LLMss lies in hallucination, which is their tendency to confidently generate plausible but factually incorrect texts (Ji et al.) 2023 Zhao al.) 2023b). As shown in Figure given a question, "What year was the Argentine actor who directed El Tio Disparate born?" which requires factual knowledge to answer, the most advanced LLMs often provide an incorrect answer. While LLMs have the remarkable capability to recall information from their training data, effectively updating or controlling the factual knowledge within these models remains challenging (Luo al. 2023).
In recent years, large language models (LLMs) such as ChatGPT
Equal contribution. Xingxuan Li, Yew Ken Chia, and Bosheng Ding are under the Joint Ph.D. Program between DAMO Academy and their corresponding universities. Ruochen Zhao is under the AISG Ph.D. Fellowship Programme.

---

Published as a conference paper at ICLR 2024
Knowlldea with Dynamic Knowladge Adapting
Question
fidentifeed domainn:: tactual (Wikidata, Wikpedia)
What year was the Argentine actor who directed EI Tio Disparate bom?
Rationale First. the Argentine actar who
Disparate Femando Birni
Retrieve (Wikidata) 1: SELECT famswer WHERE (wddEE To Disparate wihldreecto Tonswer, | Palto Crissa
(a) Chain of Thought Celf-Consistency
Retrieve (Wikipedia)) 1: Who directed El Tio Disparate? >>
Rationales:
First, the Argentine actor who directed EI Dispanade is Farnando Birii
El Tio Disparata directed by Palno Oriaga.
Corrected rationale Oringa:
Argentina actor who directed Tio Disparate Palio
Second, Feraando Bim was born in 1925.
The answer 1925.
Rationale 2: Secund, Palloo Oriege was born in 1961)
Losa
Retrieve (Wikidata) 2: SELECT Tanswer WHERE (ddtPallo Ortega votidate birth
(b) Ratrieval-Based Methods (Verity-ande -aritt-and-Edit
Fan
Tanswer J 8 March 1941
mayrity
Retrieve (Wikipedia) 2: When was Palloo Ortaga bom? -o Palito Oriaga was born
Match 1941.
Verifying questions: Who diecced El Tie Disparale?
Corrected rationale
Oriaga
Match 1941.
When was Fernando Birni bom?
Retrieved Edit ence: Pallo Oriega directed EI Tie Disparata. famando Biriwas bom in 1925.
Corrected rationales: First the Argentine aisoor who directed Ti Dispsrate is Pallti Origaa Second, Pallo Orioga was born in Match 1941.
The answer 1925.
Eiror
empagatioo
The answer is 1941.
Figure 1: Comparison of different methods: (a) chain-of-though with self-consistency (Wei et al. or CoK (this work). 2022). (b) verify-and-edit (Zhao et al., (023c). and (c) and performs dynamic knowledge CoK incorporates heterogencous sources for knowledge retrieval
adapting. For clarity and succinct presentation, only pivotal steps are shown in the figure. Refer to Appendix A for the prompt design of each method.
A promising direction to address hallucination in generation is to augment the LLMs with external knowledge (Mialon et al. 2023). These methods involve incorporating LLMs with a retrieval sys- tem, which seeks to utilize external factual knowledge to guide the generation process. Instead of relying solely on the internal training knowledge of LLMs, these methods can fetch relevant infor- mation from external knowledge sources, such as web documents (Shi et al. 2023) and knowledge bases (Xie et al. 2022). Furthermore, to tackle more complex questions that require intricate rea- soning, Zhao al. (2023c) recently proposed
Eddff-and-EEdit (VE) framework, which improves chain-of-thought (CoT) reasoning (Wei et al. 2022) of LLMs by incorporating a retrieval system.
However, these methods have three inherent limitations. First, they use fixed knowledge source for all questions, which may fail to retrieve specialized and domain-specificc knowledge. For instance, it may not be effective to query Wikipedia for a medical question. Second. to generate retrieval queries, existing methods primarily rely on LLMs, which are predominantly pre-trained on natural language sentences, and thus may not be effective in generating structured queries like SPARQL, which is used to query knowledge graphs. Third, existing strieval-augmentt methods lack progressive correction capability, leading to potential error propagation. For example, in Figure I we define each rationale to be thought step (sentence) within the COT. Verify-and-Edit retrieves knowledge for each rationale in parallel and independently. Since the second rationale depends on the first, errors can carry over from the verification step to the edit step, making the retrieved knowledge misaligned with each other and the actual question, resulting in an incorrect final answer. Similarly, ReAct (Yao et al., 2023) also leaves errors from prior (reason act) steps in the prompt, causing potential noise and bias for LLM inference.
To address these limitations, we propose LLMs dynamically using heterogeneous knowledge sources. An example of how CoK functions is shown in Figure [ for the question, "What year was the Argentine actor who directed El Tio Dis- parate born?", COT with self-consistency (Wei et al.) 2022) is first utilized to generate preliminary rationales, pinpoint the relevant knowledge domains, and select answers that lack majority consen- sus for further processing. In the subsequent dynamic knowledge adapting stage, an adaptive query generator (AQG) is employed to generate queries for the knowledge sources within the selected do- mains. To effectively retrieve knowledge with heterogeneous formats, AQG can adaptively generate queries of the corresponding types, such as SPARQL and natural sentence (see Figure 2). Subse- quently, by executing the generated queries, supporting knowledge is obtained and utilized to edit the first rationale (i.e., rectify the director from Fernando Birri to Palito Ortega); it ensures mistakes do not propagate into the subsequent generation of the second rationale. The same process is then applicd to edit the second rationale. Finally, with the corrected chain of rationales, CoK derives the final answer
(CoK), framework that augments
2

---

Published as a conference paper at ICLR 2024
your Argentine scto
The anoowr is 1941
Reasoning Generation Knowlodgg Domain Seleciion
Arownr Consolidttion
Conrected Ratonale
Conracted Rationale
Correction
Goneraton
Goneration
Krowledge Reiravel
Supporting Krowiddee
Know Retrieool
Supporting
Knowindge
Rationale
Factual
(SPAROL.)
WMloedia (n.s.)
(SQL)
Lams-ALLaie
Query
(n.s.)
UpToCaae in.s.)
Query
Supoorting Knowkagee
Physizs
SclenceQAA Physics (n.a.)
PysicaCleesriom (n.s.
CheIGPT
Biolgy
SclenoOAA Biology (nx))
CK-12 (n.a.
Acaptive
Figure 2:
Our proposed chain-of-knowlldde (CoK) framework, consisting of (I) Reasoning prepa- ration, (II) Dynamic knowledge adapting, and (III) Answer consolidation. n.s.: natural sentence.
Given that different knowledge sources require distinct query languages, AQG holds a crucial role in generating queries. AQG is versatile and can either be fine-tuned model like Llama-2 (Touvron) al. 2023) with LoRA (Hu et al. 2021) or an off-the-shelf LLM like ChatGPT. By leveraging both unstructured and structured knowledge sources, CoK allows for better factual accuracy, improved reliability, and easier information updates.
To summarize, our key contributions the following: (1) We introduce chain-of-knowledg (CoK), novel framework to enhance the factual correctness of LLMs with heterogeneous knowledge sources; (2) We propose an adaptive query generator (AQG), specially designed to generate queries tailored to each knowledge source. AQG is versatile and can seamlessly transition between fine- tuned models and black-box LLMs; (3) CoK corrects the rationales progressively, ensuring that inaccuracies from preceding rationales do not propagate into the subsequent steps; (4) We perform extensive experiments on knowledge-intensive tasks spanning a range domains, including factual, medical, physical, and biological. CoK outperforms the CoT baseline by 4.3 % on average.
2 THE SEAIN-OF-KNOWLIEE
FRAMEWORK
As shown in Figure 2 the CoK framework consists of three stages: (1) reasoning preparation, (2) dynamic knowledge adapting, and (3) answer consolidation. the first stage, given a knowledge- intensive question. CoK generates preliminary rationales, i.e., reasoning units/sentences in the rea- soning chain of CoT, and answers while identifying the relevant knowledge domains. Questions that do not yield majority consensus in their answers enter the dynamic knowledge adapting stage, in which an adaptive query generator (AQG) is employed to generate queries to retrieve knowledge from the knowledge sources of the identified domain. The rationales are progressively revised and generated based on the retrieved knowledge. The final answer is then derived based on the corrected rationales. Refer to Appendix A.I for the prompts used for each step of our framework.
2.1 REASONING PREPARATION STAGE
In real-world scenarios, when facing a complex knowledge-intensive question, it is necessary to generate intermediate rationales before producing the final answer (Wei et al. 2022). Moreover, before delving into external knowledge sources to address the question, is crucial to identify the relevant knowledge domains for effective retrieval. Thus, the reasoning preparation stage consists of two essential components, namely, reasoning generation and knowledge domain selection.
3

---

Published as a conference paper at ICLR 2024
Table 1: An example of generated query, execution results, and formatted knowledge for rationales of SPARQL. Knowl. stands for knowledge.
SPARQL
Rationale Generated query
Souleyman Sané's son, Leroy Samé, is professional football player.
SELECT "answer WHERE { wwd/Souleyman ymune Sané/ wdt//child/ Panswer.
Execution results
Lenuy Sané
Formatted knowl.
The entity sentence "Souleyman Sand's son, Leroy Sané, professional football player" Leroy
Reasoning Generation Previous studies have demonstrated the importance of intermediate ratio- nales for LLMs to answer complex reasoning questions (Wei al. 2022). In this work, we utilize the few-shot chain-of- -oin---ftthhhhh (CoT) prompting to generate rationales (Wei et al., 2022). Moreover, we employ the self-consistency method (Wang et al. (023) to determine whether external knowledge is necessary to answer the question. In sampling various reasoning paths and answers, self-consistency found to be highly correlated with accuracy. Thus, predictions with high consistency are preserved without modification. Only questions with "uncertain" answers, i.e., their consistency falls below a specified threshold, undergo further stages of processing. Such filtering technique is found to be useful in identifying incorrect predictions by previous works Yao et al. 2023 Zhao et al. 2023c).
Knowledge Domain Selection To ensure the retrieval of the most pertinent knowledge to the question, we introduce the knowledge domain selection step. As shown in Figure [ CoK inte- grates four distinet knowledge domains: factual, medical, physics, and biology. Moreover, multiple domains can be identified for answering a single question. To illustrate, when presented with the question "Who proposed the theory which explains the cause of tides?", both physics (gravitational force the Moon causes tides) and factual (Isaac Newton first proposed the universal gravitation and explained tidal forces exerted by celestial bodies) domain knowledge are required to answer the question. The knowledge domain selection is completed through in-context learning.
2.2 DYNAMIC KNOWLEDGE ADAPTING STAGE
Once the preliminary rationales and the identified knowledge domains are obtained, the next stage is dynamic knowledge adapting. i.e., rectifying rationales based on the retrieved knowledge. To minimize error propagation, CoK conducts knowledge retrieval and correction of the rationales se- quentially. The preceding corrected rationales are used to generate the next rationale, which then undergoes the same knowledge retrieval and correction step.
Knowledge Retrieval Upon identifying relevant domains to the question in the reasoning prepa- ration stage, all knowledge sources within these domains are utilized for knowledge retrieval. The knowledge retrieval consists of two steps: query generation and execution.
A) Query Generation Depending on the nature of the knowledge sources, each source is linked to the most appropriate query language, which could either be structured, such as SPARQL or SQL, or unstructured, such as natural language sentences. For instance, Wikidata is linked to the SPARQL query as it consists knowledge graphs. The flashcard source linked to the natural sentence query as it takes the format of natural sentence pairs. An example of generated queries for SPARQL is shown in Table I For instance, given a sentence "Souleyman Sané's son, Leroy Sané, a pro- fessional football player", a SPARQL query, "SELECT ?answer WHERE {wd: Wnd::/Suueymarr Sané/ wdt: / child/ ?answer. }", is generated to retrieve relevant knowledge from Wiki- data. To facilitate the generation of both structured and unstructured queries,
adaptive query gen- crator (AQG) is used. AQG is a versatile plug-in component, which can be either a failor-finetuned model an off-the-shelf LLM. Details of AQG will be elaborated in Section 3
B) Query Execution Once the queries are generated, the subsequent step is their execution to acquire and convert the knowledge into formatted knowledge (see Table [ specialized method is devised to execute queries and format the results for each query language. For SPARQL queries, entity linking initially performed substitute entity spans with IDs, followed by acquiring results by invoking the API of wik k data. org. Regarding SQL queries, they are executed directly to
Examples of generated queries for each querying language are in Appendix D.5
4

---

Published as a conference paper at ICLR 2024
fetch the results, which could be singular value subset of the original table. The outcomes from both SPARQL and SQL are then formatted into markdown text. For natural sentence queries, knowledge is retrieved from domain-speciffe knowledge sources either through sentence similarity matching or by utilizing a search engine.
Rationale Correction Existing methods such ReAct (Yao et al. 2023 and Verify-and-Edi (Zhao et al. 2023c) keep all retrieved information the context throughout the process, no matter if contains reasoning mistakes. This often leads to errоr propagation and misguides further gener- ations. To overcome this weakness, CoK involves a progressive rationale correction step. Given the current rationale and the formatted knowledge from various knowledge sources, corrected ratio- nale is generated to replace the current one. This step helps in rectifying any factual incorrectness and preventing error propagation.
Next Rationale Generation Using the question and preceding corrected rationales, the next ratio- nale is generated, and the process is reiterated for the new rationale until final answer is produced.
2.3 ANSWER CONSOLIDATION STAGE
Ultimately, the LLM is prompted with the question and corrected rationales to generate a consol- idated answer, which is expected leading more accurate answer. This hypothesis is further examined through a series of experiments, as detailed in Section
3 THE ADAPTIVE QUERY GENERATOR
CoK incorporates heterogeneous knowledge sources from four different domains, including factual, medical, physics, and biology. Each of these knowledge sources necessitates the use of a unique query language for retrieval, which could be either structured or unstructured. Therefore, we design the adaptive query generator (AQG) to facilitate query generation for different knowledge sources.
Unstructured Query Languages Natural language sentences are the most natural way that hu- man beings search for information. AQG utilizes two distinct approaches for generating unstruc- tured queries based on the knowledge sources. A) For general factual knowledge sources, such as Wikipedia, ChatGPT is utilized. B) For domain-specifir knowledge sources (e.g., Flashcard, Sci- enceQA Physics, and ScienceQA Biology), using ChatGPT may lead to hallucination as it may not have comprehensive knowledge of the specific domains. Therefore, we instruction-tune LLaMA- 2-7B using LoRA with pairs of input texts and output queries. Furthermore, the domain of the training data is par with the respective knowledge source. Consequently, the AQG is equipped with the requisite knowledge for generating queries with greater precision.
Structured Query Languages Querying unstructured knowledge sources often leads to the re- trieval of irrelevant and redundant information. On the other hand, structured knowledge sources (e.g., Wikidata and tables) provide direct factual results. To generate structured queries, AQG uti- lizes two approaches based on the query languages. A) When generating commonly used query languages like SQL., employ ChatGPT. It is empirically inferred that ChatGPT included SQL during its pre-training, providing it with advantages in generating SQL queries (OpenAl 2023). All pertinent details are incorporated into the prompt to enhance the precision of query generation. For instance, when generating SQL queries, we include both the table schema and data snippets. B) For less common languages like SPARQL, we instruction-tun tune LLaMA-2-78 using LoRA with Santence-SPARQL pairs. The training data is collected to match the logical granularity of the ratio- nales, thereby facilitating more accurate query generation. For example in SPARQL, both training data and rationales contain single entity and relation within each sentence. Inspired by chain-of- hindsight (Liu et al.) 2023). besides giving the correct queries, we also append negative examples such "incorrect queries:.." during instruction-tuning
Detailed query language, model, and training datasets of each knowledge source are in Table of Appendix. The constructions of instruction-tuning datasets and training details are in Appendix We also evaluate the performances of AQG in Appendix F2
Details of the exccution process for each knowledge source is in Appendix
5

---

Published as a conference paper at ICLR 2024
Table 2: Main experimental results across various domains. Acc.: accuracy. E.M.: exact match.
Factual
Medical
Physics
Biology
Method
FEVER
HA E.M.
FeTaQQ BLEU
MedMCQA
MMLU Physics
MMLU Biology
Acc.
Acc.
Acc.
Acc.
Standard (3-shot)
51.8%
22.7%
20.7
61.6%
44.3%
80.6%
CoT (3-shot)
57.8%
29.9%
17.3
59.6%
41.9%
81.5%
CoT-SC (3-shot)
59.9%
30.8%
60.3%
42.7%
81.1%
VE (3-shot)
60.6%
31.8%
21.6
67.8%
39.9%
81.9%
CoK (3-shot)
63.4%
34.1%
25.0
70,5%
45.5%
83.0%
Standard (6-shut)
53.4%
24.0%
23.1
64,4%
44.7%
81.1%
CoT (6-shot)
55.6%
34.4%
19.4
66.4%
43.5%
81.7%
CoT-SC (6-shot)
56.2%
33.4%
65.8%
42.7%
82.2%
VE (6-shot)
57.2%
34.4%
23.1
67.1%
43.1%
78.9%
CoK (6-shot)
58.5%
35.4%
26.0
73.3%
47.0%
84.4 %
4
EXPERIMENTS
4.1 SETUP
Models In our experiments, we utilize ChatGPT 1000--35-tuubb-06111 as the black-box box LLM for the reasoning preparation and answer consolidation stages. To ensure reproducibility, fixed the decoding temperature to for all generations. Except for the self-consistency step, we set the temperature to 0.7, allowing for the sampling of five rationales and answers, recommended we edit the results with CoK.
by Wang et al. 2023). When less than half of the answers agree ?
Knowledge Sources We choose authoritative knowledge sources for each domain. Specifically, for the factual domain, we use Wikidata, Wikipedia, and Wikitables; for the medical domain, we use medical Flashcard and UpToDate; for physics, we refer ScienceQA Physics and PhysicsClass- room; and for biology, we utilize ScienceQA Biology and CK- 12. Details are in Appendix C
Tasks We collect set of mnowledge-intensiee tasks from various domains, including FEVER (Thorne et al., 2018), HotpotQA (Yang et al., 2018). and FeTaQA (Nan et al., (022] in the fac- tual domain; MedMCQA (Pal al. 2022) in the medical domain; Physics and Biology tests from
MMLU (Hendrycks et al. 2021 the physics and biology domains. Details in Appendix E
Baselines We compare CoK with both widely used baselines and state-of-the-art methods to pro- vide comprehensive overview: A) Standard prompting (Standard) directly predicts the swer (Ouyang et al. 2022). B) Chain-of-thought (CoT) (Wei et al. 2022) generates several inter- mediate rationales before the final answer improve the complex reasoning capability of LLMs. C) COT with self-consistency (CoT--C) (Wang et al.) 2023) replaces the naive greedy decoding in CoT with sampling diverse of rationales and outputs the most consistent answers. D)) Verify- and-Edit (VE) (Zhao et al. (023c) is oftt--of-the-art CoT-based framework that seeks to improve the prediction factuality by post-editing rationales with external knowledge. E) ReAct (Yao al.) 2023) combines agent thoughts and open-domain knowledge search to reach final answer. Fol- lowing the baselines, we evaluate using the few-shot setting and ensure that all methods use the same number of demonstration samples.
4.2 RESULTS
CoK Consistently Outperforms CoT As shown in Table CoK consistently outperforms CoT and CoT-SC each dataset. On factual-domain tasks, the average improvement 3-shot and
On FEVER, the output space is limited 3 answer choices, we always have high consistency. Thus, less than out of of the answers agrec.
'Note that self-consistency is not applicable for FeTaQA as it is an open-ended generation task, and we can
have near-equivalent generations that are nevertheless not exact matches. More details are in Appendix B report results for ReAct separately in Table as uses the PaI.M model Chowdheeyy et al. 2022
'Detailed prompts for baselines are in Appendix A Our work focuses on few-shot settings, thus not includ ing supervised methods baselines.

---

Published as a conference paper at ICLR 2024
Table 3: Results of retrieval-based methods FEVER and HotpotQA. ReAct results are adapted from Yao al. (023).
Table 4: Results of using single or multiple knowledge domains and sources MedM- CQA (3-shot).
Knowl. Method Domains
FEVER (3-shot)
HotpotQA (6-shot)
Knowl. Sources
Acc.
Method
Acc.
A Acc.
E.M.
AF.M.
CoT
59.6%
Rec-
64.6%
+4.2%
34.2%
+0.8%
CoK
Medical
Flashcard
67.1%
ReAct-CCoT-SC
62.0%
+1.6%
35.1%
+1.7%
CoK
Medical Flashcard, UpTo
69.2%
CuT-SC
59.9%
33,4%
Verify-and-Edth Edit
60.6%
+0.7%
34.4%
+1.0%
CoK
Medical,
Flashcard, UpToDate.
70.5%
CoK (ours)
63.4%
+3.5%
35.4%
+2.0%
Biology ScienceQA, CK-12
6-shot is prominent on HotpotQA and FEVER, registering at 2.6% and 4.3% respectively. This suggests that CoK not only effective on multi-step reasoning datasets (HotpotQA), but benefits less single-hop datasets (FEVER) as well with its accurate retrieval abilities. On domain datasets, such as MedMCQA, and MMLU Physics and Biology, CoK achieves an average accuracy improvement of 4.9% over the CoT baseline 3-shot and 6-shot settings. We notice that CoT has worse performances than standard prompting on FetaQA, MedMCQA, and MMLU Physics. This illustrates that, while CoT is effective for addressing complex reasoning tasks, it struggles with hallucination in its rationales when handling knowledge-inten intensive tasks, lcading to incorrect answers. This outcome aligns with the findings Yao al. (2023) and Zhao et al. (2023c) as well. With dynamic knowledge adapting, CoK can effectively reduce hallucination in the rationales and we include analysis on the factual accuracy in Section 5.3
CoK vs. Other Retrieval-based Methods As shown in Table 2 CoK consistently outperforms state-of-the-art retrieval-based method Verify-and-Edit (VE) (Zhao et al. 2023c). For FEVER and HotpotQA, we additionally compare with the results reported in ReAct Yao et al. 2023) in Table Since the results ReAct are reported on the PaLM model (Chowdhery et al., 2022). to add a more justified perspective, report the performance improvement gained top of the CoT-SC baseline. Compared with ReAct, CoK demonstrates a more substantial improvement over CoT-SC, especially on HotporQA. More specifically, for HotpotQA, CoK exhibits improvements of 2.0% compared to 0.8% by ReAct. On FEVER, CoK shows 3.5% improvement, which is on par with the 4.2% improvement gained by ReAct. This is attributed to the fact that FEVER is less multi-hop compared to HotporQA, thus benefitting less from an improved CОТ. VE conducts knowledge retrieval and editing for all rationales in parallel, and ReAct leaves past errors in the prompt, potentially leading to error propagation. CoK alleviates this issue with progressive knowledge adapting. It is also worth noting that CoK costs much less than ReAct, shown with detailed cost analysis Appendix [
Effect of Number of Demonstrations As shown in Table 2, CoK consistently exhibits enhanced performance across multiple datasets under both 3-shot and 6-shot settings. Sev- eral studies show that increasing the number of demonstra- tions (shots) in the prompt can potentially lead better per- formances on reasoning tasks (Wei et al. (022). However, this is not universally true for knowledge-intensse example, as shown in Table 2 the performance of CoT on MMLU Biology with six shots (81.7%) is nearly identical that with three shots (8 (8155%). This occurs because the bottle- questions
Nuttbal
tasks. For
neck for L.LMs in answering knowledge-intensse accurately is their insufficient knowledge, not their reasoning capability. The performance on FEVER for all reasoning- based methods decrease with six shots. This is likely due to the fact that FEVER questions are single-hop and require less reasoning. Thus, increased guidance reasoning could lead to potential noise. This finding is consistent with ReAct (Yao et al. 2023). where the authors state that increasing beyond 3-shot for FEVER does not lead better performance.
-0.0
Figure 3: A heatmap on distributions identified domains of each dataset.
7

---

Published as a conference paper at ICLR 2024
Table 5: Parallel dynamic knowledge adapting.
Table 6: Comparison of the factual accuracy of rationales on HotpoiQA.
Table 7: Human study results on the factuality of reasoning chains.
Method
HolpotQA (3-shoot)
CoT
29.9%.
Method
Rationale
Rationale
Predictions
Cok
CoT-SC
Tie
Verify-and-Edit
CoK (parallel)
31.8%
31.2%
CoT-SC CoK
54.3% 66.3%
52.1% 69.5%
Incomrett predictions
Coneet predictions
44%
68%
2416
4%
28%
32%
CoK (dynamic)
34.1%
All predictions
56%
14%
30%
ANALYSIS
5.1
SINGLE VS. MULTIPLE KNOWLEDGE DOMAINS AND SOURCES
CoK integrates a step to select the appropriate knowledge domains for each question. This step is crucial to ensure that CoK can retrieve the most pertinent knowledge to correct the rationales and answer the questions accurately. It is possible that multiple knowledge domains can be chosen for one question, and within each domain, there are several knowledge sources. In this subsection, we investigate the necessity of utilizing multiple knowledge domains and sources. We also include an evaluation of the domain selection performance in Appendix F.1
As outlined in Section 2.1
Single vs. Multiple Knowledge Domains We show the domain distributions identified for each dataset Figure 3 Notably, we find that CoK predominantly selects one knowledge domain for each dataset, while small number of cases call for multiple domains. For instance, the primary knowledge domain MedMCQA Medical, and 17.8% of the questions identify Biology as rel- evant domain as well. Furthermore, conduct ablation experiments to demonstrate the necessity of utilizing multiple domains. As shown in Table compared to only using Medical domain knowl- edge, CoK using additional knowledge from the Biology domain further improves the performance by 1.3%. This indicates that knowledge spanning multiple domains needed for answering some questions, underscoring the necessity of incorporating various knowledge domains.
Single vs. Multiple Knowledge Sources Within one domain, numerous credible knowledge sources exist, and is unfeasible for single source to encompass all knowledge from the domain. Therefore, it is important to utilize multiple knowledge sources within one domain. For instance, as shown in Table 4 the performance of CoK improves by 2.1% when utilizing both Flashcard and Upt as medical knowledge sources, compared to using only Flashcard.
5.2 PARALLEL VS. DYNAMIC KNOWLEDGE ADAPTING
As aforementioned, dynamic knowledge adapting helps CoK prevent error propagation, here we take closer look at how much improvement it brings in. As shown in Table the performance of CoK improves by 4.2% compared with CoT when dynamic knowledge adapting is applied. However, parallel editing leads poorer performance due
propagation for rationales.
5.3 EVALUATING FACTUALITY IMPROVEMENT OF THE RATIONALES
While the main results have shown that CoK effectively improves the performance of LLMs in knowledge-intensive tasks, we are also interested reducing hallucination for the generated ratio- nales. Hence, we conduct quantitative and qualitative evaluations to assess the factual accuracy.
Quantitative Evaluation To automatically evaluate how CoK can reduce hallucination in the model outputs, we employ an existing fact-checking method to compare the original and corrected rationales. Specifically, we use ProgramFC (Pan et al. 2023) which is state-of-the-an method for judging the factuality of claims with respect to Wikipedia. As shown in Table 6 we observe that CoK has improved factual accuracy compared to the CoT-SC baseline on the HotpotQA dataset. No- tably, the factual accuracy of CoT-SC decreases for rationale 2 compared rationale 1, which could
Note that using external sources may have limitations such noise conflicts between different sources. We mainly address this selecting authoritative knowledge sources, and discuss this further in Appendix G
8

---

Published as a conference paper at ICLR 2024
be due to error propagation. On the other hand, the factual accuracy of CoK improves slightly for the second rationale, which indicates that correcting previous rationales helps the LLM to generate more factual rationales in future steps.
Human Evaluation To qualitatively examine whether CoK could output factually consistent rea- soning chains, we also conducted a human study. Specifically, two volunteers are given 100 outputs randomly selected from HotpotQA and FEVER datasets. The selected outputs are balanced, where 50 CoK outputs resulted in incorrect answers, and the other 50 resulted in correct answers. The vol- unteers are asked to select which reasoning chain is factually correct, or if there is tie. Then, they are asked to answer whether the better CoT should lead to better results. Details on the instructions and setup can be found in Appendix H.I The results are given in Table 7 We could observe that volunteers consistently confirm that CoK-generater reasoning chains are factually consistent while the CoT-SC chains are not. For incorrect predictions, humans still believe that 44% of the time, the CoK-generated CoT is improved on factual consistency, although it may not contain the necessary information for correct answer. Among these instances, humans believe 73% of the time that these improved CoTs should have led to better answers. This implies that, even though the CoT quality has been improved, many failure cases are caused by reasoning errors. Case studies can be found in Appendix H.2] In general, the two volunteers show Cohen Kappa's agreement of 0.43, which is considered moderate agreement (Landis & Koch 1977).
6 RELATED WORK
Phowledge-Intensing NLP While language models can generate highly coherent text and demon- strate reasoning abilities, many real-world tasks require knowledge beyond the local context. For example, fact-checking tasks may require models to locate suitable evidence on the internet or refer to external knowledge (Thorne et al., 2018). In the realm of natural language processing (NIPP), a task is deemed to be knowledge-intensing when it exceeds the reasonable expectation of human capa- bility to solve it without access to external knowledge. The resolution of such knowledge-intents NLP tasks typically involves the utilization of retriever-reader systems. Initially, retriever extracts a limited collection of pertinent documents from the knowledge source, after which reader employs the context extracted to generate an appropriate response (Chen al., [017 Lewis al. 2020 Guu
et al.
2020). Hence, there is an urgent need to develop effective models for knowledge-intensive
tasks
Petroni et al. (02T).
Augmented Language Models The discipline of augmented language models (ALMs) addresses hallucinations of traditional LLMs by equipping them with improved reasoning capabilities and the capacity utilize external resources (Chung al.) 2022). Furthermore, LLMs can learn to leverage external tools or models to accomplish the relevant tasks (Schick et al., 2023 Shen et al.) 2.23). ALMs can employ these enhancements independently or combine them in a specific order to complete a given task, ultimately resulting in enhanced capabilities (Mialon et al. 2023 Zhao et al. 2023a). However, previous works do not consider knowledge from multiple domains and lack progressive editing throughout the generation process, which could lead to error propagation. In this work, we propose an efficient framework to solve knowledge-intent intensive tasks by progressively augmenting them with diverse sources of external knowledge.
CONCLUSIONS
In this paper, we introduce factual correctness of large language models (LLMs), CoK represents a promising and comprehen- sive solution to progressive in multiple domains. We address the challenge of accurate query generation by proposing the adap- tive query generator (AQG) which supports both unstructured and structured query languages. The AQG can be easily transitioned between fine-tuned models and black-box LLMs. Our experimental results on knowledge-intensive intensive tasks demonstrate the substantial improvement achieved by CoK. Furthermore, the modularity of CoK allows its application to various LLMs and different formats of knowledge sources, which addresses important challenges, including privacy concerns, knowledge source reliance, and rapid information updates.
(CoK), a novel framework designed to enhance the
wewlleegggruudeee generation by incorporating heterogeneous sources

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was substantially supported by DAMO Academy through DAMO Academy Research Intern Program. This research/project is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: ALSC-PhDD-2021-01-000) Soujanya Poria is supported by the grant AISG3-GV-2023-010.
REFERENCES
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity retrieval. In Proceedings of ICLR, 2021.
Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Hanwang Zhang. KQA pro: dataset with explicit compositional programs for complex question answering over knowledge base. In Proceedings of ACL, 2022.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open- domain questions. Proceedings of ACI., 2017.
Hailin Chen, Fangkai Jiao, Xingxuan Li, Chengwei Qin, Mathieu Ravaut, Ruochen Zhao, Caiming Xiong, and Shafiq Joty. Chatgpt's yne-year anniversary: Are open-source large language models catching up? arXiv preprint arXiv:2311.16996 2024.
Liying Cheng, Xingxuan Li, and Lidong Bing. Is gpt-4 a good data analyst? arXiv preprint 10X.5:2305..1038, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204. 02311, 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling cntructiionfinetunne language mod- els. arXiv preprint arXiv:2210. 1416, 2022.
Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing, Shafiq Joty, and Boyang Li. gpt-3 good data annotator? In Proceedings of 2023.
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. Can machine translation sys- tems be evaluated by the crowd alone. Natural Language Engineering, 2017.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre- pre-training. In Proceedings of ICML, 2020.
Tianyu Han, Lisa C. Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K. Bressem. Medalpaca an open-source collection of medical conversational ai models and training data. arXiv preprini arXiv:2304.08247, 2023.
Dan Hendrycks, Collin Burs,, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proceedings of ICLR, 2021.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1rXivv2206.09960 2021.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii. Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination natural language generation. ACM Computing Surveys, 2023.
Tom Koemi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák. Martin Popel, and Maja Popović. Findings the 2022 conference on machine translation (WMT22). In Proceedings of WMT, 2022.
10

---

Published as a conference paper at ICLR 2024
J Richard Landis and Gary G Koch. The measurement of observer agreement for categorical data. biometrics, 1977.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. aeuientl- generation for knowledge-intenee NIPS, 2020.
nlp tasks. In Proceedings of
Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Chain of hindsight aligns language models with feedback. arXiv preprint arXiv:2302.0222 02676, 2023.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. In Proceedings of NIPS, 2022.
Ziyang Luo, Can Xu, Pu Zhao, Xiubo Geng, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Augmented large language models with parametric knowledge guiding. arXiv preprint 2rX4v:2305.04757, 2023.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yos Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: a survey, arXiv preprint MXXv:22302.0782.. 2023.
Linyong Nan, Chiachun Hsich, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojeiech Kryšcińíski, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Is- abel Trindade. Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev, and Dragomir Radev. FeTaQQ: Free-form table question answering. Transactions of the Association for Computational Linguistics, 2022.
OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.0211 2022.
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa : A large- scale multi-subject multi-choice dataset for medical domain question answering. arXiv preprint arXiv:2203. 14371, 2022.
Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan Luu, William Yang Wang, Min- Yen Kan, and Preslav Nakov. Fact-checking complex claims with program guidedd reasoning. In Proceedings of ACL, 2023.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thome,, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of NAACI., 2021.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.0476.1 2023.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint
MxXi:2303.17580 2023.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen tau Yih. Replug: Reugeeed box language models, 2023.
James Thome, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large- scale dataset for fact extraction and VERification. In Proceedings NAACL, 2018.
11

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:230.009288 2023.
Priyansh Trivedi, Gaurav Maheshwari, Mohnish Dubey, and Jens Lehmann. Le-quad: A corpus for complex question answering over knowledge graphs. In Proceedings of ISWC, 2017.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, seff-consistency improves chain of thought reasoning in language models. In Proceedings of ICLR, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thoughe prompting elicits reasoning in large language models. In Proceedings of NIPS, 2022.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien- Sheng Wu, Ming Zhong, Pengeheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. UnifiedSKG: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. In Proceedings of EMNLP, 2022.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of EMNLP, 2018.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In Proceedings of ICLR, 2023.
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, et al. Retrieving multimodal information for aug- mented generation: survey, arXiv preprint arXiv:2303. 10868, 2023a.
Ruochen Zhao, Xingxuan Li, Yew Ken Chia, Bosheng Ding, and Lidong Bing. Can chatgpt- like generative models guarantee factual accuracy? on the mistakes of new generation search engines. arXiv preprint arXiv:2304. 100004/10066. 1076, 2023b.
Ruochen Zhao, Xingxuan Li, Shafiq Joty. Chengwei Qin, and Lidong Bing. Verify-and-edit: A knowledge-enhanced chain-of-thouge framework. In Proceedings of ACL, 2023c.
12

---

Published as a conference paper at ICLR 2024
A PROMPTS USED IN DIFFERENT METHODS
A.1 CHAIN-OF-KKOWLEEDGH (HoTPOTQA)
A.1.1 REASONING GENERATION
Q: This British racing driver came in third at the 2014 Bahrain GP2 Series round and was born in what year
A: First, at the 2014 Bahrain GP2 Series round, DAMS driver Jolyon Palmer came in third. Second, Jolyon Palmer (born 20 January 1991) is British racing driver. The answer is 1991.
Q: [uestion]
A:
A. 1.2
KNOWLEDGE DOMAIN SELECTION
Follow the below example, select relevant knowledge domains from Available Domains the Q- Available Domains: factual, medical, physical, biology
Q: This British racing driver came in third at the 2014 Bahrain GP2 Series round and born in what year
Relevant domains: factual
Q: Which of the following drugs can be given in renal failure safely?
Relevant domains: medical
Q: Which object has the most thermal energy?
Relevant domains: factual, physical
Q: Is the following trait inherited acquired? Barry has scar on his left ankle.
Relevant domains: biology
Q: [Question]
Relevant domains:
A.1.3 RATIONALE CORRECTION
Strictly follow the format of the below examples. The given sentence may have factual errors, please correct them based on the given external knowledge. Sentence: the Baaher-Behhe-aameee Knowledge: discoverer or inventor of Alpher-Bettee Bethe fethe-Famow paper is Ralph Alpher. Edited sentence: the Bacher-Bethe-Gameeee paper was invented by Ralph Alpher.
paper was invented by Ralph Alpher.
Sentence: Ralph Alpher advised by Hans Bethe.
Knowledge: doctoral advisor of Ralph Alpher is George Gamow. Edited sentence: Ralph Alpher was advised by George Gamow.
Sentence:
[Ratioante]
Knowledge: [Knowledge]
Edited sentence:
A.1.4 NEXT RATIONALE GENERATION
Q: This British racing driver came in third at the 2014 Bahrain GP2 Series round and was born in what year
A: First, at the 2014 Bahrain GP2 Series round, DAMS driver Jolyon Palmer came in third. Second, Jolyon Palmer (bomn 20 January 1991) is a British racing driver. The answer is 1991.
Q: [Question]
A: First, [Corrected first rationale]. Second,
13

---

Published as a conference paper at ICLR 2024
A.1.5 ANSWER CONSOLIDATION
Q: This British racing driver came in third at the 2014 Bahrain GP2 Series round and born in what year
A: First, the 2014 Bahrain GP2 Series round, DAMS driver Jolyon Palmer came in third. Second, Jolyon Palmer (born 20 January 1991) is a British racing driver. The answer is 1991.
Q: [Question]
A: First, [Corrected first rationale]. Second, [Corrected second rationale]. The answer is
A.2 COT, CoT-SC
Q: This British racing driver came in third at the 2014 Bahrain GP2 Series round and was born in what year
A: First, at the 2014 Bahrain GP2 Series round, DAMS driver Jolyon Palmer came in third. Second, Jolyon Palmer (born 20 January 1991) is British racing driver. The answer is 1991.
Q: [Question]
A:
A.3 VERIFY-AND-EDIT
A.3.1 VERIFYING QUESTION GENERATION
Write question that asks about the answer to the overall question. Overall Question: The Sentinelese language the language of people of one of which Islands in the Bay of Bengal?
Answer: The language of the people of North Sentinel Island is Sentinelese.
Question: What peopleś language is Sentinelese?
Overall Question: [Question]
Answer: [Rationale]
Question:
A.3.2 VERIFYING ANSWER GENERATION (RATIONALE EDITING)
Barnes House (born 20 January 1969) a British racing driver, currently driving for Renault Sport Fl Team in the Formula One World Championship.
Jolyon Palmer (born 20 January 1991) is British racing driver, currently driving for Renault Sport F1 Team in the Formula One World Championship.
Ming Xi (born 20 January 2015) is British racing driver, currently driving for Renault Sport Fl Team the Formula One World Championship.
The 2014 Bahrain GP2 Series round was a pair of motor races held on and 7 April 2014 at the Bahrain International Circuit in Sakhir, Bahrain second for the Carlin team and DAMS driver Jolyon Palmer came in third.
part of the GP2 Series. Julián Leal finished
Q: This British racing driver came in third at the 2014 Babrain GP2 Series round and was born in what year
A: This British racing driver came third at the 2014 Bahrain GP2 Series round and was born in 1991.
Knowledge
Q: [Verifying question]
A:
14

---

Published as a conference paper at ICLR 2024
B FURTHER EXPERIMENT DETAILS
B.I FETAQA
Although CoK and several baseline methods including CoT-SC and VE rely self-consistency for other tasks, we note that consisteeny is not applicable for FeTaQQ as it is an open-ended gen- eration task. As result, it is possible have near-equivalent generations that are nevertheless exact matches, and self-consistency becomes less useful. Therefore, we do not use self-consistency for VE and CoK, instead opting retrieve from external knowledge sources for every question in FeTaQA. We also do not include CoT-SC results for FeTaQQ in Table 2
C QUERY EXECUTION OF KNOWLEDGE SOURCES
C.I WIKIDATA (SPARQL)
As shown in Table 1 the SPAROL query generated by AQG contains entity and relation spans. To make the query executable, we conduct entity linking, replacing the spans with entity and relation IDs. We utilize the GENRE model for entity linking (Cao al. 202T). GENRE is the first system that retrieves entities by generating their unique names in an autoregressive fashion. Consequently, GENRE capable of performing entity linking without ambiguities. Next, the query is executed on Wikidata to retrieve the results. Finally, we transform the reasoning step and the results into natural sentence format, which serves as the final supporting knowledge.
C.2 WIKIPEDIA (NATURAL SENTENCE)
We directly query generated natural language sentence within the domain wi k kipedia org.
C.3
TABLE (SQL)
Given a generated SQL query, we execute query on the given table to obtain the result, which may be single value sub-selection of the table. Thereafter, consolidate the query result with the original question which is provided to the LLM for generating the final answer. As the query may be inaccurate in cases, also provide the original table to the LLM when generating the final answer.
C.4 FLASHCARD (NATURAL SENTENCE)
Given medical reasoning step, AQG generates sentence of relevant medical knowledge as the query. Subsequently, we compare the embeddings of this query with sentences from the Medical Flashcards knowledge base and select the sentence with the highest cosine similarity as the final supporting knowledge. Hence, this ensures that the supporting knowledge is factually correct.
C.5 UPTODATE (NATURAL SENTENCE)
We directly query generated natural language sentence within the domain uptodate, com, which is an authoritative medical website.
C.6 SCIENCEQA PHYSICS (NATURAL SENTENCE)
Given a physics reasoning step, AQG generates sentence of relevant physics knowledge as query. Subsequently, we compare the embeddings of this query with sentences from the ScienceQA Physics knowledge source and select the sentence with the highest cosine similarity as the final supporting knowledge. Hence, this ensures that the supporting knowledge is factually correct.
C.7 PHYSICSCLASSROOM (NATURAL SENTENCE
We directly query generated natural language sentence within the domain
physicsclassrooae com, which is authoritative physics website.
15

---

Published as a conference paper at ICLR 2024
Table 8: Query language, AQG model, and training datasets of each knowledge source. Knowl. stands for knowledge. Lang. stands for language.
Knowl. Domain
Knowl. Souree
Query Lang.
AQG Model
Dataset Source
Train. Set
Eval. Set
Factual
Wikidata
SPARQL
LLaMA-2-11 2--4----8--
LoRA L-R-L-RAA
LC-quad & KQA-pro
19,010
4,779
Wikinedia
gpt-3.5
Factual
Table
SQL
3pt-3.5-turbo-0613
Medical
Flashcard
D.s.
LLMMA-27BB-LRR-
Medical Flashcard
34,000
Medical
UpToDate
spt-3,5-turb-00133
Physies
ScienceQAA Physics
LaMA-2-7B-LoKA
SetenceQA Physics
810
Physies
Physicsclastroom
n.5.
gpt-3.5-turbo-0611
Biology
ScienceQAA Biology
LLMMA-2-7B-LoRA
ScienceQA Physics
1,5%6
Biology
CK-12
D.S.
gpt-3.5-turbo-0113
C.8
SCIENCEQA BIOLOGY (NATURAL SENTENCE)
Given a biology reasoning step, AQG generates sentence of relevant biology knowledge as the query. Subsequently, we compare the embeddings of this query with sentences from the ScienceQA Biology knowledge source and select the sentence with the highest cosine similarity as the final supporting knowledge. Hence, this ensures that the supporting knowledge is factually correct.
C.9 CK-12 (NATURAL SENTENCE)
We directly query ok12.org// htt.ooogg////////..ooooooooo//// which is an authoritative biology website.
generated natural language sentence within the domain
D
ADAPTIVE QUERY GENERATOR
D.1 WIKIDATA (SPARQL)
D.1.1 INSTRUCTION-TUNING DATASET
To create the instruction-tuning dataset, we utilize filtered version of LCC-qud (Trivedi al. 2017) and KQA-pro (Cao et al. 2022) datasets. This dataset consists of natural questions inputs and their corresponding SPARQL queries as outputs. Before training, we replace the entity and relation IDs in the SPARQL queries with entity and relation spans. This modification allows the model to learn the semantic meaning of the SPARQL queries more effectively. During the inference phase, utilize entity linking to convert the spans back into IDs. The size of the dataset is provided in Table 8
D.1.2 DATA EXAMPLE
Below is instruction that describes task, paired with an input that provides further context. Write a response that appropriately completes the request.
### Instruction:
Generate correct SPARQL query that returns the answer of the following question. Generate four incorrect SPARQL queries of different types.
### Input:
What periodical literature does Delta Air Lines use as a mouthpiece?
### Output:
Correct query: select distinct ?obj where wd:/Delta Air Lines/ wdt:/house publication/ ?obj ?obj
wdt:/instance of/ wd:/periodical// }
Incorrect query 1: select distinct ?obj where { Delta Air Lines wdt:/hosee publication/ ?obj ?obj
wdt:/instance of/ wd:/periodical/ }
Incorrect query 2: select distinct ?obj { wd:/Delta Air Lines/ wdt:/ouse publication/ ?obj ?obj
wdt:/instance of/ wd:/magazine//
Incorrect query 3: select distinct ?obj where { wd:/Delta Airlines/ wdt:/house publication/ ?obj
?obj wdt//nssaacee of/ wd:/periodical//
16

---

Published as a conference paper at ICLR 2024
Incorrect query 4: select distinct ?obj where (obb wdt:/instance of/ wd./magazine/, wd:/Delta Air Lines/ wdt//hose publication/ ?obj.
D.2 FLASHCARDS (NATURAL SENTENCE
D.2.1 INSTRUCTION-TUNNG DATASET
We employ natural sentence format for querying Medical knowledge. To instruction-tune AQG specifically for this purpose, we utilize the Medical Flashcards dataset (Han et al. (003). This dataset consists of question-answerte as anatomy, physiology, pathology, and pharmacology. It contains summaries and mnemonics of crucial medical concepts, making ideal choice for instruction-tuning the AQG effectively handle medical knowledge queries. The size of the dataset provided in Table
pairs covering various subjects in the medical source, such
D.2.2 DATA EXAMPLE
Answer the question truthfully.
### Instruction:
Answer this question truthfully.
### Input:
What is the relationship between very low Mg2+ levels, PTH levels, and Ca2+ levels?
### Output:
Very low Mg2+ levels correspond to low PTH levels which in turn results in low Ca2+ levels.
D.3 SCIENCEQA PHYSICS (NATURAL SENTENCE)
D.3.1 INSTRUCTION-TUNING DATASET
To instruction-tune our AQG for physics knowledge, we utilize the physics segment the Sci- enceQA dataset (Lu et al. 2022 Each entry in this dataset consists of question, options, context, answer, lecture, and explanation. The lecture contains necessary knowledge to answer the question. We the question and the options as input and the lecture the output for instruction-tuning the model.
D.3.2 DATA EXAMPLE
Answer the question truthfully.
### Instruction:
Answer this question truthfully.
### Input:
The objects are identical except for their temperatures. Which object has less thermal energy?
Choose from: 300-gram glass of water at a temperature of 75 a 300-gram glass of water at temperature of F
### Output: The two glasses of water have the same mass but different temperatures. Since the 75 glass of water is colder than the 8 glass of water, it has less thermal energy.
D.4 SCIENCEQA BIOLOGY (NATURAL SENTENCE)
D.4.1 INSTRUCTION-TUNING DATASET
To instruction-tune our AQG for biology knowledge, we utilize the biology segment of the Sci- enceQA dataset (Lu et al. 2022
D.4.2 DATA EXAMPLE
Answer the question truthfully. ### Instruction:
Answer this question truthfully.
17

---

Published as a conference paper at ICLR 2024
Table 9: Examples of generated query, execution results, and formatted knowledge for rationales of each query language. Knowl. stands for knowledge.
SPARQL
Rationale
Souleyman Sané's 's son, Leruy Samé, is prolessional football player.
Generated query
SELECT WHERE wwd/Souleymm Sané/ wdtt/child/ Panswer.
Execution results
Lesuy Sané entiyy sentence "Souleyman son, Leroy Sané, professional football player" Leroy
Formatted knowl.
SQL
Rationale Generated query
Does Oklahoma any indoor football teams?
SELECT FROM table WHERE Type 'Indoor Football';
Execution results
Club
Type
Venue
Oklahoms Defendery
Indoor
Tulsa Convention Center
knowl.
['Clubb,, 'Type', "Venue') J, ['Oklahoma Defenders', TIddoo Football', Convention Center') ]]
Natural Sentence
Rationale
Splenomegaly a condition in which spleen enlarged.
Generated
What conditions may feature splenomegaly? Normocytic
Execution results
extravascular bemolysis is associated enlargement spleen. Normocytic anemia with extravascular hemolyyis is associated with enlargerment of the spleen (splenomegaly). circulation.
Formatted knowl.
spleen plays a in removing damaged
### Input:
In a group of sheep, some individuals have white wool and others have black wool. In this group, the gene for the wool color trait has two alleles. The allele L is for white wool, and the allele is for black wool. Flicka, sheep from this group, has white wool. Flicka has one allele for white wool and one allele for black wool. Based on this information, what is Flicka's phenotype for the wool color trait? Choose from: LI, white wool.
### Output:
An organism's phenotype for a trait its observable version of that trait. Flicka's 's observable version of the wool color trait is white wool. So, Flicka's 's phenotype for the wool color trait is white wool.
D.5 EXAMPLES OF EACH QUERYING LANGUAGE
We include examples of generated query, execution results, and formatted knowledge for rationales of each query language in Table
D.6 CONTRASTIVE INSTRUCTION-TUNING
We implement simple approach to train the model for SPARQL with a contrastive objective, where the correct query and wrong queries modeled autoregressively in the same sequence. Concretely, given a sequence z which includes the input tokens, correct query tokens and wrong query tokens, the query model is trained with the log-likelihood loss:
log p(x) = log II
1(x (())p())))))
(I)
1=1
where 1(x;) 1 if the i-th token 2, is part of query and 0 otherwise.
D.7 TRAINING DETAILS
2-7b-hf) the base model. We utilize LoRA for parameter-efficien fine-tuning, and load the weights in 8-bit format, For each knowledge source, the model is trained for epochs, utilizing NVIDIA A40 GPU. We maintain training batch size of 32, with a gradient accumulation step set at 2. All the other parameters are left at their default values.
We employ Llama-2 (meta catta-lllma//iin
18

---

Published as a conference paper at ICLR 2024
Table 10: Details of the evaluation datasets.
Domain
Dataset
of Samples
Factual
FEVER
1000
Factual
HotpoiQA
308
Factual
FeTaQQ
500
Medical
MedMCQA
146
Physics MMLU Physics
253
Biology
MMLU Biology
454
Table 11: Evaluation of domain selection performance.
Domain Precision
Recall
F1
Factual
96.0%
96.0%
96.0%
Medical
94.3%
96. 1
95.2%
Physics 89.9%
100.0%
94.6%
Biology
100.0%
92.8%
96.2%
E EVALUATION DATASETS
The evaluation datasets collect datasets from four different domains, including factaul, medical, physics, and biology. Details of the dataset are in Table IO We adopt exact match as the evaluation metrics for HotpotQA, which is more strict evaluation.
F ANALYSIS
F.I DOMAIN SELECTION
As CoK relies on selecting relevant knowledge domains, it is important that the domain selection step is of high quality. Hence, we randomly sample 50 questions from each domain and compare the predicted domains with our manually annotated domains. As each question may be relevant for more than one domain, we report the precision, recall, and F1 scores. shown in Table TT. we find that while the domain selection is not perfect, the overall Fl scores more than 94% across all the domains. Hence, we believe that the current domain selection process is adequate.
F.2 MODELS OF ADAPTIVE QUERY GENERATOR
Table T2 demonstrates the performances of ChatGPT and instruction-tuned LlaMA-2-7B SQL and SPARQL generation. SPARQL is evaluated on 4,779 samples from LC-quad and KQA-pro. SQL evaluated on 15,900 samples from WikiSQL and we use the exact-match metric evaluate the generated queries with gold queries.
G DISCUSSION OF LIMITATIONS
Knowledge Sources As CoK relies on external knowledge sources, there are some ethical im- plications. Notably, LLMs using CoK may still generate inaccurate information the knowledge sources contain unreliable information. Hence, this could cause misinformation or manipulation of public opinion. Another limitation is that there may conflict between different knowledge sources in theory. To address the two limitations, we selected authoritative knowledge sources such as Wiki- data which are unlikely to contain inaccurate or conflicting information. As a result, the risk from the knowledge sources are reduced.
Knowledge Retrieval On the other hand, CoK may not produce useful outputs the knowledge retrieval step is unable retrieve facts that relevant to the given question. However, believe
19

---

Published as a conference paper at ICLR 2024
Table 12: Performances of ChatGPT and instruction-tuned LlaMA-2-7B SQL and SPARQI. generation.
Method
SQL Eval. Acc.
SPARQL Eval. Acc.
ChatGPT
57.1% 38.6%
8.9%
Finetuned Model
41.1%
Chain of Knowledge Human Evaluation
question, and generated the following reasoning chains before answering, which reasoning chain is factually consistent opinion? you think that reasoning chain lead to better predictions?
Large Language Model trying
Remember, you can use google sources could
up! Wikipedia your
When the question is a narrative sentence (without question mark), the objective of anewer to Judge the validity of the claim! (Supports, refutes, or not enough in information)
Question:
Henry II of France was not involved with the cause of war about religion.
Reasoning Chain 1:
First, Henry II of France reigned from 1547 to 1559, during time of roligious conflict between Catholics and Protestants. Second, Henry of France known for his involvement in the French Wars of Religion, which were primarily fought religious differences.
Reasoning Chain
First, Henry France reigned
during religious conflict Catholics and Second, Henry France Catholic and played significant religious
conflicts
Figure 4: Human evaluation instructions.
that this is a general limitation of retrieval methods, as retrieval results inevitably contain some noise due to lack of relevant data inaceurate queries. To address this challenge. have designed the CoK framework to be modular and flexible. Hence, the adaptive query generator models can be easily swapped for other models that may be suitable for the given task. Rather than focusing on using specific query generator models, our focus is that heterogeneous knowledge sources can be effectively incorporated with LLMs improve their factual correctness and performance on knowledge-intensive tasks.
Reasoning Capability LLMs As CoK relies on the reasoning capability of LLMs, failure cases may stem from reasoning failures of LLMs. We believe this is general limitation of generative models, as LLMs inevitably generate reasoning errors. Case studies of such failures can be found in Appendix H.2 To address this challenge. CoK is designed to be modular and flexible. And the black-bor LLM can be easily swapped for more advanced models possessing enhanced reasoning capabilities.
20

---

Published as a conference paper at ICLR 2024
1. Which reasoning chain factually consistent your opinion? (you google search!)
Reasoning Chain
Reasoning Chain
Tie
2. Do you think that reasoning chain will better answer predictions?
Yes, will lead BETTER answer than the other chain.
No, it will lead to a WORSE answer than the other chain.
Tle, it will likely be the same answer.
Figure 5: Human evaluation questions.
H HUMAN STUDY
H.1 SETUP
In the human study, volunteers are proficient English speakers in relevant disciplines. The instruction specifically asks the volunteers to verify their knowledge from Google, especially the Wikipedia data source. The specific instructions given to the users are shown 4 The questions given to the users are shown in 5 First, the user asked which reasoning chain factually consistent in his/her opinion. Here, we use direct assessment rather than a comparative measure (for example, one more factually correct than the other). Intuitively, factual consistency should not be "more" or "less". Similar direct measures also preferred by the community, such the direct assessment in Machine Translation (Koemi et al. 2022 Graham et al.) 2017). If they are both incorrect or both correct, the could choose "Tie" Then, the user is asked whether he/she thinks the better reasoning chain will lead to better answer predictions. In scenarios where the user answers "Tie" to the first question, he/she will also answer "Tie" for the second question.
For evaluation, 100 samples are randomly chosen from HotpotQA and Fever datasets. The order of the reasoning chains (produced by CoK CoT-SC) is randomly perturbed for each question.
H.2 EXAMPLES
when we improve the factual consistency of the CoTs, the outputs could still be false due to LLM's reasoning errors. We copy three such examples below:
As mentioned in section 5.3
Example 1:
Prompt: [3-shot CoT prompt]
Q: Anne Sullivan was born in June of 1866.
A: First, Anne Sullivan was born on April 14, 1866 in Feeding Hills, Agawam, Massachusetts, United States. Second, Anne Sullivan born April 14, 1866 in Feeding Hills, Agawam, Massachusetts, United States. The answer is
ChatGPT: SUPPORTS.
Example 2:
Prompt: [3-shot CoT prompt]
21

---

Published as a conference paper at ICLR 2024
Table 13: Details costs (by tokens).
Method
Dataset
tokens
ReAct
HoipotQA
16388
Verify-and-Edic
HotpotQA
630
CoK
HolpolQA
787
ReAct
FEVER
848
Verify-and-Edit
FEVER
286
CoK
FEVER
329
Q: Practical Magic is based on French novel that was written by Alice Hoffman.
A: First, Practical Magic a 1998 American fantasy romantic drama film based on the 1995 novel of the same name by Alice Hoffman. Second, Alice Hoffman is an American author. The answer is
ChatGPT: SUPPORTS.
Example 3:
Prompt: [3-shot CoT prompt]
Q: Saturn Corporation has no other names.
A: First, The Saturn Corporation, also known Saturn LLC, was a registered trademark established on January 7, 1985, as subsidiary of General Motors. Second, There is no information available on any other names for Saturn Corporation, but it is also known as Saturn LLC.. The answer
ChatGPT: SUPPORTS.
In the first example, is mentioned twice in the prompt that Anne Sullivan was born in April. However, the LLM still supports the claim that she was born in June. In the second example, the CoT specifies that the novel American. However, ChatGPT overlooks the nationality and supports the claim that it is based on a French novel. In the third example, the CoT mentions repetitively that Saturn Corporation is also known as Saturn L.C.. However, ChatGPT supports the claim that has no other names.
These examples show that, even though the CoT is successfully improved in terms of factual con- sisteney, the final answer may still be incorrect due to reasoning errors inherent to LLM itself. In the human study for wrong predictions, 44% of the time humans claim that CoK still generates im- proved CoTs. Among these 44% instances, 73% of the time humans think these CoTs should have led to better answers.
I COST ANALYSIS
As CoK always edits instances below a certain consistency threshold, there is a cost advantage compared to other methods such as ReAct. The costs are on par with methods such as Verify-and- Edit.
A table of the costs is shown in [3 The costs are calculated based on tokens used per instance. Overall, the costs for CoK are on par with Verify-and-Edit. The extra costs are incurred by the dynamic knowledge editing stage, which is shown to boost performance in the main results. CoK also costs much less than ReAct, incurring only around 40% of ReAct's costs. Specifically, it costs 787 compared to 1638 for HotpotQA, and 329 compared to 848 for FEVER.
The API cost for gpt- 5. 5-turbo is currently $0.0015 IK tokens for input, and $0.002 / IK tokens for output.
For details of the cost calculations, as the output length is the same for all methods, we only calculate the input tokens. Following the original ReAct paper Yao et al.) 2023), calculate based on 3-shot prompts for FEVER and 6-shot prompts for HotpotQA. Verify-and Eddi and CoK tokens per instance calculated based the CoT-SC threshold, which results in editing 86 out of 308 instances for
22

---

Published as a conference paper at ICLR 2024
HotpotQA 6-shot, and 127 out of 1,000 instances for FEVER 3-shot. The plain ReAct method, on the other hand, applies the ReAct prompt to every instance.
23


Reviews:
[{"review_id": "bfmlrjvVlf", "reviewer": "Reviewer_84zL", "summary": "This is a very interesting paper on Chain of Knowledge and integrating the question answering from LLM with traditional source of knowledge such as KG to improve the overall accuracy of question answering in domain-specific text.\nMain contributions:\n1. Chain of Knowledge for factual correctness\n2. Adaptive query generator for identifying knowledge source and converting queries in respective form\n3. Progressive correction of rationale using CoK\n4. Following domains have been chose for the work factual, medical, physical, and biological.", "questions": "* For the given SPARQL query SELECT ?answer WHERE { wd:/Souleymane Sane/ wdt:/child/ ?answer . ´ }, do you use some library to convert the entities/relation in their specific identifier form (entity linking), how do you know which is the correct identifier, in specific cases where similar natural language entities have two identifiers.\n* Did you try domain specific questions with ChatGPT? Upon trying the example given in Figure 1 on ChatGPT, I could see the result as George Gamow. Maybe you should come up with an example where ChatGPT would hallucinate. Or if the idea here is to show open source LLM, then this example makes complete sense. Although it should be mentioned in this case which model is used (Llama2-7b-chat? or instruction tuned by yourself)\n* It is possible that the knowledge injected through Wikidata doesn't answer the question asked, what would be done to answer the question in that case?\n* Can you please explain how do you find specific relation for querying with an entity? There is a possibility of multiple entity attached with one relation. It is also possible that multiple relations connected with multiple entities, and it may be difficult to identify the relation depending on the question. What would be done in that case?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is very well written.\nThe authors have considered good number of scenarios for knowledge, and carefully selected wikidata as the knowledge source, provided the limitation on retrieval of knowledge in general.\nI enjoyed reading this paper.", "weaknesses": "Please see the questions section for more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dCrQqzah2R", "reviewer": "Reviewer_bxPp", "summary": "This paper proposes a chain-of-knowledge framework to augmenting LLM in question answering by dynamically retrieving the grounding information from multiple sources. The proposed method first gives the rationale in steps. Then the search query is generated by determining the domain involved in each rationale and the final query. Eventually, the rationale is corrected by the information obtained from the query to give a more plausible answer. The authors conducted experiments on multiple datasets and defeated the compared few-shot methods.", "questions": "1. How does the author ensure that LLM can give faithful answers (query) when performing knowledge adaptation?\n2. The proposed method leaves the decomposition of the question (i.e., the generation of each rationale) to ChatGPT. I am interested in whether this method is only applicable to chain-style question? For more complex questions, where the corresponding SPARQL contains more than one inference path, can the proposed method handle it?\n3. When generating SQL (SPARQL) queries, how do the authors guarantee that the produced queries are valid?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This article proposes a novel idea to reduce the hallucination of LLM by correcting each rationale in the chain of thought.\n2. The proposed method considers structured and unstructured queries for heterogeneous data sources.\n3. The authors conducted detailed experiments and analysis on multiple datasets from different domains.", "weaknesses": "1. The general outline of the suggested approach is somewhat understandable, but the crucial stages lack the necessary level of specificity. In my opinion, a key point in the effectiveness of the proposed method is how to construct accurate queries to retrieve grounding information, but the description in section 3 is still not clear enough. For example, how to build a high-quality question-sparql dataset to ensure the accuracy of this step. While the prompt used with the training loss function is provided in the appendix by the authors, I believe it is important for the main body of the paper to be self-contained, and critical steps like this should not solely rely on the information provided in the appendix.\n2. How does the author determine the table or tables used when generating a SQL query for a rationale, and how can they ensure that the query generated by ChatGPT is valid and answerable if table schema information is not provided. Ditto for SPARQL.\n3. Baseline performance on the hotpotQA dataset seems too poor. Existing work [1] has achieved an accuracy of 0.37 when reviewing ChatGPT.\n\n[1] Zheng, Shen, Jie Huang, and Kevin Chen-Chuan Chang. \"Why does chatgpt fall short in providing truthful answers.\" ArXiv preprint, abs/2304.10513 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y9A2B0Ilt7", "reviewer": "Reviewer_SrPL", "summary": "The paper introduces the chain-of-knowledge (CoK) framework designed to augment the performance of LLMs by reducing instances of information hallucination and improving factual correctness. The CoK operates in three stages: reasoning preparation, dynamic knowledge adaptation, and answer consolidation. A distinctive feature of CoK is its ability to dynamically tap into diverse knowledge sources, including structured databases like Wikidata, using an adaptive query generator (AQG) capable of generating varied query languages, such as SPARQL, SQL, and natural language queries. The framework emphasizes progressive correction of the generated rationales, minimizing the risk of error propagation across reasoning steps. Experimental evaluations indicate that CoK enhances the performance of LLMs on knowledge-intensive tasks across different domains, surpassing the chain-of-thought (CoT) baseline by an average of 4.3%.", "questions": "- How does the CoK framework handle situations where knowledge sources provide conflicting information? How are you going to solve it?\n- What are the computational overheads introduced by the AQG and the dynamic knowledge adaptation process, especially when querying multiple heterogeneous sources?\n- Have there been considerations or plans to extend the CoK framework to accommodate real-time or streaming knowledge sources?\n- Can you provide insights into the training and fine-tuning process of the AQG, especially its adaptability across different query languages and knowledge sources?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- CoK addresses the inherent limitations of existing methods by leveraging diverse knowledge sources, both structured and unstructured, improving the accuracy and factual correctness of LLM outputs.\n- The adaptive query generator (AQG) showcases versatility, enabling seamless transition between specialized models and generic LLMs, allowing effective querying across different knowledge source formats.\n- Progressive rationale correction reduces the risk of error propagation, ensuring more reliable generation of answers.\n- Comprehensive experiments covering various knowledge domains, demonstrating a consistent performance boost over the CoT baseline.", "weaknesses": "- While the paper emphasizes the use of diverse knowledge sources, it might benefit from a more explicit discussion on the scalability and efficiency of the framework as the volume and variety of sources grow.\n- The framework's dependence on the AQG's capability to generate effective queries for all types of knowledge sources might be a potential bottleneck, especially for highly specialized domains.\n- Although the paper mentions improvement over the CoT baseline, deeper insights into the limitations and areas where CoK might not perform optimally would be beneficial. Except for the limitations of “Knowledge Sources” and “Knowledge Retrieval” discussed in Appendix G.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nIjM7qFODb", "reviewer": "Reviewer_TzUb", "summary": "This paper proposed chain-of-knowledge (CoK), a novel framework that augments LLMs by dynamically incorporating grounding information from heterogeneous sources. The CoK framework consists of three stages, reasoning preparation, dynamic knowledge adapting, and answer consolidation. This paper provided rich experiments and analysis.", "questions": "Q1. The authors specifically evaluated the proposed framework in experiments, with a focus on the biology, medical, and physics domains, showing better results compared to the VE method. The knowledge sources used by the author include Wikidata, medical Flashcard, UpToDate, ScienceQA Physics, and ScienceQA Biology. I'm wondering if the authors also incorporated these knowledge sources into the VE method. If not, I believe such a comparison would be unfair.\n\nQ2. LLMs generate multiple results based on self-consistency, if the consistency falls below a threshold, the proposed method is initiated to correct the answer. I'm curious about how often such a process needs to be introduced in practical experiments. Which of the three extraction methods (natural language, SPARQL, and SQL) do the authors consider to be more reliable for LLMs?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. This paper proposed chain-of-knowledge (CoK), a novel framework to enhance the factual correctness of LLMs with heterogeneous knowledge sources. CoK can dynamically select one or multiple knowledge sources based on the types of questions.\n\nS2. Cok participates in Cot which modifies incorrect answers in each reasoning step by introducing external knowledge, thereby generating the correct answer.\n\nS3. This paper performs extensive experiments on knowledge-intensive tasks spanning a range of domains, including factual, medical, physical, and biological.", "weaknesses": "This paper bears a high resemblance to VE with the main distinction being the inclusion of multiple knowledge sources, which might be considered a relatively ordinary contribution. The AQG retrieves relevant knowledge from multiple knowledge sources by converting questions into SPARQL, SQL, and natural language queries. The SPARQL query generator is fine-tuned based on question-SPARQL pairs. I think this training process may make it challenging for AQG to generate SPARQL queries for complex and compositional questions. Because it cannot guarantee that the sub-questions derived from the CoT decomposition are all simple.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)