Knowledge-Augmented Language Model Verification
Authors: Jinheon Baek
Soyeong Jeong
Minki Kang
Jong C. Park
Sung Ju Hwang
Venue: EMNLP 2023 Main (2023)
Keywords: Knowledge-Augmented Language Models
Verification
TL;DR: We propose to not only verify the errors of knowledge-augmented language models in knowledge retrieval and answer generation but also rectify them if errors are detected.

Abstract: Recent Language Models (LMs) have shown impressive capabilities in generating texts with the knowledge internalized in parameters. Yet, LMs often generate the factually incorrect responses to the given queries, since their knowledge may be inaccurate, incomplete, and outdated. To address this problem, previous works propose to augment LMs with the knowledge retrieved from an external knowledge source. However, such approaches often show suboptimal text generation performance due to two reasons: 1) the model may fail to retrieve the knowledge relevant to the given query, or 2) the model may not faithfully reflect the retrieved knowledge in the generated text. To overcome these, we propose to verify the output and the knowledge of the knowledge-augmented LMs with a separate verifier, which is a small LM that is trained to detect those two types of errors through instruction-finetuning. Then, when the verifier recognizes an error, we can rectify it by either retrieving new knowledge or generating new text. Further, we use an ensemble of the outputs from different instructions with a single verifier to enhance the reliability of the verification processes. We validate the effectiveness of the proposed verification steps on multiple question answering benchmarks, whose results show that the proposed verifier effectively identifies retrieval and generation errors, allowing LMs to provide more factually correct outputs. Our code is available at https://github.com/JinheonBaek/KALMV.

Knowledge-Augmented Language Model Verification
Jinheon Baek
Soyeong Jeong
Minki Kang Jong C. Park Sung Ju Hwang
KAIST
(jinheon.bar baek, starsuzi, ZZxc1133, jongpark, sjhwang822ekaisst.o ac.
Abstract
updating the LM parameters while using only the knowledge internalized in their parameters. How- ever, while the generated answers from LMs look plausible and sound, they are often factually incor- rect, which is a problem widely known as halluci- nation (Rohrbach et al., 2018; Bang et al., 2023; Zheng et al., 2023). Hallucination is a critical prob- lem when deploying LMs, since it poses risk of spreading misinformation, potentially misleading users who rely on the information.
Recent Language Models (LMs) have shown impressive capabilities in generating texts with the knowledge internalized in parameters. Yet, LMs often generate the factually incorrect re- sponses the given queries, since their knowl- edge may be inaccurate, incomplete, and out- dated. To address this problem, previous works propose to augment LMs with the knowledge retrieved from an external knowledge source. However, such approaches often show subop- timal text generation performance due to two reasons: 1) the model may fail to retrieve the knowledge relevant to the given query, 2) the model may not faithfully reflect the retrieved knowledge in the generated text. To overcome these, we propose to verify the output and the knowledge
To mitigate hallucination of LMs, recent works have proposed to augment L.Ss with the knowledge retrieved from external knowledge sources (c.g.. Wikipedia and Wikidata) (Lazaridou et al., 2022; Mallen et al., 2023; Back et al.. 2023). Moreover, some other works have proposed to check the factu- ality of generated texts and refine them by using the knowledge in LMs themselves from the external knowledge sources (Madaan al., 2023; Gao al., 2023; Jiang al., 2023; Gou al., 2023; Xu al., 2023; Feng et al., 2023). However, while the afore- mentioned
snowledge-augmeented LMs with separate verifier, which is small LM that is trained to detect those two types of errors through instruction- con.tnttioo-iintuningg... Then, when verifier recognizes егrоr, by either retrieving new knowledge or gener- ating new text. Further, we use an ensemble of the outputs from different instructions with a single verifier to enhance the reliability of the verification processes. We validate the ef- fectiveness of the proposed verification steps on multiple question answering benchmarks, whose results show that the proposed verifier effectively identifies retrieval and generation errors. allowing LMs to provide more factu- ally correct outputs. Our code is available at http://ggiiaah..o///hheeee/////
rectify it
auggddgg-augmmnttiionnn strategies are effective in reducing hallucinations, we find that there still exists couple of challenges: 1) the re- trieved knowledge may not be relevant to the given question from the user, and 2) the generated answer may not be grounded in the retrieved knowledge, as illustrated Figure I and shown in Figure 2.
In this work, we aim to overcome these subop- timalities of knowledge-augmentee LMs. In other words, our goal is to verify whether the retrieved knowledge used for augmenting LMs related to generating the answers for the given questions and whether the generated answers include the relevant parts of the retrieved knowledge. To this end, we propose to train a small, tailorable LM that is able to verify the aforementioned two failure cases of knowledgg-augmeented LMs in retrieval and gen- eration steps. More specifically, we first automati- cally construct the training labels by categorizing the failure of knowledge anowledge-augmented LMs into two
1 Introduction
Recent Language Models (LMs) (Brown et al., 2020; Chowdhery et al., 2022; Chung et al., 2022), which have a large number of parameters and are on massive datasets,
further iestruction-iietuune have achieved remarkable successes on various lan- guage tasks. For example, they are able to perform closed-bood bbok shro question answering, which aims to provide an answer to a user's 's query without

---

Question: Where was Michael F. P Phelps bom?
Input Prompt Below are facts that might meaningful to answer the given question.
Retrieval
Below are facts that might be meaningful
Input Prompt:
Retrieval
the given question.
(Michael
Phelps, occupation Swimmer)
(Michael F. Phelps, plcce
Baltimore)
(Michael Phelps, spouse. J johnson)
(Michael F. Phelps, date of birth, Jun 1985)
Question: Where was Michael F. Phelps tom? Answer:
Knowledge Base (KB)
Question: Where was Michael F. Phelps bome Answer:
Knowledge Base (KB)
Generated Answer:
Generated Answer:
Michael Phelps is a swimmer and married Nicole J ohnson on un 13, 2016.
Michael F. P Phelps was bomn on J uly 16, 1997, Los Angeles, California, United
Retrieval rror
Grounding Error
Question
Retrieval Error
Refuse Do retrieval again
LQQ
Grounding Error
Refuse
Do generation again
Retrieved Knowledge
Correct Answer
Return generated answer
Generated Answer
KALMV (Ours)
Verification
Action
language models first retrieve the relevant knowledge the given query from the external knowledge base and then augment the LMs with the retrieved knowledge generate the factually correct responses. However, there
Figure I:
Existing
types of common errors: 1) the retrieved knowledge might be irrelevant the given query (retrieval emrrr): 2) the generated answer might be grounded in the retrieved knowledge (grounding error). Our proposed KALMV can detect those types of errors in knowledge retrieval and grounding, and also iteratively rectify them, reducing hallucinations.
cases: retrieval error and generation error, based on the triplet of the input question, retrieved knowl- edge, and generated answer. Then, we instruction- finetune the LM with pairs of certain verification instruction and its associated label, during verifier training. At the inference step, we validate the generated texts through our verifier, to filter out potentially incorrect generations due to retrieval or generation failures, to prevent the generation of texts with inaccurate information. Note that there exists concurrent work (Peng et al., 2023) that proposes to check whether the generated answers from LMs are grounded in the knowledge provided LMs, by using API calls proprietary LLMs or heuristic measure (F1). However, this work clearly differs from our method, since we further verify the relevance of the retrieved knowledge in addition to the answer groundedness, through instruction-finetuning of LMs.
gle verifier. Notably, one extra advantage of our verifier is that it is plug-and-play module that works with any public proprictary LMs, since only require input-output pairs of LMs for verifi- cation without any architectural changes. We refer to our proposed method Knowledge-Augment Language Model Verification (KALMV).
We experimentally validate the effectiveness of our KALMV on two different Question Answering (QA) tasks, namely open doen-domain QA and knowl- edge graph QA. The experimental results show that our KALMV can effectively verify the failure cases of knowledge-augmentee LMs in knowledge retrieval and answer generation steps, contributing to significant reduction of the hallucination. Also, further analyses demonstrate the effectiveness of our error- pror-rectiffing and ensemble strategies.
Our findings and contributions are threefolds:
We point out the underexplored challenges of knowledge-augmenter LMs, which are re- trieval of irrelevant knowledge and unfaithful knowledge grounding.
In addition, we further propose refining the out- put from mhowledge-augmenn ifier identifies the error in either the knowledge retrieval or the knowledge reflection. Specifically, we repeat the answer generation process until the model retrieves the knowledge relevant to the given question and incorporates the correctly retrieved knowledge into the generated answer, based on the verifier outcome. Also, since detecting errors of Knowledge-augmented LMs with single in- struction given to the verifier might be inaccurate, we further construct an ensemble over multiple outputs from different instructions with a sin-
LMs if our ver-
We introduce a novel verifier that identifies whether the retrieved knowledge relevant the question and reflected in the answer, and further present useful strategies for rectifying incorrect answers as well as improving the effectiveness of the verifier via ensembling.
We validate our KALMV on open-domain and knowledge graph question answering tasks, demonstrating its effectiveness in verifying the errors of mnoweedge-uugmented LMs.

---

2 Background and Related Work
Trivedi al., 2022: Back et al., 2023; Shi et al., 2023; Peng et al., 2023) use the large or instruction- finetuned LMs to incorporate the external knowl- edge, which allows us to design only the input text to LMs without requiring additional training thanks to their strong generalization capabilities. Follow- ing this trend, we focus on cntructiionfinetunne LMs, while exploring their two underrepresented challenges: incorrect knowl- edge retrieval and unfaithful knowledge reflection.
Language Models Pre-trained Language Models (LMs) (Devlin et al., 2019; Liu et al., 2019; Radford et al., 2018; Raffel ct al., 2020), which are trained on large corpus with self-supervised learning, show impressive performances across diverse natu- ral language tasks and used as the base architec- ture. Recently, large language models (Brown et al.. 2020; Chowdhery et al., 2022; Touvron al., 2023) having billions of parameters are able to respond to a user's query without any model training on the target task. On the other hand, finetuning LMs on a massive collection of natural language datasets phrased as instructions (Wei et al., 2022; Chung et al., 2022; Sanh et al., 2022), which is known instruction finetuning, also enables the LMs to attain reasonable zero-shot learning abilities with- out focused training on the target task. However, while large and instruction-inetumee performance improvement on factual tasks (e.g.. question answering), they are still suboptimal since they cannot memorize all the world knowledge and may contain distorted facts. To overcome this chal- lenge, recent studies propose augmenting LMs with external knowledge, which we discuss below.
Knowledge- Augeeeeed Fact Checking Similar to the motivation of the aforementioned knowledge- augmented LMs, recent works (Mallen et al., 2023; Gao et al., 2023; Peng et al., 2023; Jiang et al., 2023; Xu et al., 2023) propose to check the factu- ality of the answers generated by LMs using the external knowledge. Typically, these approaches generate the answer response to the user's query with LMs, and then identify whether the generated answer aligns with the retrieved knowledge. How- ever, there are significant differences between our work and the existing methods. First of all, they assume that the retrieved knowledge is pertinent, which is yet unrelated and unhelpful sometimes, making the model generate incorrect predictions. In contrast, our proposed verifier can recognize the relevance of the retrieved knowledge before incor- porating it into the LMs. Second, previous works suppose that the retrieved knowledge used for fact- checking is accurately reflected in the generated an- swer; however, LMs often ignore the given knowl- edge and hallucinate the answer. whereas we can detect and rectify such the grounding error. Lastly, unlike most fact-checking methods that always pro- vide the answer with its refinement, our method can further decline to provide answers unless they are validated as correct. These differences highlight the novel contributions of our verification approach, compared against previous fact-checking methods.
LMs show
Knowledge-Augmentee LMs Early works aim to incorporate knowledge from external knowledge sources (e.g., Wikipedia) into LMs, in order to en- hance their performances on tasks that require fac- tual knowledge, such question answering. While such previous knowledge-augmentted LMs (Zhang et al., 2019; Guu et al., 2020; Yamada et al., 2020; Qin et al., 2021; Borgeaud et al., 2022) show per- formance improvements on snowledge-intensin tasks, in order to integrate the external knowledge, they utilize the specific pre-training but also require changing the model architecture, which are not eas- ily generalizable across different LMs and tasks. Similarly, while some recent works (Lewis al., 2020; Kang al., 2022: Li et al., 2022; Izacard et al., 2022) propose augmenting LMs with exter- nal knowledge during finetuning, they also require specific training on each target task and dataset, and often require architecture modifications. However, training the task- and data-specific LMs with model updates are computationally prohibitive the size of LMs increases exponentially. Also, previous ap- proaches involving architecture changes are not ap- plicable to black-box LMs (e.g., ChatGPT), which are accessible only through API. Considering these challenges, recent methods (Lazaridou et al., 2022;
3 Method
We now formally describe knowledge-augne LMs, and present our method, Knowledge Aug- mented Language Model Verification (KALMV).
3.1 Knowledge-Angmented Language Models We begin with the explanation of language models.
Language Models In our problem setup, the goal of Language Models (LMs) is to generate a factu- ally correct answer in response to an input query from a user, which is formally defined as follows:

---

# - LM(a),,, where æ and ÿ are the input and out- put pair, each of which consists of sequence of tokens, and LM is the language model. We assume that LMs are already trained on massive instruction- finetuning datasets, which are capable of perform- ing diverse tasks (e.g., question answering) (Wei et al., 2022: Chung et al., 2022), and also not fur- ther trainable since we sometimes cannot update the parameters of LMs due to their huge sizes or inaccessibility (OpenAI, 2023; Anil al., 2023). Note that, while previous works (Petroni et al., 2019; Roberts et al., 2020) show that LMs are capa- ble of memorizing the knowledge seen during train- ing, such naive LMs encounter several challenges when dealing with factual questions. In particular, LMs cannot memorize all the factual knowledge due their limited number of parameters. Also, some knowledge is changed and updated over time; however, LMs remain static unless they are further trained while training them is also very expensive.
ticular, as shown in Figure 2, there are significant occurrences of retrieval and grounding errors.
3.2 KALMV: Learning to Verify
Knowledge-Augmeeteed Language Models
To overcome the challenges of existing knowledge- augmented LMs, we propose a novel verification method that identifies not only the relevance of the retrieved knowledge to the input question but also the reflection of the knowledge in the generated an- swer, which we refer to Moowlege-Augmentee Language Model Verification (KALMV).
Verification of Retrieved Knowledge Given the triplet of the input query, the retrieved knowledge. and the generated answer (az, k. 9), we aim to verify whether the retrieved knowledge & is rel- evant to the input query æ. Since recent LMs (Wei et al., 2022; Chung et al., 2022) can contextual- ize multiple sentences and understand their un- derlying relationships, use such small and instructioonfinetuueed LM identify the related- ness between the input query and the knowledge. To be specific, we prompt the verifier LM to deter- mine the relevance based on the verification instruc- tion as well as the input, knowledge, and gener- ated answer triplet (æe, k. g). formalized as follows: O₂ = fiiiiiiere..... x k. ÿ), where Verifiery de- notes the LM for retrieved knowledge verification, and o₁ denotes its output. Note that we formulate the verification task as multiple-choice question- answering task, i.e., the verifier should produce either "A" for incorrect retrieval or "B" for correct.
Knowledge-Augmented LMs In order to tackle the aforementioned challenges of naive LMs, some works (Lazaridou al., 2022; Mallen et al., 2023; Back et al., 2023) propose to augment LMs with the knowledge retrieved from the external knowl- edge base, called Knowledge-augmented LMs. For- mally, let K. be the external knowledge base, which could be an encyclopedia (Wikipedia) consisting of millions of documents or a knowledge graph (Wikidata) consisting of billions of facts. Then, we first retrieve the pertinent knowledge & from the knowledge base I. based on its relevance score the input query x, by using the retriever model denoted as follows: k Retriever (ar, KC) where k E K. After that, the retrieved knowledge is in- corporated into the input of the LM along with the input query, as follows: ý LM( k). This knowl- edge augmentation strategy brings impressive per- formance improvements on factual language tasks by reducing the hallucination issue of LMs.
Verification of Generated Answer Our next ob- jective is to identify whether the generated answer from LM is grounded in the retrieved knowledge. To achieve this, similar to the retrieved knowledge verification process explained in the above para- graph, we use the separate, small-size, instruction- finetuned LM for answer verification. Formally, given the input query, retrieved knowledge, and generated answer triplet (æ, k, ÿ). as well as the in- struction describing the task of generated answer verification, the verifier LM produces the output to- ken, namely "A" or "B" where "A" represents that the retrieved knowledge is not reflected in the gen- crated answer and "B" represents the vice versa, for- malized as follows: oy Veri firriiee,,, (i, æ, k, ÿ). Thus far, we propose to detect the errors of knowledge-augmentee LMS in knowledge retrieval and answer generation by using distinct LM-based verifiers. However, it is inefficient to perform two
However, despite the enormous successes of the aforementioned wwwlledge-augmentee LMs, there exist remaining issues that have largely underex- plored. First, the knowledge retrieved to augment LMs might be irrelevant to answer the given ques- tion, since the retrieval is not always accurate in real-world scenarios. Second, even if the retrieved knowledge is useful, LMs sometimes reflect the irrelevant part of the retrieved knowledge, or might completely ignore the knowledge and generate the answer based on their incorrect knowledge. In par-

---

individual verification processes, since both verifi- cation formulations are identical. Also, the knowl- edge retrieval and answer generation processes are sequential, which means that verifying the gener- ated answer is unnecessary if the retrieved knowl- edge is irrelevant. Therefore, we further combine two verification procedures into one by changing the task instruction accordingly with the single ver- ification LM (Verifier) Specifically, Veri fier produces one among the following three options: A. the retrieved knowledge is not helpful answer the question; B. the generated answer is not grounded in the retrieved knowledge; C. all the other cases.
3.3 Strategies for Rectifying Errors of Knowledge-Auugmetted Language Models Our verification method provides distinct advan- tage in contrast to existing knowledge aagggeegeeeeeeeeedd LMs and soowledge-augmentn proaches. That is, existing approaches always pro- vide the answers to users even they are not re- liable; however, our method can withhold the an- swers if errors are detected by the proposed verifier, which can enhance the reliability and trustworthi- ness of LM-based systems. However, instead of simply refraining from responding user queries, it is more worthwhile rectify errors in the knowl- edge retrieval and answer generation stages. Thus, we further propose simple yet effective strategies, iteratively correcting errors detected by our verifier.
fact-checking ap-
Instruction-Finetunnng for Verifier While re- cent
LMs might be capable of performing the proposed verification task, it may be more beneficial to tailor the LM to the verifica- finttuttionnnnttuiing..
Rectifying Errors in Knowledge Retrieval The retrieved knowledge from the external knowledge base might be irrelevant to answer the question due to the retrieval error, which may mislead LMs generate an incorrect answer. To overcome this is- sue, we retrieve the new knowledge iteratively until our verifier confirms that the retrieved knowledge is related to answering the question, for a certain number of times (e.g., ten times). Specifically, the knowledge with the highest relevance score to the question is retrieved, while excluding any knowl- edge that has been used in the previous iterations.
tion task through additional instruction finetuning. To perform this, we require the following input- output pairs: {(2, k. b). where the input con- sists of the given question, retrieved knowledge, and true answer, and the output is the verification la- bel which we automatically generate. In particular, we first examine whether the retrieved knowledge includes the correct answer, y C k, annotated in the training data, and then label it as retrieval error when the knowledge does not include the correct answer. Similarly, if the retrieval correct yet the generated answer y from LM(x,, k) does not have overlapping tokens with the retrieved knowledge k; we label it as the generation error. Finally, for all cases where the generated answer is correct, we la- bel it as correct'. Then, by using the inputs phrased instructions and their corresponding labels, instruction-finetuee the proposed ifrifier.
Rectifying Errors in Answer Generation Even though the retrieved knowledge is pertinent to the given question, LMs sometimes ignore the knowl- edge augmented to them and then generate the answer based their inaccurate knowledge. To tackle this issue, similar to what previously did on knowledge retrieval, we iteratively generate the answer until the answer is confirmed by the verifier. for the specific number times. Note that, in order to generate the answer differently across different trials, we leverage the top-k sampling (Fan et al., 201 8) that enables stochastic generation processes.
Ensemble Verification To identify retrieval and generation errors in knowledge-augmentte augmented LMs, we forward the instruction along with the query, knowledge, and generated answer to the verifier. However, it might be inaccurate to determine the errors only with single instruction, since recent LMs are sensitive even minor changes in the input prompt (Zhao al., 2021; Lu al., 2022; Zhou et al., 2022) and also our small-size verifier LM might not fully understand the given input con- text. Therefore, we design various instructions, forward them to our single verifier, and ensemble the multiple outputs from the verifier with average.
Experimental Setups
In this section, we describe the datasets, models, evaluation metrics, and implementation details. We provide the additional details in Appendix A.
4.1 Tasks and Datasets
We evaluate our Knowledge-Augmennte Language Model Verification (KALMV) on factual Open- Domain Question Answering (ODQA) and Knowl- edge Graph Question Answering (KGQA) tasks.
'There might be more sophisticated techniques to automat- ically assign verifier labels, which we leave as future work.

---

Open-Domain Question Answering The goal of open-domain question answering (ODQA) task is to generate answers in response to factual ques- tions usually with the relevant knowledge retrieved from the external knowledge source. As the knowl- edge source, we use Wikipedia which is an open encyclopedia consisting of millions of documents. For datasets, we use Natural Questions (Lee et al., 2019) that is modified from Kwiatkowski et al. (2019) for ODQA and HotpotQA' (Yang et al., 2018), both of which are designed with Wikipedia.
LLM-Augmenter
This baseline (Peng et al., 2023) first augments LMs with knowledge retrieval, and then verifies whether the retrieved knowledge is reflected in the generated answer with Knowl- edge F1 (Shuster et al., 2021) that measures over- lapping terms between the knowledge and the an- swer. Yet, unlike our KALMV, it cannot identify re- trieval errors but also uses heuristic metric for ver- ification. In addition to the aforementioned L.M-- Augmenter w/ Knowledge F1, we also include the LLM-AAgmenter w/ Confidence that verifies the answer based on its generation probability.
Knowledge Graph Question Answering ad- dition to ODQA, we evaluate our KALMV method on knowledge graph question answering (KGQA), whose goal is to answer the questions that are an- swerable by the facts over knowledge graphs. For datasets, we use WebQSP (Yih et al., 2016) that is modified from Berant et al. (2013) filter out unanswerable questions, and Mintaka (Sen al., 2022). Further, for the knowledge source, we use Wikidata which includes billions of facts that are represented as the triplet: (subject, relation, object), and we follow the standard preprocessing setup for KGQA (Saffari et al., 2021; Baek et al., 2023).
This is our Stowledge-Augmennte Language Model Verification (KALMV) method, which not only verifies both the retrieval and gener- ation errors with the instruction-finetuee tailored verifier, but also iteratively rectifies errors.
KALMV
4.3 Evaluation Metries
Following the standard evaluation protocol of gen- erative QA (Mallen et al., 2023; Back et al., 2023), we use F1 which measures the number of overlap- ping words between the generated answer and the labeled answer with precision/recall, EM which measures whether the generated answer exactly the same as the labeled answer, and accuracy which measures whether the generated answer includes the labeled answer. For KGQA, following Baek et al. (2023), we further consider a set of alternative names of the labeled answers available in Wikidata.
4.2 Baselines and Our Model
We compare our KALMV against relevant base- lines that augment LMs with external knowledge and have strategies to reduce hallucinations. Note that models including verification refrain from providing answers if the verifier identifies errors.
4.4 Implementation Details
Naive Language Models This baseline uses only the LMs without incorporating external knowledge.
We use the same retriever across different models for fair comparisons. In particular, for ODQA, we use BM25 (Robertson et al., 1994) that considers the term-based matching, following Mallen et al. (2023). Also, for KGQA, we use MPNet (Song et al., 2020) that is based on the dense retrieval, following Baek et al. (2023). For the input prompt to LMs for all baselines and our model, we fol- low the existing works (Mallen al., 2023; Back et al., 2023) which use the simple prompt, such as "Context: (Context]. Question: (Question). An- swer: " Regarding the LMs to generate answers, we use FLAN (Chung et al., 2022) with three dif- ferent sizes: Base, Large, and XL having 250M, 780M, and 3B parameters, respectively. In our KALMV, use the FLAN Base as the verifica- tion LM, and we instructionnfinectue it with the batch size of and the learning rate of 5e-5 with AdamW (Loshchilov and Hutter, 2019) as the op- timizer. In addition, we set the maximum number
This baseline aug- ments LMs with the knowledge retrieved from the external knowledge base (Wikipedia Wikidata).
Knowledge-Augmeetted LMs
Adaptive Retrieval This baseline (Mallen et al., 2023) adaptively augments the LMs by retrieving the knowledge only when the external knowledge is necessary. In particular, if the entity that appeared in the question is less frequent, they retrieve the knowledge and provide it to the LMs. This model, namely Adaptive Retrieval with Entity, is appli- cable to questions that have pre- annotatdt entities (i.e.. KGQA); therefore, we also include its variant, namely Adaptive Retrieval with Confidence, that augments LMs with retrieval only when the answer generation probability of naive LMs is low.
hhtpss//huggingface http:///uuggngggaeeecoo///attstt open

---

edge retrieval and answer generation steps. In addi- tion, for knowledge graph QA, we also validate our Table 1: Results Natural Questions and HotpotQA for open-domain question answering and WebQSP and Mintaka for knowledge graph question answering. with FLAN of different sizes as the LM. We emphasize the best results in bold.
Base (250M)
Large (780M)
XL (3B)
Datasets
Methods
F1
EM
Ace
F1
EM
Acc
F1
EM
Acc
Naive I Language Models Knowledge- Auddeeetd LMs
7.53
3.24
4.57
11.09
6.29
7.81
16.89
11.16
12.94
18.06
12.30
15.26
18.61
13.74
16.40
19.03
14.13
16.90
Natural Questions w/ Wikipedia
Adaptive Retrieval w/ Confidence
16.70
11.02
14.07
18.16
13.07
15.60
20.89
15.76
18.28
J..M-uugeeneer w/ Knowledge Fi
19.58
13.56
16.81
28.53
21.22
25.32
31.00
23.06
27.59
ALL--nggeettrr w/ Confidence
19.91
14.14
17.19
20.19
14.97
18.29
22.88
17.17
20.49
KALMV (Ours)
52.98
42.36
50.43
56.80
46.13
53.57
67.43
58.06
63.17
Naive Language Models
14.25
9.68
10.36
16.80
11.78
12,41
21.97
16.22
Ghoolecdg-Auggeenne LMs
31.20
22.77
25.13
33.46
25.29
27.37
35:47
27.08
29.14
HotpotQA w/ Wikipedia
Adaptive Retrieval w/ Confidence
26.82
19:10
21.11
26.80
19.65
21.23
29.41
21.55
23.54
I.MM-Augmenter w/ Knowledge
32.89
23.24
26.12
39.40
28.55
31.60
46.97
34.54
37.72
LLM-Augneenter w/ Confidence
34.75
25.67
28.20
35,78
27.29
29.38
40.57
3
33.71
KALMV (Ours)
64,06
$2.31
55.84
63.74
52.39
55,98
67.21
54,99
58.07
Naive I Language Models
32.53
21.33
25.77 78
40.33
30.08
32,74
46, 20
36.43
40.11
LMs Adlaplive Retrieval w/ Eniiyy
53.57
43.25
53.68
42.37
26.13
62.28
49.45
36.02
59.28
49.13
37.79
46.32
47.81
35.68
49.32
51.99
41.54
51.16
WebQSP w/ Wikidata
Aduptive Retrieval w/ Confidence
46.76
36.49
43.66
48.32
36.56
51,98
$3.17
43.32
$3.89
ALM--nggenterr w/ Knowledge
56.42
45.95
56.26
44.41
27.79
64.56
51.95
38.12
61.96
J.M--Augenterr w/ Confidence
56.62
47.33
56.36
44.35
28.79
64.47
50.63
36.62
60.67
KALMY (Ours)
74.31
63.92
77.78
54,79
45.46
82,71
67.10
50.81
83.21
Naive Language Models
16.16
8.53
10.59
20.90
12.83
14.46
26.99
19.08
21.22
Knowledge- Peee--uugmeeett L.ss
24.28
15.46
19.15
24.57
15.39
23.77
27.74
18.23
22,92
Adaptive Retrieval w/ Eniity
23.66
14.68
17.87
25.96
16.45
22,92
30.34
24.20
Mintaka
Adaptive Retrieval w/ Confidence
21.46
13.15
16.06
25.34
16.28
22.07
29.00
20.68
23.70
w/ Wikidata
I.I. A.M.--egeettrrr w/ Knowledge F1
27.99
18.18
22.14
28.19
18.07
27.15
34.23
22.77
28.05
LLM-Angmenter w/ Confidence
28.16
18.74
22.26
28.46
18.88
27.42
33.24
22.55
27.31
KALMY (Ours)
59.29
$1.52
59.13
$3.15
42.30
62.87
58.15
48.44
59.11
Table 2: Results on WehQSP and Mintaka, where we use Wikipedia as the knowledge source and report results with Fl.
Venliotter Questions
WeaCSP
Mintala
HotpotQA
Datasets
Methods
Base
Large
XL.
Naive Language Models
46.20
Know lengelege--aae Augmeetedd Ls
27,96
27.39
26.40
Adaptive Retrieval or Contirece
36.15
41 11.68
44.89
WeboSP
LLM-Auggneer Knowledge
Dab De1 Ca:
Jata DiL oz Dor
Bade De. Oz
Tate Mr. Or.
L-MM-Aggenter wi Contitinnss
30.01
28.75
29.70
Figure 2: Ratios of verification types and verification accu- racies on them, on each dataset with the FLAN Base as LMs.
KALMY (Ours)
56.70
600663
63.75
Nuive Language Models
16.16
20.90
26.99
Knowledgs--e Auggmnnggeeeeddd LMs
27.10
26.25
28.32
Adaptive Retrieval w Confidence
Mintaka
I. LLM-Ange M-AAgmenter wi knoneeedge In
29.84
40 30
43.87
M-Aggngeetttr Contidence
28.81
27.64
30.91
KALMY (Ours)
70.83
KALMV on the setting where LMs are augmented with the documents from Wikipedia in Table 2. on which it also outperforms baselines substantially. Note that LLM-Augmenter, which verifies whether the generated answers are grounded in the retrieved knowledge, shows decent performance compared to other baselines. However, KALMV outperforms it by large margins, which suggests the importance of verifying the retrieval error and training the sepa- rate LM compared using the heuristic measure to verify only the groundedness in answer generation.
of crror-rectifying steps in the range of {1, 3).
and filter out answers that are determined to have errors by our verifier after the maximum step. Fur- ther, for the ensemble, we use 5 different outputs, which have the probabilities of three choices (Sec- tion 3.2), from 5 different instructions, and average probabilities to select one option for verification.
5 Experimental Results and Analyses
Main Results We conduct experiments on two question answering tasks: open-domain QA with Wikipedia and knowledge graph QA with Wikidata. As shown in Table our proposed KALMV sig- nificantly improves the performance of knowledge- augmented LMs on all datasets across different LM sizes by effectively verifying errors in the knowl-
A Analyses on Verification To understand how the proposed verifier works, we analyze it in multiple aspects. In the first bar of each subplot in Fig- ure 2, we report the percentages of the knowledge retrieval error, the knowledge grounding error, and the correct generation, and we can see that the most common errors come from the incorrect knowledge

---

Natural Questions
HotpetOA
WebOSP
Mintaka
Table 3: Ensemble and Sensitive Analyses on retrieval,
call
Predision
verification, and generation, the Natural Questions data.
16
Verification
Generation
Categories
Types
Acc
Ft
Acc
F1
55.91
52.98
Ensembble
No
76.45
53.37
48.40
50.68
Rectzying Steps
Rectaying Stops
5
Rectiyang Steps
Retrieval Models
BM25 DPR
55.91
52.98
Figure 3: Varying the number of rectifying steps, on each dataset with F1, Recall, and Precision as the verifier metries.
69.53
61.53
54.72
55.68
15 (250M)
50,00
Verification
(250M)
55.91
50.43
52.98
ChatGPT
65.71
43.17
33.16
56.68
retrieval, which signifies the importance of verify- ing the retrieved knowledge. Also, on the remain- ing three bars in Figure 2, report the verifier accuracy on each class category and then observe that our KALMV is able to detect errors in a bal- anced way across different verification categories.
54.52
62.35
Generation LMs
FLAN (3B)
79.11
56.76
63.17
67.43
CharGPT
77.14
55.65
69.42
72.23
Table 4: Results on Transfer Settings, where our KALMV is trained on the Source dataset and tested on the Target dataset.
Source
Target
F1
EM
Ace
We also report the performance of our verifier with regards to FI, recall, and precision scores in Figure 3, while varying the number of rectifying steps. In particular, precision denotes the propor- tion of the correct verification out of all verification predicted as correct; meanwhile, recall evaluates the proportion of the correctly predicted verifica- tion out of all actual correct verification. As shown in Figure 3, recall and Fl scores reach their almost highest points around two to three rectifying steps, while precision scores decrease slightly. These re- sults suggest that, by increasing the number of rec- tifying steps, the coverage of our KALMV in deliv- ering correct answers (i.e., recall) increases much, albeit with a slight compromise in the proportion of correct answers delivered (i.e., precision).
Natural Questions HotpoiQA
Natural Questions
52.98
42.36
50.43
Natural Questions
56.26
46.70
53.02
HotpoiQA
HorpoiQA
64.06
52.31
55.84
Natural Questions
HorpotQA
55.08
42.17
45.56
WebQSP
WebQSP WebQSP
74.31
63.92
77.78
Mintaka
69.86
60.00
72.47
Mintaka
Mintaka
$9.29
51.52
59.13
WebQSPP
Mintaka
48.06
40.25
of retrievers. Also, for sensitive analyses on ver- ification and generation, we further include Chat- GPT (OpenAI, 2022) as a reference model to under- stand the proprietary model's performance, Regard- ing verification, observe that FL.NNbbaedd instruction-finetueed verifier is superior to the Chat- GPT (Peng et al., 2023), which suggests that cus- tomizing the available LM to our target verification task with further training is more worthwhile than using the general-purpose large LMs. Moreover, for generation LMs that make answers to the given questions, large LMs obviously outperform the per- formance of relatively small LMs, since large LMs might be more skilled and knowledgeable in an- swering questions. Note that our KALMV can ac- curately identify the errors even when coupled with ChatGPT as well as the other instruction finetumed TO (Sanh et al., 2022), confirming its versatility.
Please note that we also provide the case study on the three verification categories in Table 7.
Ablation & Sensitive Analyses To see how much our ensemble strategy contributes to the per- formance gain, and also how sensitive the com- ponents in KALMV are across different models, we perform ablation and sensitive analyses on en- semble, retrieval, verification, and generation parts. First, as shown in the first row of Table 3, ensemble, which forwards multiple verification instructions to the verifier and averages their results, improves the performance of both the verification and answer generation steps, demonstrating its efficacy.
instruetion-iinttueed
A nalysss on Generalization Unseen Data It is worthwhile noting that our KALMV can be di- rectly applicable to other datasets without any fur- ther training on them. To show this, we first train the verifier of KALMV on the source data (e.g., Natural Questions) and then evaluate KALMV on the target data (c.g., HotpotQA), with FLAN Base used as the LM for generation and verification. As shown in Table 4, we observe that our KALMV has the capacity to generalize to other data without much performance degradation. Furthermore, for the Natural Questions dataset, the verifier trained
For sensitive analyses, we first change the knowl- edge retriever for open apenndomminn QA from the sparse (BM25) to the dense (DPR) retriever (Karpukhin al., 2020). As shown in Table 3, while the dense retriever further brings performance improvement against the sparse retriever on most metrics, our KALMV consistently detects errors of knowledge- augmented LMs with high performance regardless

---

on the HotpotQA might be stronger than the ver- ifier trained on the same Natural Questions, from the observation of the KALMV's performances on Natural Questions from models trained on each of HopotQA and Natrual Questions datasets, which further signifies its generalization ability.
of detecting errors of knowledge-augmmntee LMs in two different perspectives: knowledge retrieval and answer generation. However, each component and strategy of the proposed KALMV method a bit separated. Specifically, the retriever and verifier are not jointly trained, while the signal from train- ing the verifier may help improve the retriever's performance, Also, regarding the enror rectifying steps, while wwwwledge-augmenten LMs, the previous and cur- rent rectifying steps are handled separately. How- ever, the current step may get benefits from the results the previous steps. We leave developing and building more ideas on improving components of our proposed KALMV method as future work.
6 Conclusion
iteratively correct failures on
In this work, we proposed Ghowledge-Augmente Language Model Verification (KALMV), which identifies not only the relevance of the retrieved knowledge to the input query but also the faithful- ness of the reflection of knowledge in the generated answers, in order to prevent incorrect answer gen- crations with end, we developed a verifier that can detect errors in both the knowledge retrieval and answer genera- tion stages by instruction- firtriuiinninnuinnggggg LMs. Further, during inference, we proposed to rectify errors by re-retrieving knowledge and re-generating answers if our KALMV detects errors, and also perform an ensemble over multiple verification outputs from different instructions, to improve the efficacy the verifier. We validated KALMV on two question answering tasks and showed its effectiveness in sig- nificantly reducing hallucinations. We believe that KALMV will bring substantial practical impact in improving the reliability of LM-hased systems, especially since it is plug-and-play module.
LMs. To this
Ethics Statement
Hallucination, which is phenomenon where the language models generate responses that are plau- sible and sound yet factually incorrect, is critical problem especially when deploying LMs in pro- duction since it can induce the spreading of misin- formation. In this work, the proposed knowledge- augmented language model verification (KALMV) method contributes to significantly reducing hal- lucinations of LMs, by verifying their retrieved knowledge and generated answers, and further rec- tifying them if errors are detected. However, there may be some cases where our verifier misclassifies the failure cases of mnowledge-augmente LMs as correct, potentially leading to severe negative con- sequences, especially mission-critical domains and systems. Therefore, it is important for us to put more effort into making LMs more reliable and trustworthy with advanced verification methods.
Limitations
In this section, we faithfully discuss the current lim- itations and potential avenues for future research. First, we propose to instruction-iiettune the veri- fier LM to customize it to the proposed verification task that aims to detect errors in knowledge re- trieval and answer generation steps. Then, through our experimental results and analyses, we show that our proposed verifier trained by the automatically generated input-output pairs (See Section 3.2) is effective in identifying errors. However, the auto- matic label-generation processes that we suggest are indeed simple and they may introduce the po- tential to incorrectly generate the verification label in some particular scenarios (e.g., multi-step rea- soning with multiple sources of knowledge). There- fore, someone may improve the labels required for cntrrcctionfinntuniii verifiers by annotating them manually with humans or designing more sophisti- cated strategies, which we leave as future work. Second, our work initiates a new problem setup
Acknowledgements
This work supported by the Institute of Infor- mation & communications Technology Planning & Evaluation (IITP) grant funded by the Korea gov- ernment (MSIT) (No. 2019-0-00075, Artificial In- telligence Graduate School Program (KAIST) and No. RS-2022-00187225 Korean Language Model Technology for Efficient Pre-training), the National Research Foundation of Korea (NRF) grant funded by the Korca gov- ernment (MSIT) (No. RS-2023-00256259, and the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korea Government (MSIT) (NRF- 2018RI I ASA 1059921).
Development of Large

---

References
Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemer, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NeurIPS.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa. Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier- Heller-Hllls,emm, Gau- rav Mishra, Erica Moreira. Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernández Abrego. Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks. Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave. Mostafa Dehghani, Sunipa Dev. Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vladimir Feinberg, Fangxi- anyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, and et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305. 10403.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham. Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi. Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov. Ryan Sepassi. David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaramarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Mor- eira, Rewon Child, Oleksandr Polozov, Katherine Lee. Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways. arXiv preprint xXXv:2204.023111.
Jinheon Back, Alham Fikri Aji, and Amir Saffari. 2023. Knowledge-augmentt language model prompting for zero-shot knowledge graph question answering. arXiv preprint arXiv:2336.0411 0336...66....
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji. Tiezheng Yu. Willy Chung, Quyet V. Do. Yan Xu, and Pascale Fung. 2023. A multitask, multilin- gual, multimodal evaluation of chatgpt on reason- ing. hallucination, and interactivity, arXiv preprint arXiv2230..44033.
Hyung Won Chung. Le Hou, Shayne Longpre, Barret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery. Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi. Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instructionfinetuuee language models. arXiv preprint arXiv:2200.1
Jonathan Berant, Andrew Chou, Roy Frostig, and Perey Liang. 2013. Semantic parsing on freebase from guesti pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2013, 18-21 October 2013, Grand Hyull Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL. ACL.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean- caan-Bapisste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy. Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore. Chris Jones. Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals. Simon Osindero. Karen Si- monyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. 2022. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Bal- limore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 2206-2244 PMLR.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In NACCI. Association for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann N. Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguisties, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers. pages 889-898. Association for Computational Lin- guistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Vosss, Gretchen Krueger, Tom Henighan, Rewon Child,
Jiazhan Feng. Chongyang Tao, Xiubo Geng, Tao Shen, Can Xu, Guodong Long, Dongyan Zhao, and Daxin Jiang. 2023. Knowledge refinement via interaction between search engines and large language models. arXiv preprint arXiv:2305.07400

---

Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, N. Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023, Rarr: Researching and revising what language models say, using language models. In ACI. 2023. Association for Computational Linguis- tics.
prompting for open-domain question answering. arXiv preprint arXiv:2203.01155.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguisties, pages 6086-6096, Florence, Italy. Association for Computational Linguisties.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujuu Yang, Nan Duan, and Weizhu Chen. 2023. CRITIC: large language models can self-correct with tool-interactive critiquing. arXiv preprint 230.::3005.1738.
Patrick S. H. Lewis, Eihan Perez, Aleksandra Pik- tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kütder, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval- Retrieval-uggmented -uggmented generation for wwwwlede-intensiin NLP tasks. In Advances in Neu- ral Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. 2020. Retrieval augmented language model pre- trainirinig. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 9929-3938. PMLR.
Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix X. Yu. and Sanjiv Kumar. 2022. Large language models with controllable working memory. arXiv preprint arXiv:2211.05110
Gautier Izacard, Patrick S. H. Lewis, Maria Lomeli, Lucas Hosseimi, Fabio Petroni, Timo Schick, Jane Dwivedi-YY Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models. CoRR, abs/2208.03299.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint 1rXiv:1907.11699
Zhengbao Jiang, Frank Xu. Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Ac- tive retrieval augmented generation. arXiv preprint arXiv:2305.0698:
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Minki Kang, Jinheon Back, and Sung Ju Hwang. 2022. KALA: srowledge-augmenter language model adap- tation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL 2022, Seatte, WA, United States, July 15, 2022, pages 5144-5166 Association for Computational Linguistics.
Yao La, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In ACL. Association for Computational Linguistics.
Vladimir Karpukhin. Barlas Oguz. Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing, EMNLP 2020, November 16-20, 2020. Association for Computational Linguis- tics.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao. Sarah Wiegreffe, Uri Alon. Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Scan Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh,, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint 2333
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:452-466.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannanch Hajishirzi, and Daniel Khashabi. 2023. When not trust language models: Investigating effectiveness and limitations of parametric and non- parametric memories. In ACL 2023. Association for Computational Linguistics.
OpenAL. 2022. Introducing chatgpt. https://openai. com/blog/ chatgpt.
commbbog/chattpt..
Angeliki Lazaridou, Elena Gribovskaya, Wojeiech Stokowiee, and Nikolai Grigorev. 2022. Internet- augmented language models through few-shot
OpenAL 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08777.

---

Baolin Peng. Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang. Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feed- back. arXiv preprint arXiv:2302.
Empirical Methods in Natural Language Processing, EMNLP 2021 / Punta Cana, Dominican Republic, 7-11 November, 2021. Association for Computational Linguisties.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafcai, Antoine Chaffin, Arnaud Stiegler, Arun Raja. Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma. Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang. Mattco Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language mod- els knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods Natu- val Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNI.P-IICNI.P 2019, Hong Kong, China, Novem- ber 3-7, 2019. Association for Computational Lin- guistics.
Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang. Maosong Sun, and Jie Zhou. 2021. ERICA: improving entity and rela- tion understanding for pre-trained language models via contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Confer- ence on Natural Language Processing, ACU/JUNNP 2021, (Volume 1: Long Papers), Virtual Event, Au- gust 1-6, 2021, pages 3350-3363. Association for Computational Linguistics.
Priyanka Sen, Alham Fikri Aji, and Amir Saffari. 2022. Mintaka: A complex, natural, and multilin- gual dataset for end-to-end question answering. In COLING. International Committee on Computational Linguistics.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Min- joon Sco, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. 2023. REPLUG: retrieval- augmented black-box language models. arXiv preprint arXiv:2301.12652.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language under- standing by generative pre-training.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn, Res.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguisties: EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 16-20 November, 2021, pages 3784- 3803. Association for Computational I. Linguistics.
Adam Roberts. Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parame- ters of language model? In EMNLP.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu, 2020. Mpnet: Masked and permuted pre- training for language understanding. In NewrIPS.
Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock Beaullee, and Mike Gatford. 1994. Okapi at TREC-3. In Proceedings The Third Text REtrieval Conference, TREC 1994, Gaithers- burg, Maryland, USA, November 2-4, 1994, volume 500-225 of NIST Special Publication, pages 1099 126. National Institute of Standards and Technology (NIST).
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv preprint 2r32 13977.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018 Object hallu- cination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018. Association for Computational Linguistics.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Interleav- ing retrieval with chain-of reasoning for knowledge-intenn -intensiiie multi-step questions. arXiv preprint arXiv:2212. 10509.
Amir Saffari, Armin Oliya, Priyanka Sen, and Tom Ayoola. 2021. End-to-end entity resolution and question answering using differentiable knowledge graphs. In Proceedings of the 2021 Conference on
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoe V. Le. 2022. Finetuned language models are zero-shot learners. In The Tenth

---

International Conference on Learning Representa- tions, ICLR 2022, Virtual Event, April 25-29, 2022.
Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-seng Chua. 2023. Search-in-the-cher To- wards the accurate, credible and traceable content generation for complex knowledge-intensivy tasks. arXiv preprint 2rXiv:2304. 14732.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: deep con- textualized entity representations with entity-aware sclf-attention. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6442-6454. Association for Computa- tional Linguisties.
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop hopii-hoppo question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Com- putational Linguistics.
Wen-tau Yih, Matthew Richardson, Christopher Meek. Ming- Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In ACL. The Association for Computer Linguistics.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: enhanced language representation with informative entities. In Proceedings of the 57th Conference of the Associa- tion for Computational Linguistics, ACL 2019, Flo- rence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pages 1441-144 1451. Association for Computa- tional Linguistics.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In ICML, Proceedings Machine Learning Research. PMLR.
Shen Zheng. Jie Huang, and Kevin Chen-Chhan Chang. 2023. Why does chatgpt fall short in providing truth- ful answers? arXiv preprint 2r0iv:2304. 10513.
Chunting Zhou, Junxian He. Xuezhe Ma. Taylor Berg- Kirkpatrick, and Graham Neubig. 2022. Prompt con- sistency for zero-shot task generalization. In Find- ings the the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, De- cember 7-11, 2022, pages 2613-2626. Association for Computational Linguistics.

---

Table Relative Increment of Computational Costs, which the verifter of our KALMV approach additionally yields, com- pared to the model snooledgg--uumenteer LMs with knowl- edge retrieval and answer generation) without the verification.
B Additional Experimental Results
B.1 Verification Cost
As it is worthwhile to investigate the increment of computational costs incurred by answer verification of our KALMV compared to the one without veri- fication, we measure the relative increment in costs that our verifier additionally brings compared to the whole costs of running base knowlcdge-augmeent LMs, and report it in Table 5. In particular, fol- lowing the main experiment settings, we use the FLAN Base (250M) as the verification LM and use three different sizes of FLAN: Base (250M), Large (780M), and XL (3B), the generation LM. Also, we set the cost of knowledge retrieval and answer generation (e.g.. cost of running the entire coowledge-uugeented LMs) as 100, and then re- port the relative increment from using the proposed verification. As shown in Table 5, our KALMV yields only the marginal increment, since not only do we use the smaller LM (Base) compared against larger LMs (Large and XL) for verification, but also the proposed verification LMM generates only token (e.g., A. B, or C) unlike the generation LM that decodes multiple tokens. For example, verify- ing answers with KALMV 34 times faster than generating answers with Flan XL on the WebQSP data, which suggests that ours is highly efficient. Yet, each rectifying step of our KALMV method incurs a cost that is approximately equivalent to the LMs
Datasets
Base
Large
XI.
WebQSP
5.60%
3,40%
3.07%
Mintaka
6.07%
3.47%
3.57%
Natural Questions
10.51%
9.26%
6.54%
HotpotQA
5.02%
4.24%
3.67%
A Additional Experimental Setups
Here we provide additional experimental setups, in- cluding the instruction that we use for verification.
Instruction Prompt In Table 6, we provide set of 5 different instructions that we use for verifi- cation ensemble well as instruction prrrution-finetunnng finetuning verifiers (Please refer Section 3.2 for details).
LLM-Augmenter Details In our experiments in Section 5, we include this LLM-Augmenter model as our major baseline (Peng et al., 2023), and we now describe
more detail. Note that the main focus of this baseline is to verify whether the gener- ated answers from large LMs are grounded in the re- trieved knowledge, and they propose two strategies to identify the groundedness. Specifically, the first strategy is the one that measures the Knowledge Fl score between the retrieved knowledge and the gen- erated answer, which we already used for compar- isons against our KALMV in our main experiments. On the other hand, the second strategy is to ask pro- prietary LMs (e.g.. ChatGPT) to verify the ground- edness of the generated answer in the retrieved knowledge. However, for the second one, it is in- feasible for us to run every experiment with private Large LMs, and also it is clearly unfair to compare the public LMs against the proprietary LMs since their training data and capacity may be largely dif- ferent. Nevertheless, we show that our KALMV superior to L.gM-uugeeeter with ChatGPT on ver- ification and answer generation in Table 3. More- over, LLM--ugmenter with ChatGPT is known to have similar performances to the one that we compare (i.e, LLM-Augmenter w/ Knowledge FI) according to Peng et al. (2023), which may further support the fact that our KALMV is more effective on verification compared to the -uggmnnte based on ChatGPT since our KALMV significantly outperforms LLM-Augmenter w/ Knowledge FI.
cost of running entire with verification. To be specific, let's assume that, through the KALMV framework, the error in the generated answer is identified, the rectifying step is subsequently performed, and the new answer ver- ified as correct. Then, it takes twice as slow as the model without rectification. Yet, fortunately, since not every generated answer is verified as incorrect, the number of samples that require rectifying steps is far less than the number of all samples (c.g.. only 38% of samples require rectification on WebQSP).
B.2 Case Study
In Table 7, we provide examples of our KALMV framework on three verification categories: incor- reet knowledge retrieval, incorrect answer genera- tion, and correct answer generation, on knowledge- augmented LMs. As shown in Table 7, KALMV can detect the errors of anugeeeee-augmented LMs by contextualizing and understanding the relation- ships between the input question, retrieved knowl- edge, and generated answer effectively.

---

Table 6: A list of instructions that we use for verification with ensemble as well as for instruction-finett verifiers, Note that the variable inside the set parentheses {} is replaced with actual string (e.g. input question, knowledge, and generated output).
Indices
Instructions
The following multiple choice question about question answering task. In this task, you should generate an output given question with passage. The passage is retrieved from Wikipedia, which may may not be helpful answer the question.
Question: {question]
Passage: {passage}
Output: (answer)
Options:
The passage unhelpful to answer the question.
B. The passage is helpful to answer the question, yet the generated output for the question is incorrect.
C. The generated output for the question is correct.
Select one option:
2 Question: {question)
Passage: {passage}
Output: {answer]
Options:
passage unhelpful to answer the question.
B. The passage is helpful to answer the question, yet the generated output for the question is incorrect.
C. The generated output for the question is correct.
Select one option:
3
Given question and passage from Wikipedia, should generate output as follows:
Question: {question}
Passage: {passage}
Output: (answer)
This multiple choice question, and, based on the above information, you need select one option among three, as follows:
A. The passage is unhelpful to answer the question.
The passage helpful answer the question, yet the generated output the question incorrect.
C. The generated output for the question is correct.
Select one option:
Here is question, passage, and generated output from the question and passage. Based on them, you need to select one option among the three.
Question: {question}
Passage: {passage}
Output: {answer)
Options:
passage unhelpful to answer the question.
B. The passage is helpful to answer the question, yet the generated output for the question is incorrect.
C. The generated output for the question is correct.
Seleet one option:
5
Given question, passage, and output, which option the best?
Question: (question)
Passage: {passage)
Output: {answer)
Options:
A. The passage is unhelpful to answer the question.
B. The passage is helpful to answer the question, yet the generated output for the question is incorrect. generated output for the question is correct.
Select one option:

---

Table 7: Examples of three types of verification outputs, such as retrieval error, generation etror, and correct answer of knowledge- augmented LMs, determined by our KALMV on the Natural Question dataset with FLAN Base as the generation LM.
Types Retrieval Error
Examples
Question: who sang the song good morning good morning?
Knowledge: Good Morning Call
Correct answers: ("Gene Kelly'. "Donald O'Connor", "Judy Garland', 'Debbie Reynolds', 'Mickey
Rooney'1
Generated answer: The Beatles
Retrieval Error
Question: when did taylor swift's first album release?
Knowledge: 1989 the fifth studio album by American singer-songwrite Taylor Swift released on October 27, 2014, through Big Machine Records. Swift began composing the album following release of previous studio effort, Red (2012). Over the course of the two-year songwriting period, she collaborated with producers Max Martin and Shellbackāe" Martin served as album's executive producer alongside Swift. The album's 's title was named after the singer's birth year and inspired by the pop music of the 1980s.
Correct answers: ['October 24, 2006', '2005']
Generated answer: October 27, 2014
Retrieval Error
Question: who sang ran all the way home?
Knowledge: In 2007, the song was covered by Paul McCartney who sung it, and Allen Toussaint playing the piano, their contribution Goin" Home: A Tribute Fats Domino (Vanguard).
Correct answers: The Impalas']
Generated answer: Paul McCartney
Generation Error
Question: what is the hot coffee mod in andreas?
Knowledge: Hot Coffee a normally inaccessible mini-gama gani- in the 2004 video game Grand Theft Auto: San Andreas, developed by Rockstar North. Public awareness of the existence of the mini game arrived with the release of the Hot Coffee mod, created for the Microsoft Windows port of GTA:: San Andreas in 2005. This mod enahles access to the mini-gama
Correct answers: ('a normally inaccessible mini-game"]
Generated answer: euables
the mini-game
Generation Error
Question: where does a roadrunner live in the desert?
Knowledge: Roadrunners inhabit the deserts of the southwestern United States, Mexico, and Central America. They live arid lowland or mountainous shrubland, widely dispersed in dry open country with scattered hrush. They are non-migratory, staying in their breeding area ycar-round.] [10] The greater roadrunner is not currently considered threatened in the US, but is limitet- Correct answers: ['mountainous shrubland", 'arid lowland') Generated answer: United States
Generation Error
Question: what angle the
pisa?
Knowledge: The height of the tower is 55.8 metres (183.27 feet) from the ground on the low side and 56.67 metres (185.93 feet) the high side. The width of the walls at the base is 2.44 m (8 ft 0.06 in). Its weight is estimated 14,500 metric
(16,000 short tons). The seventh floor has two fewer steps on the north-facing staircase. Prior to restoration work performed between 1 1990 and 2001, the tower leaned at an angle of 5.5 degrees, but the tower now leans at about 3.99 degrees. This means the top the is displaced horizontally 3.9 metres (12 10 from the centre.
has 296 294 steps; the
Correct answers: ('about 3.99 degrees'. '3.99 degrees', 'at about 3,99 degrees'I
Generated answer: 5.5 degrees
Correct Answer
Question: who wrote he ain't ain't heavy he's brother lyrics?
Knowledge: "He Ain't Heavy, He's se's My Brother" is popular music ballad written by Bobby Scott and Bob Russell. Originally recorded by Kelly Gordon in 1969, the song became worldwide for The Hollies later that year and again for Neil Diamond in 1970. It has been covered by many artists in subsequent years. The Hollies" and Rufus Wainwright's versions of the song featured in the film Zoolander.
Correct answers: l'Bobyy Scott', BBob Russell]]
Generated answer: Bobby Scott and Bob Russell
Correct Answer
Question: who is the lead singer of depeche mode?
Knowledge: David Gahan (bom David Callcott; 9 May 1962) is an English singer-songwriter, best known the baritone lead singer
electronie band Depeche Mode since their debut 1980. He is also a successful solo artist, relcasing albums in 2003 (Paper Monsters) and 2007 (Hourglass).
Correct answers: ('David Gahan')
Generated David Gahan
Continued the next page

---

Table Continued from the previous page
Types
Examples
Correct Answer
Question: when was the first hunger games book published?
Knowledge: The Hunger Games was first published in hardcover on September 2008, by Scholastic, featuring a cover designed by Tim O' O'Brien. It has since been released in paperback and also as an audiohook and chook. After initial print of 200,000, the hook had sold 800,000 copies by Fehruary 2010. Since its release, The Hunger Games been translated into 26 languages, and publishing rights have been sold in 38A territories. The novel the first in The Hunger Games trilogy, followed by Catching Fire (2009) and Mockingjay (2010). A film adaptation, directed by Gary Ross and co-written and co- co-produced Collins herself, was released 2012. Correct answers: ['September 14, 2008", '2008"] Generated answer: Septemher 14, 2008


Reviews:
[{"review_id": "k8x3XjQ8i6", "reviewer": "Reviewer_Fsm3", "summary": "", "questions": "1. I’m not quite sure what it means to evaluate the verification accuracy of generation LMs and generation accuracy of verification LMs (Table 3). Can you please clarify how this works and what the takeaway from these numbers is supposed to be?\n2. “Therefore, we design various instructions, forward them to our single verifier, and ensemble the multiple outputs from the verifier with average” (L392-394). Are you instruction-fine-tuning with each of these five different instructions?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive and helpful comments. During the response period, we have made every effort to faithfully address all your comments below.\n\n---\n\n> **Weakness 1.1:** The analysis in Figure 3 (i.e., varying the number of rectifying steps for verification) could be improved, by reporting the performance (e.g., accuracy, F1, precision) of generated answers instead of reporting the verification performance. \n\nWe would like to clarify that, in Figure 3, **precision already captures the answer generation performance** (i.e., precision denotes the proportion of correctly generated outputs out of all generated outputs that are predicted as correct). On the other hand, **recall captures the answer verification performance** (i.e., recall denotes the proportion of correctly verified and correctly generated outputs out of all outputs that are labeled as correct). Therefore, Figure 3 shows both the answer generation and the answer verification performances.\n\n---\n\n> **Weakness 1.2:** I’m not quite sure about the takeaway from Figure 3.\n\nWe apologize for the confusion. In order to understand the key takeaway of Figure 3, we need to jointly look at the results on both precision and recall. As explained in the response to Weakness 1.1, precision captures the answer generation accuracy; meanwhile, recall captures the coverage of the knowledge-augmented LMs in delivering the correct outputs. Therefore, if the precision is low and the recall is high, the number of times that the user receives the correct outputs (recall) is increased, while the proportion of correct outputs on all the received outputs (precision) is decreased, i.e., the coverage is high yet the reliability is low. \n\nRegarding the takeaway from Figure 3, when increasing the number of rectifying steps, the recall tends to increase while the precision tends to decrease, which suggests that our KALMV can provide more numbers of correct responses to the user while the user may experience reduced reliability. Thus, **Figure 3 shows the trade-off between the coverage and the reliability with respect to the number of rectifying steps**, and perhaps can further suggest that using 1 or 2 rectifying steps may offer the optimal balance between them.\n\n---\n\n> **Weakness 1.3:** It seems that, in Figure 3, reporting the verifier performance as the function of rectifying steps does not show something that is fundamentally changing. \n\nPlease see our responses to Weakness 1.1 and Weakness 1.2. The performance of the verifier on precision and recall with respect to the number of rectifying steps shows the fundamental changes in model behaviors (e.g., the trade-off between the reliability and the coverage), beyond the point that it can rectify the retrieval and generation errors. \n\n---\n\n> **Weakness 1.4:** In Figure 3, does the end-answer recall peak at 2 rectifying steps? Why does it drop after 2?\n\nAs shown in Figure 3, the end-answer recall drops after 2 rectifying steps on most datasets. This is because a large portion of errors comes from retrieval (Figure 2) and the retrieved knowledge becomes less relevant to the input query after several rectification steps (e.g., we should retrieve and use the top-5 ranked knowledge if retrieval fails 4 times), making more incorrect answers. Consequently, when the number of rectifying steps is large, since more numbers of incorrect answers may be filtered out by our verifier due to their noise and fundamental errors, the recall (coverage) may decrease after 2 rectifying steps.\n\n---\n\n> **Weakness 2:** I’m not sure whether including the adaptive retrieval (Mallen et al., 2023) makes sense as a strong baseline since it does not perform retrieval for every query, which seems likely to be worse than knowledge-augmented LMs that always perform retrieval. \n\nWe include this adaptive retrieval method (Mallen et al., 2023) as a baseline, since it indeed contributes to improving the performance of knowledge-augmented LMs by adaptively retrieving the knowledge. In other words, this work shares a similar focus to our work that aims at improving the knowledge-augmented LMs while exploring a different approach - adaptive retrieval, unlike our approach to verification. Therefore, for this shared reason for improving the knowledge-augmented LMs, we consider adaptive retrieval as a fair and valid baseline.\n\nSpecifically, as shown in Figure 1 of adaptive retrieval (Mallen et al., 2023), it outperforms the naive knowledge-augmented LMs when the sizes of LMs are relatively large, which confirms that it is a strong baseline. Also, in Table 1 of our work, we also observe that the adaptive retrieval baseline consistently outperforms the naive knowledge-augmented LMs on Flan XL (3B) in most settings; again, the adaptive retrieval models serve as clear baselines. However, based on the comparison between this adaptive retrieval and our verification method in Table 1, both of which aim at enhancing knowledge-augmented LMs, we demonstrate that our verification strategy is superior to adaptive retrieval in performance, which further signifies the importance of our verification for the knowledge-augmented LMs.\n\n---\n\n> **Weakness 3.1:** One drawback with applying the proposed KALMV in practice is the relative expense of having to verify and rectify errors at each step along with both retrieval and answer generation, which is not necessarily an indictment though since there are arguably settings where factuality matters. \n\nAs you pointed out, we strongly believe that, in many practical applications (e.g., mission-critical domains such as biomedicine), **providing factually correct answers in response to user’s queries might be much more beneficial at the expense of using some computational costs** than providing unreliable answers promptly without verification. Also, please note that, while we use extra costs for verifying answers and rectifying errors (See response to Weakness 3.2 below), the proposed KALMV incomparably outperforms existing knowledge-augmented LMs without verification, as well as we use the smaller LM (250M) for verification (i.e., the verification cost is indeed marginal compared to answer generation).\n\n---\n\n> **Weakness 3.2:** The computational expense of the proposed KALMV on answer generation and verification can be further quantified, for future reference.\n\nThank you for your suggestion. We additionally measure the relative increment in the computational costs for answer verification, compared to the model without verification.\n\nIn particular, following the main experiment settings in our work, we use the FLAN base (250M) as the verification LM and use three different variants: Base (250M), Large (780M), and XL (3B), as the generation LM. Also, we set the cost of knowledge retrieval and answer generation (e.g., cost of running the entire knowledge-augmented LMs) as 100, and then report the relative computational cost increment from using the proposed verification. As shown in Table A.1, our KALMV brings only the marginal computational cost increment, since not only do we use the smaller LM (Base) compared against larger LMs (Large and XL) for verification, but also the proposed verification LM generates only one token (See Lines 352 - 359) unlike the generation LM that decodes multiple tokens. For example, our verification model is 34 times faster than generating the answer with Flan XL on the WebQSP dataset, which suggests that our proposed verification method is highly efficient. \n\nTable A.1: Relative computational cost on verification against knowledge retrieval and answer generation of knowledge-augmented LMs, where we use the Flan Base for verification.\n|Data|Base|Large|XL|\n|---|---|---|---|\n|WebQSP|5.60%|3.40%|3.07%|\n|Mintaka|6.07%|3.47%|3.57%|\n|Natural Questions|10.51%|9.26%|6.54%|\n|HotpotQA|5.02%|4.24%|3.67%|\n\nOn the other hand, each rectifying step incurs a cost approximately equivalent to that of performing initial knowledge-augmented LMs with verification. For instance, if the error is identified, the rectifying step is performed, and the subsequent answer is verified as correct, it takes twice as long. Fortunately, since not every generated answer is verified as incorrect, the number of samples that require rectifying steps is far less than the number of all samples (e.g., only 38% of samples require rectification on WebQSP). We will include the results and discussions on cost in the revision.\n\n---\n\n\n> **Question 1.1:** I’m not quite sure what it means to evaluate the verification accuracy of generation LMs and generation accuracy of verification LMs (Table 3). Can you clarify it?\n\nWe apologize for the confusion. The leftmost column in Table 3 represents the specific category for analyses. For example, the verification accuracy of the “Generation LMs” category denotes the accuracy of the verification LM where the generation LM is selected among T0 (3B), FLAN (3B), and ChatGPT. Similarly, the generation accuracy of the “Verification LMs” category denotes the accuracy of the generation LM where the verification LM varies across T5 (250M), FLAN (250M), and ChatGPT. \n\n---\n\n> **Question 1.2:** What is the takeaway from the results in Table 3?\n\nThe main purpose of experiments in Table 3 is to vary the models on each category, in order to perform either ablation studies or sensitivity analyses on each component of our KALMV. For example, in the “Verification LMs” category, we validate the verification performances of three different models, such as T5 (250M), FLAN (250M), and ChatGPT, and then show that the instruction-finetuned tailorable LM (FLAN) is superior to ChatGPT on verification.\n\n\n---\n\n> **Question 2:** Do you instruction-finetuning the model with each of these five different instructions (Lines 392 - 394)?\n\nYes, your understanding is correct. Specifically, during training, we instruction-finetune the single verifier with each of the five verification instructions. Then, during inference, we forward those five instructions to the trained verifier and ensemble their results to produce the output.\n\n---\n\n> **Typo:** In Lines 584-590, Figure 2 → Figure 3.\n\nThank you for correcting it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written.\n2. The method presented in the paper is simple to understand and empirically effective, demonstrating significant improvement across 4 domains compared to multiple retrieval-based baselines without verification.\n3. Detailed analysis allows drawing further insight into the source of the gains: e.g. most retrieval errors come from retrieving irrelevant facts/documents, ensembling results of different verifier prompts is useful, etc.\n", "reasons_to_reject": "1. The analysis could be improved with a plot of rectifying steps vs. end-answer accuracy / F1 / precision / recall. Currently I’m not quite sure what the takeaway is supposed to be from the current plots (Fig. 3) of rectifying steps vs. verifier performance. It seems that verifier performance as a function of rectifying steps is likely more a function of base retrieval/answer accuracy after those steps, rather than something fundamentally changing with the verifier itself. Does end-answer recall peak after 2 rectifying steps? Why does it drop after 2 rectifying steps?\n2. I’m not sure if including adaptive retrieval (L479 / Table 1) makes sense as a strong baseline here. It seems likely to be strictly worse than knowledge-augmented LMs as it is not retrieving for every query (indeed this seems supported by empirical results in Table 1) – its usage seems to be predominantly in improving efficiency of retrieval LMs by allowing the LM not to have to retrieve for every query. \n3. The one drawback I can imagine with applying this technique in practice is the relative expense of having to verify and rectify errors at each step, following both retrieval and answer generation. This is not necessarily an indictment of this technique in general, as there are arguably settings where factuality matters above all, regardless of expense. However, perhaps the computational expense of this technique can be quantified for future reference.\n", "questions_to_authors": "1. I’m not quite sure what it means to evaluate the verification accuracy of generation LMs and generation accuracy of verification LMs (Table 3). Can you please clarify how this works and what the takeaway from these numbers is supposed to be?\n2. “Therefore, we design various instructions, forward them to our single verifier, and ensemble the multiple outputs from the verifier with average” (L392-394). Are you instruction-fine-tuning with each of these five different instructions?\n", "paper_topic_and_main_contributions": "This paper introduces a small, tailorable LM verifier to improve factuality of knowledge-augment LMs, particularly in checking whether the LM: 1. retrieves the relevant knowledge, 2. incorporates the retrieved knowledge appropriately in its generations. The verifier is trained via instruction fine-tuning to perform both types of verification. Evaluated on 4 datasets (Natural Questions, HotpotQA, WebQSP, Mintaka), the method in the paper (KALMV) significantly outperforms retrieval-based LMs without verification.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L584-590: Figure 2 → Figure 3\n"}, "extra_text": {}}, {"review_id": "QC1QymLZjq", "reviewer": "Reviewer_QSUP", "summary": "", "questions": "- Is there any kind of guarantee the answers having correct integration of retrieved results present in the Top k answer without touching upon the core mechanism of model generation? A more convincing approach is to further ensure the groundness of the next generation after a generating error show up.\n- The paper claims that the input-output pair is automatically constructed, how the label of groundedness of the augmented answer was assigned automatically? This is the point that needs to be clarify.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive and helpful comments. During the response period, we have made every effort to faithfully address all your comments below.\n\n---\n\n> **Weakness 1.1:** Is there any kind of guarantee that the answers, having correct integration of retrieved results, present in the Top-K answers without touching upon the core mechanism of model generation? \n\nWhile it is not always guaranteed that one of the Top-K answers includes the correct knowledge retrieved from the KB, iteratively generating the answers until making the correct one by using Top-K sampling is still highly effective and efficient. Specifically, ensuring that the answer is grounded in the correct knowledge is not straightforward and rather a substantially difficult task, since it not only requires the verifier to check whether the generator uses one of the retrieved documents/facts but also to identify whether the generator selects only the correct portion of the retrieved document among all the other information. Therefore, we simply sample one candidate answer among the Top-K answers at every rectifying step, and, by increasing the number of K (i.e., increasing the answer space), the model may generate the answer with a sufficient number of rectifying steps.\n\nAlso, while touching upon the core mechanism of model generation to further ensure the grounding of the generated answer in the retrieved knowledge is not the scope of our work (i.e., our target is on building a post hoc verifier rather than improving the generator) and it might be a promising avenue for future work, it has a critical drawback. In particular, one advantage of our verification method is that we only use the input and output pair (e.g., question, retrieved knowledge, and generated answer) of the answer generator, which allows us to use even the proprietary LLMs that are accessible via API calls. However, the method, which modifies the answer generation strategy, is not applicable to such private LLMs.\n\n---\n\n> **Weakness 1.2:** A more convincing approach is to further ensure the groundness of the next generation after a generation error shows up.\n\nPlease see the response to Weakness 1.1 for detailed answers. In summary, ensuring the groundness of the generated text is an extremely difficult problem since the generator further recognizes and selects the right portion of the retrieved knowledge, which is much more complex than mere copy-and-paste of the retrieved knowledge. Also, for private LMs that are accessible via API calls, we may not explicitly ensure the groundness during the answer generation; meanwhile, our verification approach can ensure it in a post hoc manner.\n\n---\n\n> **Weakness 2:** It is unclear how the label of groundedness of the generated answer was assigned automatically based on the input-output pairs.\n\nAs explained in Lines 374 - 377, **when the generated answer does not have the overlapping tokens with the retrieved correct knowledge**, we label it as a grounding (generation) error. This is because, in such cases, the generation LM does not utilize the retrieved knowledge even if the retrieved knowledge contains the correct answer. Please note that, as discussed in footnote 1 as well as Lines 662 - 678 of the Limitations section, while our automatic label generation strategy works well (Section 5) despite its simplicity, there may exist more sophisticated ways to automatically generate the labels, which we leave as future work.\n\n---\n\n> **Weakness 3:** I did not find the description about whether these baselines use the same generated model, which is important for comparison in the experiment.\n\nWe apologize for the confusion. As described in Section 4.4 (Lines 520 - 531), for all the knowledge-augmented LM baselines and our method, we use the same retriever, the same generator, and the same prompt for augmenting LMs, which we will clarify in the revision.\n\n---\n\n> **Weakness 4.1:** The proposed method is more like expanding the space to choose the answer with the verifier. \n\nYes, one contribution of our work is to continuously expand the answer space by retrieving the relevant knowledge and generating the answer until the knowledge-augmented LM reaches the correct answer, based on our verification and error rectification strategies. \n\n---\n\n> **Weakness 4.2:** There may exist a more integrated and concise structure to build the verification method that can expand the search space, e.g., aligning directly during retrieval and generation to ensure the correctness of each step.\n\nYes, we also believe that there may exist a more improved way of building the verification method. However, our work paves a new avenue for verifying and rectifying two types of errors on knowledge-augmented LMs, which have been underexplored so far; thus, further improving them with additional verification strategies would be promising future work. Also, please note that the suggested direction of aligning the retrieval with the generation may not be applicable to LLMs that are accessible via API calls, since we cannot manipulate the generation process of such LLMs and the retriever may not be jointly trainable with them.\n\n---\n\n> **Question 1 & Question 2:** \n\nPlease refer to our responses for Weakness 1 and Weakness 2 above for our answers to these questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and organized and provides a clear motivation and problem formulation. This work has a simple idea but good results.", "reasons_to_reject": "- Is there any kind of guarantee the answers having correct integration of retrieved results present in the Top k answer without touching upon the core mechanism of model generation? A more convincing approach is to further ensure the groundness of the next generation after a generating error show up.\n- The paper claims that the input-output pair is automatically constructed, how the label of groundedness of the augmented answer was assigned automatically? This is the point that needs to be clarify.\n- And I did not find the description about whether these baselines use the same generated model, which is important of comparison in the experiment.\n- The proposed method is more like expand the space to choose the answer by the help of a trained verifier. So is there a more integreted and concise structure to build this model. For example, aligning directly during retrieval and generation to ensure the correctness of each step.", "questions_to_authors": "- Is there any kind of guarantee the answers having correct integration of retrieved results present in the Top k answer without touching upon the core mechanism of model generation? A more convincing approach is to further ensure the groundness of the next generation after a generating error show up.\n- The paper claims that the input-output pair is automatically constructed, how the label of groundedness of the augmented answer was assigned automatically? This is the point that needs to be clarify.", "paper_topic_and_main_contributions": "This paper proposes a method to verify and correct the retrieval error and generation error of language models that are augmented with external knowledge. The paper also uses an ensemble of different instructions and iteratively corrects the errors until the verifier confirms the correctness of the retrieved knowledge and the generated answer.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UxWyng4MGz", "reviewer": "Reviewer_rWYL", "summary": "", "questions": "It would be helpful to include the state-of-the-art for each dataset (not just limited to the baselines here). While it is understandable to group and compare against similar approaches, it is hard for me to tell how well the model performs when comparing against such a restricted set  of methods", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive and helpful comments. During the response period, we have made every effort to faithfully address all your comments below.\n\n---\n\n> **Weakness 1:** The novelty of the approach appears to be the use of a fine-tuned LM to verify a response against retrieved facts, which is a very straightforward approach. It may have been unexplored for this particular task (i.e., question-answering against a KB) but not overall (i.e., the use of LLMs as verifiers is not new), which makes this work incremental.\n\nOur contribution is not limited to the mere fine-tuning of LMs for response verification, and the proposed output verifications of **knowledge-augmented LMs by considering the knowledge retrieval and grounding errors** are substantially different from existing approaches that verify the outputs of **LMs without considering knowledge augmentation**.\n\nSpecifically, as described in Lines 55 - 68 and Lines 207 - 235, existing verifiers, which include the usage of the external knowledge for verification, suffer from the following two problems: 1) the model may fail to retrieve the relevant knowledge to the query and 2) the generated answer may not be grounded in the retrieved knowledge. However, existing approaches overlook these challenges and rather suppose that there are no retrieval and grounding errors during answer generation. In other words, they focus on validating the generated answer itself by referring to the external knowledge, instead of considering more fundamental problems of the knowledge retrieval and grounding errors. \n\nIn contrast to such existing works, we point out the aforementioned two prevalent yet underexplored challenges of knowledge-augmented LMs (i.e., retrieval and grounding errors) and then tackle them by instruction-finetuning the verifier along with the proposed ensemble and answer rectifying strategies. More technically speaking, existing verification methods use **a pair of the input query and the generated answer** to verify its validity and do not check the retrieval and generation errors even if augmented with external knowledge. Meanwhile, our verification method uses **a triplet of input query, retrieved knowledge, and generated answer** to verify both the retrieval and generation errors with respect to the knowledge.\n\nMoreover, the existing verification method that uses only LLMs without external knowledge is limited in its capacity, as demonstrated in Table 3, since the knowledge internalized in parameters may be incomplete, inaccurate, and outdated (i.e., LLMs cannot store all the world knowledge).\n\nTherefore, we strongly believe that providing such new perspectives on handling retrieval and grounding errors for verifying the knowledge-augmented LMs unlike existing verifiers, and then tackling them with the new training strategies is not an incremental but rather a substantial contribution.\n\n---\n\n> **Weakness 2:** It seems to me that the need to fine-tune the verifier would be quite limiting, as you could then only apply this to datasets for which you had adequate amounts of data.\n\n**Our instruction-finetuned verifier that is trained on a certain dataset is directly generalizable to other datasets even without training on them**, i.e., our verifier is not limited to datasets that are used for verifier training. Specifically, we additionally conduct experiments on transfer learning, where we first train the verifier on the source domain (e.g., Natural Questions) and then generalize it on the target domain (e.g., HotpotQA) without any training samples on the target dataset, with FLAN Base as the generation and verification LMs. \n\nAs shown in Table C.1 below, we observe **that the verifier trained on the source dataset can be directly used for the target dataset without significant performance degradations**. Also, for the Natural Questions dataset, we observe that the verifier trained on the HotpotQA dataset is superior to the verifier trained on the Natural Questions. These results demonstrate that our instruction-finetuned verifier is not limited to the dataset having an adequate amount of training data, and rather is broadly generalizable across different datasets even without training samples on them. We will include the results in the revision.\n\nTable C.1: Results where the verifier is trained on Source data and tested on Target data.\n|Source|Target|F1|EM|Acc|\n|---|---|---|---|---|\n|Natural Questions|Natural Questions|52.98|42.36|50.43|\n|HotpotQA|Natural Questions|56.26|46.70|53.02|\n||\n|HotpotQA|HotpotQA|64.06|52.31|55.84|\n|Natural Questions|HotpotQA|55.08|42.17|45.56|\n||\n|WebQSP|WebQSP|74.31|63.92|77.78|\n|Mintaka|WebQSP|69.86|60.00|72.47|\n||\n|Mintaka|Mintaka|59.29|51.52|59.13|\n|WebQSP|Mintaka|48.06|40.25|46.19|\n\nAdditionally, we would like to point out that **using general-purpose LLMs that are not tailored to verifying the knowledge-augmented LMs is inferior to our instruction-finetuned smaller LM**, as shown in Table 3. This result further demonstrates that our technique for fine-tuning the smaller verifier is not limiting but rather has huge advantages in both performance and efficiency.\n\n---\n\n> **Question 1:** It would be helpful to include the state-of-the-art for each dataset. While it is understandable to group and compare against similar approaches, it is hard for me to tell how well the model performs when compared against such a restricted set of methods.\n\nThank you for your suggestion. However, we would like to emphasize that including the state-of-the-art knowledge-augmented LMs in our experiments is unnecessary, since **we already use the architectures of one of the most powerful and very recently proposed knowledge-augmented LMs** (Mallen et al., 2023; Back et al., 2023). Also, **our verification method can orthogonally improve any knowledge-augmented LMs** including the most powerful one. \n\nSpecifically, existing knowledge-augmented LMs aim at improving the **architectures of knowledge retrieval and answer generation methods**; meanwhile, our work aims at **detecting the retrieval and generation errors of any existing retrievers and generators**, including the most powerful ones, in order to improve their reliability rather than directly improving their effectiveness with model changes. \n\nAdditionally, including and comparing against models that use different LM backbones (e.g., FLAN vs GPT-4) may not provide any valuable experimental insights, since different LMs have different capacities which makes apple-to-apple comparisons between them infeasible.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The emphasis on validating the faithfulness of a model's \"explanation\" (i.e., retrieved knowledge here, but it could be other outputs like chain-of-thought) is an important topic\n\nThe paper is well structured and easy to follow\n\nThe results as compared to similar approaches seem reasonable and a good number of datasets were explored.", "reasons_to_reject": "The novelty of the approach (as I'm understanding it) appears to be the use of a fine tuned language model to verify a response against retrieved facts. This is a very straightforward approach that may have been unexplored for this particular task (i.e., question-answering against a KB) but not overall (the use of LLMs as verifiers is not new). To me, this makes the main contribution of this work very incremental\n\nIt seems to me that the need to fine-tune the verifier would be quite limiting, as you could then only apply this to datasets for which you had adequate amounts of data", "questions_to_authors": "It would be helpful to include the state-of-the-art for each dataset (not just limited to the baselines here). While it is understandable to group and compare against similar approaches, it is hard for me to tell how well the model performs when comparing against such a restricted set  of methods", "paper_topic_and_main_contributions": "The authors propose a system for validating the 1) relevance and 2) faithfulness of a language models response with respect to retrieved knowledge. Their approach solves questions by first retrieving knowledge from a knowledge base, formulating a response based on that retrieved knowledge, and then validating the response using an instruction-finetuned verifier language model that can identify whether a response fits 1 and 2 (above). They evaluate their approach on a number of datasets and achieve better performance as compared to other retrieval-based approaches", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a verifier for RAG models that verifies that the retrieved evidence is relevant to the query and that the generation is attributed in the evidence.

The reviewers agree that the tackled problem is important, and that the proposed approach is simple and effective.

The reviewers also note that using an LLM for verification for this specific task is relatively incremental, as using verifiers in NLP is well known. In addition, the paper should be edited for further clarification regarding specific questions raised by the reviewers.
Decision: Accept-Main