Venue: NAACL 2024

# Effective Large Language Model Adaptation for Improved

Grounding and Citation Generation

Xi Ye\({}^{\diamondsuit}\) Ruoxi Sun\({}^{\spadesuit}\) Sercan O. Ark\({}^{\spadesuit}\) Tomas Pfister\({}^{\spadesuit}\)

\({}^{\diamondsuit}\) The University of Texas at Austin \({}^{\clubsuit}\) Google Cloud AI

\({}^{\diamondsuit}\)xiye@cs.utexas.edu

\({}^{\spadesuit}\){ruoxia,soarik,tpfister}@google.com

Work done during an internship at Google Cloud AI.

###### Abstract

Large language models (LLMs) have achieved remarkable advancements in natural language understanding and generation. However, one major issue towards their widespread deployment in the real world is that they can generate "hallucinated" answers that are not factual. Towards this end, this paper focuses on improving LLMs by grounding their responses in retrieved passages and by providing citations. We propose a new framework, _AGREE_, Adaptation for **GR**ounding **E**nhancEment, that improves the grounding from a holistic perspective. Our framework tunes LLMs to self-ground the claims in their responses and provide accurate citations to retrieved documents. This tuning on top of the pre-trained LLMs requires well-grounded responses (with citations) for paired queries, for which we introduce a method that can automatically construct such data from unlabeled queries. The self-grounding capability of tuned LLMs further grants them a test-time adaptation (TTA) capability that can actively retrieve passages to support the claims that have not been grounded, which iteratively improves the responses of LLMs. Across five datasets and two LLMs, our results show that the proposed tuning-based Agree framework generates superior grounded responses with more accurate citations compared to prompting-based approaches and post-hoc citing-based approaches.

## 1 Introduction

Recent advancements in large language models (LLMs) have yielded demonstrably groundbreaking capabilities in natural language processing (NLP) Brown et al. (2020); Chowdhery et al. (2022). Their ability to understand, generate, and manipulate text at unprecedented scales and depths has established them as a transformative force within the burgeoning field of artificial intelligence, poised to significantly impact our increasingly data-driven world. Despite their widely spread adoption, one prominent issue of LLMs is that in certain scenarios they hallucinate: they generate plausible-sounding but nonfactual information Maynez et al. (2020); Ji et al. (2023); Menick et al. (2022), limiting their applicability in real-world settings. To mitigate hallucinations, solutions generally rely on grounding the claims in LLM-generated responses to supported passages by providing an attribution report Rashkin et al. (2023); Bohnet et al. (2022); Gao et al. (2023a) or adding citations to the claims Liu et al. (2023); Gao et al. (2023b); Huang and Chang (2023).

There has been a growing amount of interest in making LLM-generated responses more trustworthy by grounding and adding citations. One line of work uses instruction tuning Kamalloo et al. (2023) or in-context learning Gao et al. (2023b) to instruct LLMs to generate grounded responses with citations to retrieved passages, following the retrieval-augmented generation Chen et al. (2017); Guu et al. (2020); Lewis et al. (2020) framework. As LLMs are required to perform this challenging task from just instructions and few-shot demonstrations, such directions often lead to mediocre grounding quality Gao et al. (2023b). Another line of work is on post-hoc citing Gao et al. (2023a); Chen et al. (2023), which links support passages to the claims in responses using a natural language inference (NLI) model. This paradigm heavily relies on LLMs' parametric knowledge and might not extend well to less-known knowledge Sun et al. (2023).

We propose a new learning-based framework, **Agree**, **A**adpatation of LLMs for **GR**ounding **E**nhanc**Ement. As shown in Fig. 1, our framework fine-tunes LLMs to generate citations, as opposed to prompting or relying on an external NLI model used in a post-hoc way. At the training

phase, Agree collects well-grounded responses for unlabelled queries automatically from a base LLM with the help of an NLI model. Next, the collected data are used for supervising LLMs to generate grounded responses based on the retrieved passages as well as include citations in their responses. As a test-time approach, we propose an iterative inference strategy that allows LLMs to seek for additional information based on the self-grounding evaluation so as to refine its response. The tuning and test-time adaptation together enable LLMs to effectively and efficiently ground their responses in the corpus. We apply Agree framework to adapt an API-based LLM, text-bison, and an open LLM, l1ama2-13b, with training data collected using unlabelled queries from three datasets. We conduct evaluation on both in-domain and out-of-distribution datasets, comparing the proposed Agree framework against competitive in-context learning and post-hoc citing baselines. The experimental results highlight that Agree framework successfully improves grounding, in citation recall & precision, compared to the baselines by a substantial margin (generally more than 20%). We find LLMs can learn to add accurate citations to their responses with our carefully designed tuning mechanisms. Furthermore, the improvements in grounding quality achieved by tuning using certain datasets can generalize well across domains. To summarize, our main contributions include:

* A learning-based approach that adapts a base LLM to include accurate citations in its response, leveraging automatically created data;
* A test-time adaptation (TTA) method that iteratively improves responses of LLMs based on the citation information;
* Extensive experiments on two LLMs over five datasets demonstrating the effectiveness of the proposed Agree framework for improving grounding and citation generation.

## 2 Related Work

Hallucination is a prevalent issue for generative language models on many tasks Maynez et al. (2020); Raunak et al. (2021); Dziri et al. (2021); Ji et al. (2023); Ye and Durrett (2022); Tang et al. (2023); Huang and Chang (2023). It has been evaluated in different ways, investigating the grounding in generated responses Bohnet et al. (2022); Rashkin et al. (2023); Min et al. (2023); Yue et al. (2023).

Various approaches have been proposed to mitigate hallucination and improve the factuality of LLM-generated responses. Among these, our work particularly focuses on providing citations to attributable information source Liu et al. (2023); Gao et al. (2023). Unlike existing work that largely relies on zero-shot prompting or few-shot prompting Kamalloo et al. (2023); Gao et al. (2023) or use an additional NLI model Gao et al. (2023); Chen et al. (2023) to add citations, we propose a learning-based approach that tunes LLMs to generate better-grounded responses supported with citations.

More broadly, recent work also investigates methods for improving factuality of LLMs without using external knowledge, including inference

Figure 1: Our framework, Agree, combines tuning (Section 4.1) and test time adaptation (Section 4.2) for better attribution and citation generation.

 

time intervention (Li et al., 2023; Chuang et al., 2023), cross-exam (Cohen et al., 2023; Du et al., 2023), self-verify (Dhuliawala et al., 2023), or reinforcement learning (Tian et al., 2024; Wu et al., 2023). Our work differs from them in providing citations to external knowledge in the responses. Additionally, there is past work that also uses external knowledge (e.g., knowledge base) to reduce hallucination by injecting knowledge into prompts (Elaraby et al., 2023; Peng et al., 2023). While the external knowledge used for generating a response can possibly serve as a coarse and general reference, these approaches also do not offer granular, sentence-level citations as in our work.

Lastly, the proposed framework is a form of a retrieval augmented generation approach. While past work has explored using retrieval to improve LLM generation quality (Chen et al., 2017; Lewis et al., 2020; Guu et al., 2020; Izacard and Grave, 2020; Shi et al., 2023) or factuality (Shuster et al., 2021; Jiang et al., 2023; Pan et al., 2023), our approach further enables LLMs to generate citations and self-generated citations to guide retrieval.

## 3 Problem & Background

Our proposed framework aims to adapt a pre-trained LLM \({\cal M}^{\mathcal{B}}\) to \({\cal M}^{\mathcal{A}}\) that is able to provide grounded responses with citations. Given a text query \(Q\) and a corpus \({\cal D}=\{d_{i}\}\) consisting of text passages, the adapted LLM \({\cal M}^{\mathcal{A}}\) is required to generate a response \(A\) to the query that is factually grounded in the corpus \({\cal D}\) as well as providing citations \({\cal C}\) together with its response.

Following past work (Liu et al., 2023; Gao et al., 2023b), we segment LLMs' output into statements by sentences and require each of the sentences to cite a set of passages from the corpus. Specifically, let \(s_{1},\ldots,s_{n}\) be the statements in the answer \(A=s_{1},\ldots,s_{n}\). The citations \({\cal C}=\{E_{1},\ldots,E_{n}\}\) links each statement \(s_{i}\) to a set of evidence passages \(E_{i}\subset{\cal D}\).

Recall that our adaptation aims to provide better grounded responses. With citations \({\cal C}\), we can quantify the grounding quality of a response \(A\) by a grounding score \({\cal G}\):

\[{\cal G}(A,{\cal C})=\frac{1}{n}\sum_{i}\phi(\mathrm{concat}(E_{i}),s_{i}),\]

where \(\phi\) is an NLI model that assesses whether the concatenated passage \(\mathrm{concat}(\mathrm{E_{i}})\) supports \(s_{i}\). The grounding score \({\cal G}\) essentially averages how well each sentence is supported by its citations.

## 4 AGREE Framework

The proposed Agree framework takes a holistic perspective for grounding, proposing a model tuning approach that adapts the base LLM to include citations in its responses, and introducing a test-time adaptation (TTA) mechanism that leverages the citation information for actively retrieving from the corpus and iteratively refining the responses.

### Tuning LLMs

We tune the LLM to self-ground the claims in their responses by providing citations to retrieved documents. Our method is able to grant LLMs such an ability using only a collection of _unlabeled_ queries \(\{Q\}\) and an NLI model \(\phi\). As we are using unlabeled queries without reference responses, we formulate the adaptation task as tuning LLMs to achieve better grounding without heavily deviating from the original generations (such an approach of preservation has also been adopted in recent work (Gao et al., 2023a)). Conceptually, we adapt \({\cal M}^{\mathcal{B}}\) to \({\cal M}^{\mathcal{A}}\) so that the answers generated by the adapted LLM \({\cal M}^{\mathcal{A}}\) should satisfy the grounding constraints (with grounding score \(>\tau_{\cal G}\)) while maximizing the scores with respect to the base LLM \({\cal M}^{\mathcal{B}}\):

\[\max\mathbb{E}_{(A,{\cal C})\sim{\cal M}^{\mathcal{A}}(\cdot|Q,{\cal D})}{\cal M }^{\mathcal{B}}(A\mid Q,{\cal D})\mathbbm{1}\{{\cal G}(A,{\cal C})\geq\tau_{ \mathcal{G}}\}. \tag{1}\]

In practice, we adopt a data-centric approach for optimizing \({\cal M}^{\mathcal{A}}\). For a given question, we opt to **use the maximally-grounded response sampled from the base LLM** to construct the tuning data. We will detail the process in the following of this section.

Data generationAs shown in Fig 2, given the query, we first sample responses \(\{A\}\) from the base LLM \({\cal M}^{\mathcal{B}}(\cdot\mid Q,{\cal D})\) using instruction following (see Appendix A for details). For each \(A=s_{1},\ldots,s_{n}\) we create citations \({\cal C}=\{E_{i}\}\) using the NLI model, \(\phi\), to link a sentence \(s_{i}\) to the maximally supported passage \(e_{i}=\max_{d_{\in}{\cal D}}\phi(d,s_{i})\) if the passage \(e_{i}\) actually support \(s_{i}\) (i.e., \(\phi(e_{i},s_{i})>\tau\)).1 Otherwise, we do not add a citation to \(s_{i}\), and \(s_{i}\) is an unsupported statement. That is: \(E_{i}=\left\{e_{i}\right\}if\phi(e_{i},s_{i})>\tau~{}else\left\{\right\}\). We use \(U\) to 

denote the set of unsupported statements that cannot find citations. This allows us to evaluate the grounding of \(A\) as in Section3. Now, we can choose the best response \(A^{*}\) from \(\{A\}\) based on the grounding scores to form a grounded response, i.e., \(A^{*}=\operatorname*{arg\,max}_{A}\mathcal{G}(A,C)\).

We then use \(\{Q,A^{*},C^{*}\}\) (\(C^{*}\) as the citations associated with \(A^{*}\)) to teach the base LLM to generate grounded responses with citations. In addition to citations, we also instruct the LLM to clearly state the unsupported statements \(U^{*}\), as shown in Fig.2. We note that the tuning of framework does not force all training responses to be perfectly grounded. Instead, we supervise the LLM itself to identify unsupported statements. This allows the LLM to generate more flexibly and guide the retrieval process with its knowledge.2

Footnote 2: Please refer to Appendix A for more details on the tuning method.

Supervised fine-tuningWe have introduced how we construct supervision to instruct the LLM to add citations and state unsupported statements in its response. To effectively tune the LLM, we verbalize the entire process in natural language. We denote the verbalized natural language description as \(\mathrm{VERB}(A^{*},\mathcal{C}^{*},U^{*})\) (see Fig.2 for a concrete example).3 The natural language formalization also allows us to conveniently tune the LLM with standard language modeling objectives:

Footnote 3: Please refer to Appendix E for more examples of tuning data.

\[\mathcal{M}^{A}=\operatorname*{arg\,max}_{\mathcal{M}}\sum_{Q}\mathcal{M}( \mathrm{VERB}(A^{*},\mathcal{C}^{*},U^{*})\mid Q,\mathcal{D}). \tag{2}\]

We note that this actual objective, Eq2, maximizes the log probability of generating the best-grounded answer \(A^{*}\) that is selected from the generation of the base model. As \(A^{*}\) is sampled from the base model, such an objective avoids significant deviations from the original generations, which aligns with the goal of the conceptual objective (Eq1).

Multi-dataset trainingWe use multiple existing datasets to construct the adaptation data used to tune the pre-trained LLM, including Natural Questions (NQ) Kwiatkowski et al. (2019), FEVER Thorne et al. (2018), and StrategyQA Geva et al. (2021). We choose these as they contain diverse text, and the answers to the corresponding queries require different types of reasoning processes: NQ provides diverse queries naturally asked by real human users; FEVER places a particular emphasis on fact verification; and StrategyQA requires multi-hop reasoning with implicit strategy. It is worthwhile to note that Agree _only_ uses queries, leaving out ground-truth answers, to improve LLMs.

Figure 3: Illustration of the test-time adaptation mechanism. The adapted LLM retrieves from the corpus based on self-generated citation information to refine its response in an iterative way.

Figure 2: Illustration of the tuning process. We sample responses from the base model, use an NLI model to add citations to the sampled responses, and tune the base model with the best-grounded response. We also show a concrete example of tuning data on the right.

 



### Test-time adaptation

We introduce a novel test-time adaptation (TTA) method for the inference procedure, overviewed in Fig. 3. Our framework is a form of retrieval augmented generation framework - at the core of our approach lies the adapted LLM that is able to answer a query based on a set of given passages retrieved from the corpus, and, more importantly, self-ground its response to add citations to the passages as well as to find unsupported statements needing further investigation. With these capabilities, the adapted LLM can iteratively construct a set of relevant passages from the large corpus \({\cal D}\) and refine its response to the query.

The detailed procedure of TTA is shown in Algorithm (1). Given a query \(Q\) and the corpus \({\cal D}\), we first retrieve based on the query to obtain an initial set of working passages. Next, we employ the following procedure iteratively until we consume all the budget \(B\) of invoking LLM calls. At each iteration, the LLM generates a response to the query based on the working passages, adds citations to its response, and finds out any unsupported statements that do not have citations (ln 9). Then, we add the cited passages to the list of relevant passages. Lastly, at each iteration, we update the working passages - if there are unsupported statements, we include additional information retrieved based on the unsupported statements (ln 15), otherwise, we include more passages that are retrieved based on the query to acquire more complete information (ln 17). We only include passages that are new and haven't been presented to the LLM yet (In 19). Note that at each iteration, we let the LLM to re-generate a response based on the current working passages instead of editing from previous one, which we observed lead to better fluency.

The design of our proposed TTA enables efficient and flexible inference. We rely on the LLM to generate citations itself, which has the advantage of reduced overhead of invoking an additional NLI model in a post-hoc way. Also, as we iteratively refine the answer, such a process can be streamed and flexibly controlled by setting a budget in deployment.

## 5 Experiments

### Setup

Evaluation datasetsWe conduct comprehensive evaluation on 5 datasets. Recall that we train Agree on multiple datasets including NQ, Strate

\begin{table}
\begin{tabular}{l c c l} \hline \hline Dataset & Type & Corpus & \# \\ \hline \multirow{2}{*}{NQ StrategyQA} & Train & \multirow{2}{*}{2500} \\  & Factoid QA & Wiki & \\  & Multi-hotep QA & Wiki & 1000 \\ Fever & Fact Checking & Wiki & 1000 \\ \cline{2-4}  & In-Distribution Test & \multirow{6}{*}{700} \\  & Factoid QA & Wiki & 460 \\ \cline{2-4}  & Out-of-Distribution Test & \multirow{6}{*}{ASQA} & \multirow{6}{*}{Ambiguous QA} & Wiki \\  & Multi-answer QA & Wiki & 1000 \\ Enterprise & Customer Support QA & Enterprise & 580 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics used for adaptation and test datasets. In addition to in-domain test datasets, we also investigate the generalization to out-of-distribution datasets that exhibit different reasoning processes or different corpus types.



gyQA, and Fever. In addition to the two in-domain test sets, NQ and StrategyQA (we leave out the non-QA dataset, FEVER), we further test the generalization of adapted LLMs on 3 out-of-domain datasets, including ASQA Stelmakh et al. (2022), QAMPARI Amouyal et al. (2022), and an Enterprise dataset.4 In particular, ASQA and QAMPARI contain questions of ambiguous answers and multiple answers. The Enterprise dataset is a proprietary dataset which requires provided answers that are grounded in customer service passages. Such an evaluation suite allows assessing the generalization capability of the adapted LLMs for OOD question types (ASQA and QAMPARI) as well as to an entirely different corpus (Enterprise).

Footnote 4: We use FEVER to create tuning data, but do not use it for evaluations. As we use LLMs in a zero-shot setting, the LLMs do not always answer with the specific labels defined in FEVER, which might introduce inaccuracies in the evaluation of answer correctness.

ModelsWe demonstrate Agree framework with two LLMs, text-bison and LlMa-2-13B Touvron et al. (2023). We use GTR-large Ni et al. (2021) as our retriever, and use TRUE Honovich et al. (2022) as the NLI model.

BaselinesWe evaluate the effectiveness Agree in two settings, invoking LLMs once, without TTA; and invoking LLMs multiple times, with the proposed TTA.5 We compare with three baselines from recent work, including one prompting-based approach and two post-hoc citing approaches, described below.

Footnote 5: We set the budget \(B\) for LLM calls used in TTA to be 4.

**Few-shot In-Context Learning** (ICLCite): Following Gao et al. (2023b), we prompt LLMs with few-shot examples Gao et al. (2023b), each consisting of a query, a set of retrieved passages, and an answer with inline citations. The LLMs can therefore learn from the in-context examples and generated citations in the responses. It is worthwhile to note that ICLCite_is a RAG baseline that also uses retrieved passages_.

**Post-hoc search** (PostSearch): Following Gao et al. (2023b), given a query, we first instruct LLMs to answer the query _without_ passages, and then add citations in a post-hoc way via searching. We link each claim in the response to the most relevant passage retrieved from a set of query-related passages. This baseline only uses the retriever but not the NLI model.

**Post-hoc Attribution** (PostAttr): Following Gao et al. (2023a), instead of citing the most relevant passage, for each claim, we retrieve a set of \(k\) passages from the corpus, and then use the NLI model, \(\phi\), to link to the passage that maximally supports the claim. We note both baselines in the post-hoc citing paradigm only rely on LLMs' parametric knowledge.6

Footnote 6: Please refer to Appendix B for more details on experimental setup.

MetricsWe mainly focus on improving the grounding quality of generated responses, reflected by the quality of citations. Following past work Gao et al. (2023b), we report the **citation recall** (rec) and **citation precision** (pre) on all the evaluation datasets. We note tha

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{NQ} & \multicolumn{3}{c}{StrategyQA} & \multicolumn{3}{c}{ASQA} & \multicolumn{3}{c}{QAMPARI} & \multicolumn{3}{c}{Enterprise} \\  & em-rec & rec & pre & acc & rec & pre & em-rec & rec & pre & rec-5 & rec & pre & rec & pre \\ \cline{2-17} ICDCite & 47.6 & 52.1 & 56.3 & 74.5 & 13.6 & 27.8 & 39.5 & 47.3 & 49.8 & 20.3 & 22.7 & 24.5 & 30.2 & 40.5 \\ \hline PostSearch & 45.1 & 29.7 & 28.7 & **75.5** & 20.1 & 20.1 & 38.4 & 19.2 & 19.2 & **22.5** & 16.2 & 16.2 & 15.9 & 15.9 \\ PostAttr & 45.1 & 31.5 & 31.5 & 75.5 & 18.4 & 18.4 & 35.1 & 38.0 & 38.0 & 22.5 & 18.5 & 18.5 & 20.1 & 20.1 \\ \hline Agreew/o TTA & 50.0 & 67.9 & 73.1 & 74.1 & 33.4 & 50.5 & 39.5 & 65.9 & 70.5 & 20.1 & 60.1 & 64.5 & 55.8 & 67.1 \\ Agreew/TLA & **53.1** & **70.1** & **75.0** & 74.9 & **39.2** & **57.9** & **40.9** & **73.2** & **77.0** & 20.9 & **62.9** & **67.1** & **57.2** & **68.6** \\ \cline{2-17} ICDCite & 45.8 & 42.8 & 41.6 & **65.5** & 20.6 & 33.1 & 35.2 & 38.2 & 39.4 & **21.0** & 10.2 & 10.4 & 30.6 & 38.8 \\ \hline PostSearch & 35.9 & 17.5 & 17.5 & 64.3 & 8.7 & 8.7 & 25.0 & 23.6 & 23.6 & 12.0 & 27.5 & 27.5 & 13.4 & 13.4 \\ PostAttr & 35.9 & 26.0 & 26.0 & 64.3 & 12.5 & 12.5 & 25.0 & 33.6 & 33.6 & 12.0 & 28.9 & 28.9 & 18.7 & 18.7 \\ \hline Agreew/o TTA & 47.9 & 50.5 & 56.6 & 65.0 & 25.5 & 35.0 & 35.7 & 50.2 & 55.3 & 17.1 & 40.4 & 43.6 & **50.6** & 53.8 \\ Agreew/TLA & **51.0** & **62.0** & **66.0** & 64.6 & **30.2** & **37.2** & **39.4** & **64.0** & **66.8** & 17.9 & **51.4** & **53.4** & 50.4 & **55.4** \\ \hline \end{tabular}
\end{table}
Table 2: Answer accuracy and grounding (measured by citation quality) of Agree and baselines across 5 datasets. Our approach achieves substantially better citation grounding (measured by citation recall) and citation precision compared to the baselines.

 

aggregates how well each sentence is supported by the citation to the corpus, which is essentially the grounding score \({\cal G}\). Therefore, we prioritize on the evaluation of citation recall.

We also report the correctness of the generated outputs. For NQ, we report exact match recall (emrec; whether the short answers are substrings in the response). For StrategyQA, we report the accuracy (acc). For ASQA and QAMPARI, we use subsets from Gao et al. (2023b), and report the exact match recall (em-rec) for ASQA and recall-5 (rec-5, considering recall to be 100% if the prediction includes at least 5 correct answers) for QAMPARI. For the Enterprise dataset, we only report the citation quality as there are no ground truth answers for this dataset, and citation quality reflects whether the model can provide accurate information.

### Results and analyses

Tuning is effective for superior grounding:Table 2 summarizes the results obtained using our Agree framework and compares with the baselines. As suggested by the results, across 5 datasets, Agree can generate responses that are better grounded in the text corpus and provide accurate citations to its response, substantially outperforming all the baselines. When tuned with high-quality data, LLMs can effectively learn to self-ground their response without needing an additional NLI model. On the other hand, ICLCite, which solely relies on in-context learning, cannot generate citations as accurately as a tuned LLM, as suggested by the large gap on citation precision between ICLCite and Agree. We also observe similar findings as suggested by Gao et al. (2023b): PostCite often leads to poor citation quality - without being conditioned on passages, the response from PostCite often cannot be paired with passages that lead to high citation recall for the generated claims.

The performance improvements can generalize:Recall that we adapt the base LLM only using in-domain training sets (NQ, StrategyQA, and FEVER), and directly test the model on out-of-distribution (OOD) test set (ASQA, QAMPARI, Enterprise). The results suggest that the improvements obtained from training on in-domain datasets can effectively generalize to OOD datasets that contain different question types or use different types of corpus. This is a fundamental advantage of the proposed approach - Agree can generalize to a target domain in the zero-shot setting without needing any samples from the target domain, which is needed for ICLCite.

TTA improves both grounding and answer correctness:The comparison between Agree without and with TTA highlights the effectiveness of the proposed iterative TTA strategy. We observe improvements in terms of both better grounding and accuracy. For instance, TTA improves llama-2 answer correctness by 3.1 and 3.7 on NQ and ASQA, respectively. Such improvements can be attributed to the fact that our TTA allows the LLMs to actively collect relevant passages to construct better answers following the self-grounding guidance.

Discussions on answer correctness:In general, Agree/_ITA_ can achieve better correctness compared to ICLCite. Agree/_to_ITA_ achieves similar answer correctness with ICLCite, as both methods are conditioned on the same set of passages. As a result, the quality of passages heavily intervenes on the correctness of the answers. Unlike Agree and ICLCite, PostAttr purely relies on the parametric knowledge of the LLMs to answer the query. As a result, PostAttr generally achieves inferior answer correctness compared to Agree and ICLCite on these two LLMs, especially on the less capable LLM, llama-2-l3b, that has less accurate knowledge compared to bison. Moreover, on the Enterprise dataset which contains more domain-specific information, PostAttr utterly fails to recall attributable information from LLMs' parametric knowledge.

Results with different LLMs:Our approach successfully adapts both text-boson-@1 and llama-2-l3b. llama is generally less capable compared to bison, underperforming bison in terms of answer correctness and citation quality. Still, Agree also consistently outperforms the baseline, generating more grounded answers as well as providing more precise citations. This highlights that the proposed tuning-based adaptation approach is model-agnostic and is effective across LLMs of varying capabilities.

Computational efficiency:Agree framework fine-tunes the base LLM to enable self-grounding without needing for additional in-context examples or NLI models. As a result, our framework is able to achieve strong citation performance without expensive inference cost. Table 4 shows the comparison between the computation cost, measured by the 

number of tokens processed by the LLM and the NLI model, needed for one query of our methods and that of the baselines. Compared to ICLCite, Agreew/o TTA uses much fewer tokens due to not using additional in-context examples, but achieves significantly better citation quality (see Table 2). PostAttr does not use retrieved passages in the prompts and hence requires less computation on the LLM compared to our framework, but it requires additional overhead of extensively invoking the NLI model (which has 11B parameters - see Appendix A for details) to verify the each of the claims based on each of the retrieved passages. The citation performance of PostAttr also substantially lags ICLCite and Agree. Agreew/TTA requires more computation compared to Agreew/o TTA, but is able to achieve both better citation quality and improvements in answer correctness.

#### The impact of training with multiple datasets:

Agree uses multiple datasets spanning factoid QA, multi-hop reasoning, and fact-checking to construct data for adapting the base model. We expect such a combination can grant the adapted model better generalization to different types of questions and different text distributions. We conduct an analysis to investigate the benefits of using multiple datasets for tuning. Table 3 shows the performance of our approach trained using multi-datasets and a counterpart that is trained only on NQ data (AgreeNQ-only). The results suggest that training using NQ leads to inferior citation quality compared to training on the combination of three datasets across all the datasets. The performance gap is especially significant on datasets other than NQ. Moreover, training only on NQ also leads to inferior answer correctness across all the datasets. Nevertheless, training only on NQ can still improve performance compared to solely relying on in-context learning (ICLCite).

Agree as a distillation approach:Our work mainly focuses on improving the base LLM's grounding capability in a self-improving way. That is, we use the samples generated by the base LLM itself to adapt the base LLM, as opposed to distilling from proprietary models which may impose constraints on the deployment of adapted model. Nevertheless, we conduct an analysis to investigate the effectiveness as a distillation approach in a datacentric way by using data generated from a more capable model to enhance the grounding data for a less capable model. We use the data generated by text-bison-0@1 to tune llama-2-13b.7 As shown in the last row of Table 3, AgreeDistill achieves better citation quality compared to Agree on llama-2-13b, as it is trained on better grounded responses produced by a more capable model. However, llama-2 fine-tuned on data generated by bison still can't level the performance on bison as constrained by the gap between the capabilities of these

\begin{table}
\begin{tabular}{l c c} \hline \hline  & **\# Tok: LLM** & **\#**Tok: NLI (TS-11B) \\ \hline ICLCite & 2800 & – \\ PostAttr & 360 & 3520 \\ \hline Agreew/o TTA & 1210 & – \\ Agreew/TLA & 4840 & – \\ \hline \hline \end{tabular}
\end{table}
Table 4: The average computation cost (for one query) of different methods measured by the number of tokens processed by the LLM and the NLI model (based on a T5-11B architecture). Agreew/o TTA is able to achieve better citation quality compared to ICLCite, despite consuming less than half of the tokens needed for ICLCite.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{NO} & \multicolumn{3}{c}{StrategyQA} & \multicolumn{3}{c}{ASQA} & \multicolumn{3}{c}{OAMPARI} & \multicolumn{3}{c}{Enterprise} \\  & em-rec & rec & pre & acc & rec & pre & em-rec & rec & pre & rec & pre & prec & prec \\ \hline \multicolumn{9}{c}{Base model: text=13bn-0@1} \\ ICLCTE & 47.6 & 52.1 & 56.3 & **74.5** & 13.6 & 27.8 & 39.5 & 47.3 & 49.8 & 20.3 & 22.7 & 24.5 & 30.2 & 40.5 \\ \hline \multirow{3}{*}{AGreeW/o TTA} & **50.0** & **67.9** & **73.1** & 74.1 & **33.4** & **50.5** & **39.5** & **65.9** & **70.8** & **20.1** & **60.1** & **64.5** & **55.8** & **67.1** \\  & 49.4 & 62.3 & 69.1 & 74.1 & 33.0 & 45.5 & 38.4 & 56.0 & 64.5 & 19.1 & 43.7 & 49.5 & 40.5 & 59.2 \\ \hline \multirow{3}{*}{ICLCITE} & 45.8 & 42.8 & 41.6 & 65.5 & 20.6 & 33.1 & 35.2 & 38.2 & 39.4 & 21.0 & 10.2 & 10.4 & 30.6 & 38.8 \\ \cline{1-1}  & 47.9 & 50.5 & 56.6 & 65.0 & 25.5 & 35.0 & 35.7 & 50.2 & 55.3 & 17.1 & 40.4 & 43.6 & 50.6 & 52.8 \\ \cline{1-1}  & 48.1 & 47.4 & 53.6 & 62.1 & 25.0 & 30.2 & 35.0 & 44.0 & 51.2 & 15.7 & 33.1 & 38.0 & 44.7 & 49.2 \\ \hline \multirow{3}{*}{AGreeW/o TTA} & 47.9 & 59.1 & 65.1 & 64.4 & 30.5 & 41.1 & 35.2 & 58.5 & 65.2 & 17.9 & 52.5 & 52.7 & 48.1 & 55.9 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Analysis on the impact of training data. Training with multiple datasets (AgreeMulti-dataset) leads to better grounding (citation recall) and better citation precision across datasets, compared to training using the NQ dataset (AgreeNQ-only). The citation quality of a less capable model llama–2–13b can also benefit from tuning using outputs from a more capable model (text-boson-0@1).

 

two LLMs.

Qualitative analyses:We qualitatively analyze the advantages of the proposed Agree framework compared to ICLCite, the strongest among the baselines. We observe that on both text-bison-001 and llama-2-13b, ICLCite achieves inferior citation quality due to failure in following the citation format (e.g., adding citations after the periods, violating the instructions), linking a statement to a relevant but un-attributable passage (as indicated by poor citation precision), and introducing more auxiliary information not mentioned in the retrieved passages (as indicated by citation recall). Our Agree framework mitigates these issues by tuning on well-grounded responses certified by the NLI model. We also provide example outputs in Fig. 4 comparing the outputs of Agree with and without proposed TTA and observe that TTA can help find more supporting passages by active retrieving using unsupported statements (top) or iteratively find more passages to construct a more complete response (bottom).

## 6 Conclusion

We introduce a novel framework, Agree, that adapts LLM for improved grounding. The proposed framework tunes a pre-trained LLM to self-ground its response in retrieved passages using automatically collected data. The integrated capability for grounding their responses further enables the LLM to improve the responses at test time. Our evaluations across five datasets demonstrate the benefits of the proposed learning-based approach compared to approaches that solely rely on prompting or the parametric knowledge of LLMs.

## 7 Limitations and future work

Agree employs an automated data creation that relies on an NLI model, instead of humans. Thus, the citation quality is dependent on the NLI model. As suggested in Gao et al. (2023b); Honovich et al. (2022), one issue might be favoring "fully support" and cannot effectively detect "partially support". Thus, the adapted LLMs may favor adding "fully support" citations. One solution is to curate a set of human-annotated citations for "partially support", which we defer to future work. Also, our evaluation follows prior work (Rashkin et al., 2023; Gao et al., 2023a) and uses the NLI model to quantify the grounding and citation quality. Therefore, our work can encounter the same issue as past work: grounding and citation quality evaluation is limited by the capabilities of the NLI model.

Agree uses created grounded responses to LLMs via supervised fine-tuning, as we demonstrate it leads to strong empirical results. It is also possible to treat grounding as a preference and RLHF (Ouyang et al., 2022) to tune LLMs, which we leave to future work. Agree tuning incurs additional cost that is a one-time requirement for adapting the LLM. Considering the substantial grounding improvements, we believe this would be acceptable for most applications, especially for those with high-reliability requirements. Future work can possibly explore training a separate universal improved grounding model beyond task-specific adaptation.

We have mainly considered open domain question answering datasets focusing on information seeking tasks in English. Generalization to other

Figure 4: Output examples of the proposed Agree framework with text-bion-001 as the base model. TTA is able to improve the response by retrieving more relevant information to precisely support a statement (see top) or finding more passages to generate a more complete response (see bottom).

 

long form generation tasks and other languages can be important future work directions.

Lastly, adding citations to LLM-generated responses in Agree might carry a shared risk with related research - a seemingly plausible but incorrect citation could potentially make an unsupported statement more convincing to users.

## Acknowledgments

Thanks to anonymous reviewers for their helpful feedback, as well as to Jinsung Yoon, Andreas Terzis, Yanfei Chen, Ankur Taly, Lucas Zhang, and Tina Pang for their help with various aspects of this work.

 

text with citations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1, SS2.
* M. Geva, D. Khashabi, E. Segal, T. Khot, D. Roth, and J. Berant (2021)Did aristote use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics9, pp. 346-361. External Links: Link, Document Cited by: SS5.
* K. Guu, K. Lee, Z. Tang, P. Sapat, and M. Chang (2020)Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938. Cited by: SS1, SS2.
* H. He, H. Zhang, and D. Roth (2022)Rethinking with retrieval: Faithful large language model inference. arXiv preprint arXiv:2301.00303. Cited by: SS1, SS2.
* A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi (2019)The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751. Cited by: SS1, SS2.
* O. Honovich, R. Aharoni, J. Herzig, H. Taitelbaum, D. Kulkinasy, V. Cohen, T. Scialom, I. Szpektor, A. Hassidim, and Y. Matias (2022)TRUE: re-evaluating factual consistency evaluation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, pp. 3905-3920. External Links: Link, Document Cited by: SS1, SS2.
* E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022)LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, Cited by: SS1, SS2.
* J. Huang and K. Chen-Chuan Chang (2023)Citation: a key to building responsible and accountable large language models. ArXivabs/2307.02185. Cited by: SS1, SS2.
* G. Izacard and E. Grave (2020)Leveraging passage retrieval with generative models for open domain question answering. External Links: Link, 2007.02185 Cited by: SS1, SS2.
* Ziwei Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, C. Ishii, Y. Bang, A. Madotto, and P. Fung (2023)Survey of hallucination in natural language generation. ACM Comput. Surv.55 (12). External Links: ISSN 0018-921X, Link, Document Cited by: SS1, SS2.
* Zhengbao Jiang, Frank F. Xu, L. Gao, Z. Sun, L. Daiy Liu, J. Dwivedi-Yu, Y. Yang, J. Callan, and G. Neubig (2023)Active retrieval augmented generation. ArXivabs/2305.06983. Cited by: SS1, SS2.
* E. Kamalloo, A. Jafari, X. Crystina Zhang, N. Thakur, and J. Lin (2023)Hagrid: a human-llm collaborative dataset for generative information-seeking with attribution. ArXivabs/2307.16883. Cited by: SS1, SS2.
* Tom Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, T. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M. Chang, A. W. Dai, J. Uszkoreit, Q. Le, and S. Petrov (2019)Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics7, pp. 452-466. External Links: Link, Document Cited by: SS1, SS2.
* Patrick Lewis, Ethan Perez, A. Picktus, F. Petroni, V. Karpukhin, N. Goyal, H. Kuttler, M. Lewis, W. Yih, T. Rocktaschel, et al. (2020)Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems33, pp. 9459-9474. External Links: Link, 2007.02349 Cited by: SS1, SS2.
* Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and J. Wen (2023a)HaluEval: a large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Cited by: SS1, SS2.
* Kenneth Li, O. Patel, F. Viegas, H. Pfister, and M. Wattenberg (2023b)Inference-time intervention: eliciting truthful answers from a language model. External Links: Link, 2007.02185 Cited by: SS1, SS2.
* Nelson F. Liu, T. Zhang, and P. Liang (2023)Evaluating verifiability in generative search engines. ArXiv:2304.09848. Cited by: SS1, SS2.
* Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald (2020)On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 1906-1919. External Links: Link, Document Cited by: SS1, SS2.
* Jacob Menick, M. Trebacz, V. Mikulik, J. Aslanides, F. Song, M. Chadwick, M. Glaese, S. Young, L. Campbell-Gillingham, G. Irving, and N. McAleese (2022)Teaching language models to support answers with verified quotes.. ArXivabs/2203.111147. Cited by: SS1, SS2.
* Sewon Min, Kalpesh Krishna, X. Lyu, M. Lewis, W. Yih, P. Wei Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi (2023)Factscore: fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1, SS2.
* J. Ni, C. Qu, J. Lu, Z. Dai, G. Hernandez Abrego, J. Ma, V. Zhao, Y. Luan, K. B. Hall, M. Chang, and Y. Yang (2021)Large dual encoders are generalizable retrievers. ArXivabs/2112.07899. Cited by: SS1, SS2.
* Longuyang, Jeff Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. E. Miller, M. Simens, A. Askell, 

Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022. Training language models to follow instructions with human feedback. _ArXiv_ abs/2203.02155.
* Pan et al. (2023) Liangming Pan, Xiaobao Wu, Xinyuan Lu, Anh Tuan Luu, William Yang Wang, Min-Yen Kan, and Preslav Nakov. 2023. Fact-checking complex claims with program-guided reasoning. In _Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL 2023)_, Toronto, Canada.
* Peng et al. (2023) Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. _arXiv preprint arXiv:2302.12813_.
* Rashkin et al. (2023) Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turco, and David Reitter. 2023. Measuring attribution in natural language generation models. _Computational Linguistics_, pages 1-64.
* Raunak et al. (2021) Vikas Raunak, Arul Menezes, andarcin Junczys-Dowmunt. 2021. The curious case of hallucinations in neural machine translation. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1172-1183, Online. Association for Computational Linguistics.
* Shi et al. (2023) Weijia Shi, Sewon Min, Michihiro Yasunaga, Mingjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. 2023. Replug: Retrieval-augmented black-box language models. _ArXiv_ abs/2301.12652.
* Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 3784-3803, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Stelmakh et al. (2022) Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022. ASQA: Factoid questions meet long-form answers. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 8273-8288, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Sun et al. (2023) Kai Sun, Y. Xu, Hanwen Zhao, Yue Liu, and Xinhsuai Dong. 2023. Head-to-tail: How knowledgeable are large language models (lllm)? a.k.a. will llms replace knowledge graphs? _ArXiv_, abs/2308.10168.
* Tang et al. (2023) Liyan Tang, Tanya Goyal, Alexander R. Fabbri, Philippe Laban, Jiacheng Xu, Semih Yavuz, Wojciech Kryscinski, Justin F. Rousseau, and Greg Durrett. 2023. Understanding factual errors in summarization: Errors, summarizers, datasets, error detectors. In _Proceedings of ACL_.
* Thorne et al. (2018) James Thorne, Andreas Vlachos, Christostodo Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
* Tian et al. (2024) Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. 2024. Fine-tuning language models for factuality. In _The Twelfth International Conference on Learning Representations_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funstowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. _arXiv preprint arXiv:1910.03771_.
* Wu et al. (2023) Zeqiu Wu, Yushi Hu, Weijja Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023. Fine-grained human feedback gives better rewards for language model training. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Ye and Durrett (2022) Xi Ye and Greg Durrett. 2022. The unreliability of explanations in few-shot prompting for textual reasoning. In _Proceedings of the Conference on Advances in Neural Information Processing Systems (NeurIPS)_.
* Yue et al. (2023) Xiang Yue, Boshi Wang, Kai Zhang, Zuru Chen, Yu Su, and Huan Sun. 2023. Automatic evaluation of attribution by large language models. _arXiv preprint arXiv:2305.06311_.

 

Details of Data Generation

Recall that we create the tuning data by first sampling responses from the base LLM and then using the NLI model to create citations and identify unsupported statements. We provide the details on the process in the following of this section.

NLI modelWe use TRUE NLI Honovich et al. (2022) as, to the best of our knowledge, it is the state-of-the-art NLI model for evaluating whether a passage supports a claim, and it is commonly used in the recent line of work on attributed QA and evaluating grounding Bohnet et al. (2022); He et al. (2022); Li et al. (2023a).8TRUE is trained on data collected from 6 datasets of diverse tasks covering NLI, paraphrase detection, and fact verification, which leads to its strong performance across diverse types of text. Furthermore, the citation performance evaluated by TRUE highly aligns with human evaluation Gao et al. (2023b).

Footnote 8: [https://huggingface.co/google/t5_xxl_true_nli_mixture](https://huggingface.co/google/t5_xxl_true_nli_mixture)

Corpus & retrieverAs mentioned before, our framework is an instantiation of retrieval-augmented framework. For the datasets using Wikipedia as the corpus (NO, StrategyQA, ASQA, and Qmparai), we use the 2018-12-20 Wikipedia snapshot as the corpus and set up the retriever using GTR-large Ni et al. (2021).

Sampling initial responsesWe sample initial responses from the base LLM using instruction following in a _zero-shot fashion_. Given a query, we present the base LLM with query and 5 retrieved passages appended after an instruction that requires the base LLM to answer the query based on the passages; see Fig. 5 for the template of the zero-shot prompt. We note that we opt to use a zero-shot prompt as opposed to a task-specific few-shot prompt since 1) this can avoid biasing the generation with the few-shot in-context examples, and 2) this matches the expected scenario for deploying the adapted LLM to handle new queries in a zero-shot fashion.

For text-bion-001, we sample 4 responses using a temperature of 0.5. For llama-2-13b, we sample 4 responses using nuclear sampling Holtzman et al. (2019) with p=0.95.

Adding citations and identifying unsupported statementsAfter obtaining the initial response \(\{A\}\) from the base LLM. We break each response \(A\) into sentences into \(s_{1},\ldots,s_{i}\). For each \(s_{i}\), we find the maximally supported passage \(e_{i}\) (scored by \(\phi(e_{i},s_{i})\)) that the base LLM has seen during generating the initial responses. We link \(e_{i}\) to \(s_{i}\) if \(\phi(e_{i},s_{i})>0.7\) to encourage more precise citations. For a sentence \(s_{i}\) if there does _not_ exist an \(e_{i}\) such that \(\phi(e_{i},s_{i})>0.5\) (the decision boundary for entailment), we add \(s_{i}\) to the unsupported statement set \(U\).

VerbalizingWe show the template for verbalizing the data used to tune the LLM in Fig. 6. As shown in the figure, we verbalize the citations in enclosed box brackets that are added at the end of sentences (before periods) like [n], and verbalize unsupported statements after the responses.

## Appendix B Details of Experimental Setup

Details of FinetuningFor tuning, we use LORA tuning Hu et al. (2022) in experiments on both text-bion-001 and llama-2-13b. For bison, we use API to perform tuning.9 and follow all the default hyper-parameters except for training steps. We set 10% data created as development data and choose to use a training step of 1000 (chosen from 500, 1000, and 2000). For llama-2, we use the huggingface transformers Wolf et al. (2019) chat-version checkpoint.10 We find the chat-version achieves better performance than the base checkpoint in our preliminary investigation. We set lora_r to be 32, and only choose to use a learning rate of 1e-5 (chosen from 1e-4 and 1e-5) using the development set.

Figure 5: Zero-shot prompt template for sampling initial responses from the base LLM.

 

We fine-tune llama-2 on two A100 (40GB) GPU for 4 epochs.

Details of DecodingOur evaluation uses the official implementation from ALCE Gao et al. (2023b), we use the same data split and prompt template from ALCE. We use temperature 0.25 for evaluation on both bison and llama. We use one sample for evaluation since adapted LLMs tend to generate better-grounded response exhibiting less variation.

Details of DistillationFor distillation, we directly tune llama-2-13b using the data created with text-bsion-001. We also set lora_r to be 32, use a learning rate of 1e-5, and fine-tune for 4 epochs.

## Appendix C Comparison to ICLCite on More Capable LLMs

Table 5 compares Agree using llama-2-138 as the base model against ICLCite on more capable models. We directly use the results from ALCE Gao et al. (2023b). Our framework is able to substantially shorten the gap between a small llama-2 model and much more capable LLMs.

## Appendix D License of Datasets

The licenses datasets used in our work include:

* NQ Kwiatkowski et al. (2019) under Creative Commons Share-Alike 3.0 license.
* StrategyQA Geva et al. (2021) under MIT License.
* Fever Thorne et al. (2018) under Creative Commons Share-Alike license.
* Ambiguous QA Stelmakh et al. (2022) under Creative Commons Share-Alike 3.0 license.
* Qampari Amouyal et al. (2022) under Creative Commons Zero v1.0 Universal license.

## Appendix E Additional Examples of Tuning Data

Please see Fig. 7 and Fig. 8 for concrete examples of tuning data.

Figure 6: Verbalization template for creating the training data for adapting the base LLM.

 



**Input**

Task: You will be given a question and some search results. You are required to perform the following steps. First, please answer the question in 3-5 sentences, and make sure you mention relevant details in the search results. You may use the same words as the search results when appropriate. Note that some of the search results may not be relevant, so you are not required to use all the search results, but only relevant ones. If you use the provided search results in your answer, add [n]-style citations. Next, review your response and find the unsupported sentences that do not have citations. Question: what kind of sentence contains an independent clause and a dependent clause? Search Results: \([1]\) Dependent clause ... A "complex sentence" contains an independent clause and at least one dependent clause. A sentence with two or more independent clauses plus (one or more) dependent clauses is referred to as a "compound-complex sentence". (Every clause contains a subject and predicate.)... \([2]\) Sentence clause structure ... A complex sentence has one or more dependent clauses (also called subordinate clauses). Since a dependent clause cannot stand on its own as a sentence, complex sentences must also have at least one independent clause.... \([3]\) Sentence clause structure A set of words with no independent clause may be an "incomplete sentence", also called a "sentence fragment". A sentence consisting of at least one dependent clause and at least two independent clauses may be called a "complex-compound sentence" or "compound-complex sentence". Sentence 1 is an example of a simple sentence... \([4]\) Topic sentence Using a complex sentence is a great way to refer to the content of the paragraph above (dependent clause) and then bring in the content of the new paragraph (the independent clause). Here is a typical example: The beginning, dependent, clause probably refers to the content of a preceding paragraph that presented the ant as a community-focused worker... \([5]\) Sentence clause structure This simple sentence has one independent clause which contains one subject, "I", and one verb, "run". This simple sentence has one independent clause which contains one subject, "girl", and one predicate, "ran into her bedroom"... Output 
\begin{tabular}{l} Answer: A complex sentence contains an independent clause and at least one dependent clause \([1]\). A dependent clause cannot stand on its own as a sentence, so complex sentences must also have at least one independent clause \([2]\). \\ Sentences Not Supported by Citations: None. &  \\ \end{tabular} Figure 7: An example of tuning data created from NQ.