Venue: NAACL 2025

# _Verifiable by Design_:

Aligning Language Models to Quote from Pre-Training Data

Jingyu Zhang Marc Marone Tianjian Li Benjamin Van Durme\({}^{\heartsuit}\) Daniel Khashabi\({}^{\diamondsuit}\)

Johns Hopkins University

Baltimore, MD

{jjhan237,mmarone1,tli104}@jhu.edu

###### Abstract

To trust the fluent generations of large language models (LLMs), humans must be able to _verify_ their correctness against trusted external sources. Recent efforts, such as providing citations via retrieved documents or post-hoc provenance, enhance verifiability but provide no guarantees of their correctness. To address these limitations, we aim to improve verifiability with a different philosophy: _trivializing the verification process by developing models that quote verbatim statements from trusted sources in their pre-training data._

We propose Quote-Tuning, which demonstrates the feasibility of aligning models to quote. The core of Quote-Tuning is a fast membership inference function that efficiently verifies text against trusted corpora. We leverage this tool to design a reward function to quantify quotes in model responses and curate datasets for preference learning. Experiments show that Quote-Tuning significantly increases verbatim quotes from high-quality documents by up to 130% relative to base models while maintaining response quality. Quote-Tuning is applicable in different tasks, generalizes to out-of-domain data and diverse model families, and provides additional benefits to truthfulness. Our method not only serves as a hassles-free method to increase quoting but also opens up avenues for improving LLM trustworthiness through better verifiability.1

Footnote 1: github.com/JHU-CLSP/verifiable-by-design Equal advising.

Trust, but verify.

## 1 Introduction

Recent developments have enabled large language models (LLMs) to generate fluent text and follow instructions Wei et al. (2022); Wang et al. (2023); Ouyang et al. (2022b); OpenAI (2023). However, LLMs are known to produce seemingly plausible but erroneous outputs, often referred to as hallucinations Ji et al. (2022); Zhang et al. (2023b), _i.a._). This poses significant risks to downstream users due to the difficulty of fact-checking seemingly convincing generations from LLMs Yue et al. (2023); Min et al. (2023); Asai et al. (2024). One of the important desiderata for LLMs is thus _verifiability_, the ability to ground responses to supporting evidence and render the produced claims easy to verify for humans. Verifiability allows users to uncover the competency of LLMs and _calibrate_ user trust, a crucial aspect of building trustworthy human-machine relationships Muir (1987).

Recent work increases verifiability through external artifacts such as producing citations Menick et al. (2022); Gao et al. (2023), retrieving documents Lewis et al. (2020), or post-hoc attribution methods Han and Tsvetkov (2022). Although helpful, these approaches provide no guarantee of relevance or usefulness. Models generations can be unfaithful to the retrieved documents in the context Shi et al. (2023b), generative search engines often produce citations that are irrelevant or inaccurate Liu et al. (2023), and explanations alone do not lead to verifiability Fok and Weld (2024).

We overcome the windingness of previous approaches through a _verifiable-by-design_ approach: generating **verbatim quotes** from high-quality sources such as Wikipedia. By determining verbatim quotes from large-scale and high-quality corpora with efficient membership testing tools Marone and Van Durme (2023), those generations provide a natural method for attributing and verifying the correctness of generated claims.

LLMs' potential to quote is driven by the observation that they are pre-trained on internet scale data--a subset of which contains high quality, reliable information--and that they have memorized a wide range of content from the pre-training stage Carlini et al. (2020, 2023); Biderman et al. (

2023; Hartmann et al., 2023). Such analyses focus on _covert_ memorization and use adversarial prompts to extract the memorized contents Carlini et al. (2020); Nasr et al. (2023). However, it remains an open question whether one can adapt LLMs to utilize their parametric knowledge to generate _contextual_ quotations across _a wide range of_ input prompts (not just specialized or adversarial ones) on realistic tasks that require long-form generation.

We show this is indeed feasible with QuoteTuning, our proposed method that aligns LLMs to quote through preference optimization and automatic feedback, without the need for human annotation. QuoteTuning first generates responses from a pre-trained LLM, and then synthesizes a preference dataset for quoting by ranking responses by how much they quote from a desired high-quality corpus, e.g., Wikipedia. Finally, QuoteTuning aligns the model to quote from trusted sources by applying preference optimization algorithms (e.g., direct preference optimization Rafailov et al. (2023)) on the synthesized reference dataset. Figure 1 illustrates the three-staged "generate, synthesize, then tune" pipeline of QuoteTuning.

Experiment results on long-form QA and open-ended text completion show that QuoteTuning significantly increases quoting by up to 130% relative to base models while maintaining or improving downstream performance (SS4). Moreover, QuoteTuning generalizes to other domains and diverse model families, and enhances the truthfulness as measured by TruthfulQA Lin et al. (2022) (SS5.1).

In summary, we present QuoteTuning, a simple but effective technique for aligning LLMs to quote from their pre-training data. The quoted responses are _verifiable-by-design_ by inducing better verifiability without the need for human annotation and external knowledge bases (only leveraging parametric knowledge). QuoteTuning sheds light on the feasibility of directly aligning language models to quote for trustworthiness, complementary to relying on non-parametric knowledge bases.

## 2 Preliminaries

Quantifying QuotingWe define a text string \(x\) as _quoted_ from a corpus \(C\) if a verbatim copy of \(x\) is contained in \(C\).This design allows us to use Data PortraitMarone and Van Durme (2023), a membership testing tool based on Bloom Filters Bloom (1970), to efficiently check whether text n-grams have appeared in the corpus. Specifically, we use Quoted Information Precision Score (QUIP-Score) metric proposed by Weller et al. (2024):

\[{\text{QUIP}}_{C}(x)=\frac{\sum_{\text{gram}_{n}\in x}\mathbbm{1}_{C}(\text{ gram}_{n})}{|\text{gram}_{n}\in x|},\]

where \(x\) is a text string, and \(C\) is a trusted corpus, \(\text{gram}_{n}\in x\) indicates all n-grams in \(x\), and \(\mathbbm{1}_{C}(\cdot)\) is an indicator function implemented by Data Portraits that return 1 if \(\text{gram}_{n}\in C\) else 0. Intuitively, \({\text{QUIP}}_{C}(x)\) measures the percentage of n-grams in \(x\) that appeared in \(C\).2

Figure 1: Pipeline of QuoteTuning. The algorithm works by (1) sampling multiple responses from a pre-trained LLM, (2) constructing preference data via rank-by-quoting, and (3) preference optimization to quote.

 

Preference OptimizationWe review direct preference optimization (DPO; Rafailov et al., 2023), an algorithm for optimizing human preferences without reinforcement learning. Given a pre-trained LLM policy \(\pi_{\text{ref}}\) and prompt \(x\), a pair of responses \((y_{1},y_{2})\sim\pi_{\text{ref}}(\cdot|x)\) is sampled from the pre-trained model. The response pair is subsequently evaluated for preference (human annotators or automated metrics), with the more favored response labeled as \(y_{w}\) and the less preferred one as \(y_{l}\). DPO assumes a static pairwise preference dataset \(\mathcal{D}=\{x^{(i)},y_{w}^{(i)},y_{l}^{(i)}\}_{i=1}^{N}\). The loss function for optimizing the parameterized LLM policy \(\pi_{\theta}\) is the following likelihood objective:

\[\begin{split}&\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\text{ ref}})=-\mathbb{E}_{(x,y_{w},yl)\sim\mathcal{D}}\\ &\Big{[}\log\sigma\Big{(}\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi _{\text{ref}}(y_{w}|x)}-\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}( y_{l}|x)}\Big{)}\Big{]},\end{split}\]

where \(\pi_{\theta}\) is initialized as \(\pi_{\text{ref}}\), \(\sigma\) is the sigmoid function, and \(\beta\) is a hyperparameter.

## 3 Aligning LMs to Quote with Quote-Tuning

The design of Quote-Tuning is inspired by the observation that preference datasets can be used to elicit desired behaviors in LMs using preference alignment frameworks Christiano et al. (2017); Ziegler et al. (2019); Ouyang et al. (2022b). The prior work, for example, has used this approach to address factuality Tian et al. (2024), honesty Yang et al. (2023), harmlessness Bai et al. (2022b); Shen et al. (2024), and relevance Wu et al. (2023). We investigate whether automatic measures of quoting can be used to align LLMs to _quote_ from their pre-training data. We introduce our methodology here and empirically show its feasibility in SS4.

Illustrated in Alg. 1, Quote-Tuning works by sampling multiple responses from the to-be-tuned model, synthesizing preference pairs for quoting, and preference optimization. We now detail each step. First, given a pre-trained LLM policy \(\pi_{\text{ref}}\), for each prompt \(x^{(i)}\) in a prompt dataset \(\mathcal{D}_{\text{prompt}}\), we sample \(T\) responses \(y_{1},\dots,y_{T}\sim\pi_{\text{ref}}(\cdot|x^{(i)})\) from the policy. Next, we construct pairwise preference data \((x^{(i)},y_{w},y_{l})\) by selecting a pair of response \((y_{w},y_{l})\) (where \(y_{w}\) is more preferred) from \(y_{1},\dots,y_{T}\) that satisfies two constraints:

**Constraint 1: quoting.**QUIP\({}_{C}(y_{w})-\text{QUIP}_{C}(y_{l})>\delta_{\text{quip}}\), where \(\delta_{\text{quip}}>0\) is a hyperparameter. Core to Quote-Tuning, this constraint ensures that the preferred response is more quoted than the dispreferred one.

**Constraint 2: length.**\(\frac{|len(y_{w})-len(y_{l})|}{\min\{len(y_{w}),~{}len(y_{l})\}}<\delta_{\text {length}}\), where \(\delta_{\text{length}}\in(0,1)\) is a hyperparameter. Motivated by recent findings that RLHF and direct preference optimization approaches lead to increased response length Singhal et al. (2023); Dubois et al. (2023), we regularize the preferred and dispreferred responses to have similar tokenized length with each other. We provide an ablation of the length constraint in SS5.3.

If multiple pairs of responses satisfy the con 

straints, a single pair \((y_{w},y_{l})\) with the highest average QUIP-Score among the two responses will be selected.3 This ensures the dispreferred response still maintains relatively high quoting. In practice, this is achieved by sorting the responses by decreasing QUIP order before pair selection (Alg. 1, line 5). If no response pair can be selected, the prompt \(x^{(i)}\) is discarded.

Footnote 3: The design to select a maximum of one response pair per prompt is to preserve the distribution of prompts. Prior work also experimented with employing all possible preference pairs Ouyang et al. (2022); Tian et al. (2024), which we leave to future work.

The reason why we employ model self-generation as the preferred response, instead of simply using verbatim quotes (e.g., spans of Wikipedia text that contains the gold answer) is twofold: (1) Using self-generated responses keeps the synthesized preference data on-policy, which is crucial for preference optimization as shown in recent work Tajwar et al. (2024). (2) Because quotes are a subset of the model response, they are constrained to be highly contextually relevant to both the query and the surrounding response context. This relevance would be harder to achieve with standalone quotes.

Finally, having obtained the synthetic preference dataset for quoting \(\mathcal{D}\), we conduct DPO using \(\mathcal{D}\) on the pre-trained LLM policy \(\pi_{\text{ref}}\) to obtain the quoting-aligned policy \(\pi_{\theta}\).

Desirability of QuotingWe show an example of the model generation before and after QuoteTuning in Table 1 and highlight segments that are quoted verbatim from the Pile Gao et al. (2020) subset of Wikipedia along with the corresponding QUIP-Score. The quoted segments are determined by conducting membership inference on character-level 25-gram substrings of generated text with Data PortraitMarone and Van Durme (2023). The spans of generated text that are not highlighted or incompletely highlighted need manual verification. **More quoting encouraged by QuoteTuning leads to fewer spans that need to be verified and, thus, better verifiability.** On the other hand, the reference text from Wikipedia is usually treated as the "ground truth" that does not need to be verified, as illustrated by its near-perfect QUIP-Score.4 We provide an extended discussion of verifiability in SS6.

Footnote 4: The minor mismatch is due to preprocessing and potential version differences.

We emphasize that the quality of the quoting corpus \(C\) is important: \(C\) needs to be carefully selected such that it contains high-quality, low-risk text, such as our choice of Wikipedia. We show in SS5.1 that quoting from truthful corpus increase model truthfulness, and discuss implications of the selection of \(C\) on privacy and copyright in SS7.

Aside from better verifiability, Weller et al. (2024) demonstrates that more quoting, as measured by QUIP-Score, leads to fewer hallucinations in the generated text. Our analysis in SS5.1 shows that encouraging quoting leads to more truthful models. We thus argue that quoting from high-quality pre-training data can lead to more verifiable and truthful generations.

\begin{table}
\begin{tabular}{p{39.8pt} p{142.3pt}} \hline _Question:_ Who won the most MVP awards in the NBA? & **QUIP** \\ \hline Reference: Every player who has won this award and has been eligible for the Nisismuth Memorial Basketball Hat & 94.1 \\ \#I Fame has been induced. Kareem Abdul-Jabbar won the award a record six times. Both Bill Russell and Michael Jordan won the award five times, while Witt Chamberlain and LeBron James won the award four times, Russel find James are the only players to have won the award four times in five seasons & \\ \hline Llama2-Tb-Chat: Lebron James has won the most NBA MVP awards with 4 championships followed!osely by Bill Russell with 5 MVP awards and 11 championships. Michael Jordan is also among the top! MVP winners with championships and 5 MVP awards. & 3.9 \\ Quote-Tuning: The player who has won the most Most Volatile Player (MVP) awards in the National Basketball Association (NBA) is Kareem Abdul-Jabbar, who has won the award a record six times during his career with the Milwaukee Bucks and the Los Angeles Lakeers. Abdul Jabbar played in the NBA from 196 to 1989 and is widely considered one of the greatest basketball players of all time & 60.6 \\ \end{tabular}
\end{table}
Table 1: Example outputs generated by Llama2-7B-Chat before and after Quote-Tuning on NJQ. Highlighted segments are quoted from Wikipedia that appeared in the Pile Gao et al. (2020). Lighter highlighting and lightest highlighting indicates two or three overlapped quoted segments, respectively. The minimum length to be considered quoted is a character-level 25-gram match. **Quote-Tuning significantly improves quoting from Wikipedia.** 



## 4 Experiments

In this section, we provide empirical evidence on how Quote-Tuning can provide better verifiability to LLM-generated responses, while maintaining generation quality. We conduct Quote-Tuning on the long-form QA (SS4.1) and open-ended text completion (SS4.2) tasks. Additionally, we show that quoting-aligned models are more truthful than their vanilla counterparts (SS5.1), and maintain downstream performance (SS5.2).

### Improving Quoting in Long-Form QA

Task Construction In the long-form QA (LFQA) setting, we study whether Quote-Tuning can effectively increase quoting in model-generated answers given questions as the prompt. To find settings relevant to Quote-Tuning, we select datasets that induce _long-form response_ to measure quoting and also allow us to _verify the answers from trusted sources_ such as Wikipedia. Accordingly, we experiment on two datasets, NaturalQuestions (NQ; Kwiatkowski et al., 2019) and ELIS Fan et al. (2019). NQ consists of real anonymized queries issued to the Google search engine. Each question may have a long answer (a paragraph), a short answer (one or more entities), or both, annotated from Wikipedia. We employ the subset of NQ that has long answers: we sample 20K training set questions to be used as the prompt dataset \({\mathcal{D}}_{\text{prompt}}\) for Quote-Tuning, and the full development set is used as the **in-domain** evaluation set. To evaluate whether quoting can be generalized to out-of-domain questions, we use the evaluation set of the ELI5 dataset, where questions are mined from the Reddit "Explain Like I'm Five" forum, as the **out-of-domain** evaluation set.

BaselinesAside from the pre-trained LLM policy \(\pi_{\text{ref}}\), we consider the _according-to_ prompting method from Weller et al. (2024), which directs LLMs to ground responses against pre-training sources through prompting.5 Finally, we include a strong Best-of-N QUIP reranking baseline, where we sample 32 responses from the pre-trained model \(\pi_{\text{ref}}\) and rerank the response by selecting the one with the highest QUIP-Score. Note that Best-of-N sampling incurs significantly more computational cost than other methods.6

Footnote 5: We use the best grounding prompt found in Weller et al. (2024), i.e., “Respond to this question using only information that can be attributed to Wikipedia.”

MetricsTo our main interest, we measure quoting with **QUIP-Score** using the Wikipedia subset of the Pile dataset Gao et al. (2020) as the grounding corpus \(C\).7 We report the **BARTscore** Yuan et al. (2021) and **Rouge-L**Lin (2004) between generated and reference answers as metrics for adequacy of generated answers. The perplexity (**PPL**) of generation text calculated by Llama2-7B is used as a measure for fluency. We also report average generation length as preference optimization could lead to length biases Singhal et al. (2023).

ResultsWe run experiments with hyperparameters detailed in SSB. We first employ Llama2-7B-Chat Touvron et al. (2023) as the pre-trained model \(\pi_{\text{ref}}\). After DPO, the reward accuracy on a held-out evaluation set is 86.3%, indicating that the model learns quot

\begin{table}
\begin{tabular}{c|l c c c c c} \hline \hline  & & _Quoting_ & \multicolumn{2}{c}{_Adequacy_} & _Fluency_ \\ \cline{3-6} _Setting_ & _Method_ & QUIP\(\uparrow\) & Rouge-L\(\uparrow\) & BARTSc\(\uparrow\) & PPL\(\downarrow\) & Length \\ \hline \multirow{3}{*}{_In-Domain_} & LLama2-7B-Chat & 34.9 & 22.4 & -3.99 & 4.96 & 115.9 \\  & +_According-to_ prompting & 36.2 & 22.9 & -3.95 & 4.55 & 129.6 \\ NQ & +Best-of-32 QUIP rerank & 50.4 & 23.3 & -3.98 & 4.40 & 110.2 \\  & +Quote-Tuning & **54.5** & **24.2** & **-3.93** & **3.78** & 117.6 \\ \hline \multirow{3}{*}{_Out-of-Domain_} & LLama2-7B-Chat & 26.8 & **18.8** & -4.78 & 3.93 & 179.8 \\  & +_According-to_ prompting & 28.0 & 18.3 & **-4.75** & 3.56 & 225.7 \\ \cline{1-1}  & +Best-of-32 QUIP rerank & 37.6 & 18.7 & -4.78 & 3.72 & 173.8 \\ \cline{1-1}  & +Quote-Tuning on NQ & **41.4** & 18.3 & -4.84 & **3.55** & 179.6 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results on Long-Form QA datasets. QUIP and Rouge-L are in percentages. Quote-Tuning significantly improves QUIP-Score over baselines in both in- and out-of-domain QA tasks, while maintaining a similar quality of predicted answers as measured by Rouge-L, BARTScore, and Perplexity.



well. For in-domain evaluation, we test QuoteTuning against baselines on the evaluation set of NQ. Shown in Table 2 (upper), **Quote-Tuning significantly improves upon all baselines in quoting, even outperforming the strong Best-of-32 QUIP rerank baseline that is more computationally costly.** In particular, Quote-Tuning enables a significant 56.2% (\(34.9\to 54.5\)) quoting improvement relative to the un-tuned Llama2-7B-Chat model. Quote-Tuning also slightly improves answer adequacy and fluency. Because Quote-Tuning significantly increases quoting from Wikipedia, which contains high-quality text thoroughly curated by human editors, the responses from quote-tuned models benefit from the high-quality nature on Wikipedia.8 While _according-to_ prompting slightly increases quoting at the expense of notably longer generation length, QuoteTuning maintains similar answer length compared to Llama2-7B-Chat generations. An example output is available in Table 1.

Footnote 8: Similar findings are also found in Li et al. (2024), which report improved perplexity by conducting speculative decoding to copy spans from a datastore.

To test the out-of-domain generalization ability of Quote-Tuning, we use Llama2-7B-Chat quote-tuned on NQ for evaluation on ELI5. Quote-Tuning still outperforms all baselines in quoting, while maintaining adequacy and improving fluency compared to the original model. Table 2 (lower) shows that **Quote-Tuning generalizes quoting to out-of-domain prompts.**

Finally, we apply Quote-Tuning on Llama3.1 Dubey et al. (2024), Gemma2 Team et al. (2024), and Starling Zhu et al. (2024) models, and find **Quote-Tuning consistently improve quoting, adequacy, and fluency across diverse model families** (Table 3).

### Improving Quoting in Open-Ended Text Completion

Task ConstructionWe now study whether Quote-Tuning can be applied to open-ended text completion, where we measure quoting on the candidate LLM's open-ended continuation of test prompts. We sample 20K passages from the dedplicated Pile subset of Wikipedia as the training set and another 2K passages as the evaluation set. For each passage, we use the first 32 tokens as the prompt and the remainder as the reference continuation, truncated to a maximum of 128 tokens.

Baselines and MetricsWe employ the pre-trained LLM policy \(\pi_{\text{ref}}\) and Best-of-N QUIP reranking baselines following the LFQA setting (SS4.1). Instead of according-to prompting, we fine-tune \(\pi_{\text{ref}}\) on reference continuations of the train set as another baseline since \(\pi_{\text{ref}}\) in this setting is not instruction-tuned. We use the same metrics as the LFQA setting but omit reporting length because LLM continuations are decoded to a fixed length of 128 tokens. We use Llama2-7B as the pretrained model \(\pi_{\text{ref}}\), and measure perplexivity with the Mistral-7B model instead to prevent self-evaluation bias He et al. (2023).9 

ResultsAfter conducting Quote-Tuning with hypermaraters detailed in SSB, the reward accuracy on a held-out evaluation set is \(84.0\%\). As shown in Table 4, **Quote-Tuning significantly improves both quoting and fluency over all baselines**. Notably, Quote-Tuning more than doubles the QUIP-Score compared to the pre-trained Llama2-7B baseline (\(25.7\to 59.2\), a 130.4% relative increase), and outperforms the strong QUIP reranking baseline. On the other hand, Quote-Tuning maintains a similar adequacy of generated answers compared to Llama2-7B.

Interestingly, Table 4 shows that simply reranking Llama2-7B generation by QUIP can lead to a better perplexity as measured by Mistral-7B. We hypothesize that because Wikipedia is an encyclopedia that has been revised multiple times and contains mostly high-quality text, quoting from this canonical corpus also has benefits of fluency aside from better verifiability.

## 5 Analysis

### Effect of Quoting on Truthfulness

We hypothesize that besides increasing verifiability, quoting from high-quality corpora such as Wikipedia might also increase truthfulness because LLMs are aligned to rely on trustworthy sources. To verify this hypothesis, we take the quote-tuned model from the LFQA setting (SS4.1) and evaluate its performance on the TruthfulQA dataset Lin et al. (2022). We follow the standard evaluation procedure on TruthfulQA, which fine-tunes GPT-3 Brown et al. (2020) on human annotations astruthfulness and informativeness judges. We defer further details to Appendix C.

As shown in Table 5, Quote-Tuning increases model truthfulness, as well as answers that are both truthful and informative, over the untuned Llama2-7B-Chat model by a notable margin. On the other hand, informativeness slightly dropped, suggesting that the quote-tuned model is more conservative and has an increased tendency to decline to answer. We provide example outputs in Table 8. Interestingly, **Quote-Tuning can improve model truthfulness even though not explicitly tuned to do so**: because the preference optimized in Quote-Tuning is only quoting as measured by QUIP-Score, the model is not directly optimized to be factual, in contrast to works that directly aims at truthfulness or fractuality Tian et al. (2024); Li et al. (2023). The increase of truthfulness likely attributes to the fact that Wikipedia is a relatively high-quality and reliable source, and quoting more from such reliable sources leads to more truthful responses. We thus posit aligning LLMs to reliable sources is a promising approach to increase their truthfulness.

### Evaluation of Downstream Performance

Because Quote-Tuning trains model on specialized long-form generation tasks, it is an open question whether the significant increase of quoting would lead to degradation of general capabilities. Thus, we now test the model before and after quote-tuning on general capability benchmarks MMLU Hendrycks et al. (2020), GSM8K Cobbe et al. (2021), BIG-Bench Hard (BBH; Suzgun et al. 2023), and Hellaswag HS; Zellers et al. (2019).10

Footnote 10: We conduct evaluation using the lm-evaluation-harness framework under default settings.

As shown in Table 6, Quote-Tuning only leads to very small degradations (less than two points for all tested benchmarks), while significantly improving quoting. We therefore find Quote-Tuning a significantly better trade-off for verifiability with a small cost of general capability.

\begin{table}
\begin{tabular}{l l l l l} \hline \hline  & \multicolumn{2}{c}{**MMLU**} & \multicolumn{1}{c}{**GSM8K**} & \multicolumn{1}{c}{**BBH**} & \multicolumn{1}{c}{**HS**} \\ \hline LLama2-7B-Chat & 46.38 & 20.92 & 40.21 & 75.51 \\ +Quote-Tuning & 45.65 & 19.79 & 39.47 & 73.96 \\ \hline \(\Delta\) & -0.73 & -1.13 & -0.74 & -1.55 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Evaluation on general capability benchmarks. Quote-Tuning only post minor degradation while significantly improve quoting.

\begin{table}
\begin{tabular}{l l l l|l l} \hline \hline  & \multicolumn{4}{c}{**Generation**} & \multicolumn{2}{c}{**Multiple Choice**} \\ \cline{2-5} _Method_ & Truthful & Informative & Truthful\(\times\)Informative & MC1 & MC2 \\ \hline Llama2-7B-Chat & 54.2 & **92.0** & 46.6 & 30.2 & 45.3 \\ +Quote-Tuning & **61.8 (+14.0\%)** & 89.5 (-2.7\%) & **51.5 (+10.5\%)** & **32.8 (+8.5\%)** & **47.9 (+5.6\%)** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results on TruthfulQA. Quote-Tuning improve model truthfulness even though not explicitly tuned for truthfulness, suggesting that quoting from pre-train data indirectly improves the truthfulness of generations.

 



### Ablation of the Length Constraint

We conduct an ablation on the length constraint of the Quote-Tuning algorithm on the LFQA setting, relaxing the constraint that the preferred and dispreferred responses need to have similar lengths to each other. Experimental results are shown in Table 7. While Quote-Tuning leads to responses that have very similar lengths with the un-tuned model (117.6 vs 115.9 on NQ, 179.6 vs 179.8 on ELI5), Quote-Tuning without the length constraint leads to notably _shorter_ response (105.9 on NQ, 154.3 on ELI5).

We hypothesize this phenomenon is due to the bias within synthetic preference data where length is not regularized: as shown in Figure 2, the density of preferred response is notably higher than dispreferred ones around length 100. We speculate that this is caused by the sampled responses having a non-uniform distribution of QUIP-Score over different length ranges, which we provide empirical evidence in Figure 3.

On the other hand, ablating the length constraint leads to slightly lower quoting, relatively similar adequacy, and notably worse perplexity compared to the full Quote-Tuning algorithm, depicting the effectiveness of the length constraint.

## 6 Related Work

Improving VerifiabilityHallucination in LLMs Ji et al. (2022); Zhang et al. (2023); Mishra et al. (2024) has motivated approaches that improve the verifiability of LLM generations. Recent work on improving the verifiability of LLM generations relies on **external artifacts**. One emerging trend is training LLMs to produce citations that support generated claims Menick et al. (2022); Gao et al. (2023); Huang et al. (2024). While citations improve attribution, LLM can still hallucinate incorrect or irrelevant citations Liu et al. (2023), which is non-trivial to verify Khalifa et al. (2024) introduce intrinsic source citation to enable more faithful attribution to parametric knowledge, on a synthetic setting. Quote-Tuning is an approach complementary to citations that enhances verifiability. Hennigen et al. (2024) finds that explicit symbolic references to structured conditioning data, such as JSON tables, lead to faster human verification, further motivating our approach of increasing

\begin{table}
\begin{tabular}{c|l c c c c c} \hline \hline  & & _Quoting_ & \multicolumn{2}{c}{_Adequacy_} & _Fluency_ \\ \cline{2-7} _Setting_ & _Method_ & QUP\(\uparrow\) & Rouge-L\(\uparrow\) & BARTSC\(\uparrow\) & PPL\(\downarrow\) & Length \\ \hline \multirow{2}{*}{_In-Domain_} & Llama2-TB-Chat & 34.9 & 22.4 & -3.99 & 4.96 & 115.9 \\  & +Quote-Tuning & **54.5** & 24.2 & **-3.93** & **3.78** & 117.6 \\  & +QT w/o len. constraint & 53.6 & **24.4** & -3.95 & 3.88 & 105.9 \\ \hline \multirow{2}{*}{_Out-of-Domain_} & Llama2-TB-Chat & 26.8 & **18.8** & **–4.78** & 3.93 & 179.8 \\  & +QT on NQ & **41.4** & 18.3 & -4.84 & **3.55** & 179.6 \\ \multirow{2}{*}{NO \(\rightarrow\) EL15} & +QT on NQ w/o len. constraint & 40.5 & 18.6 & -4.85 & 3.84 & 154.3 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Results on the ablation of the length constraint. QT is short for Quote-Tuning. Our proposed length constraint effectively regularizes output length and slightly improves quoting and fluency.

Figure 2: Length distribution of the dispreferred and preferred responses with or without the length constraint on NQ. **Left**: No length constraint. **Right**: added length constraint with \(\delta_{\text{length}}=0.1\). Adding length constraints properly regulates length distribution of responses.



verifiability through verbatim quotes.

Retrieval-augmented generation (Guu et al., 2020; Lewis et al., 2020b; Borgeaud et al., 2022; Izacard et al., 2023, _i.a._) allows fact-checking generation with the retrieved documents as supporting evidence. Min et al. (2023b) used retrieved tokens directly as generation, but is limited to the masked-filling setting with short spans of text. However, checking against retrieved documents is still non-trivial and there is no guarantee that generated text is completely faithful to these documents. On the other hand, our framework for quoting, based on Marone and Van Durme (2023); Weller et al. (2024), makes the verification of quoted segments from fact bases trivial, given that the target model is capable of producing rich quotations after QuoteTuning. Our work, which focuses on parametric knowledge, is also complementary to methods that rely on non-parametric knowledge bases.

Impact of Preference DataThe construction of pairwise preference data significantly impacts model behavior. Tian et al. (2024) fine-tunes LLMs to be more factual by constructing preference data with automatic measures of factuality (Min et al., 2023a) and model confidence scores. Yang et al. (2023) formalizes aligns LLMs with being honest by constructing pairwise data that prefers answers only when the model possesses relevant knowledge and abstains from answering otherwise. Yuan et al. (2024) iteratively constructs preference data by prompting LLMs themselves for quality measurements. Shi et al. (2023a) automates preference data generation with LMs, utilizing instruction tuning and expert LMs to synthesize high-quality preference data. Our work also synthesizes pairwise data that give preference to the one that quotes more from a given corpus. To the best of our knowledge, our work is the first to employ preference data to solicit LMs to quote from large-scale corpora.

MemorizationWorks have demonstrated that LLMs memorize a significant portion of their pretraining data Carlini et al. (2020, 2023); Hu et al. (2022); Ippolito et al. (2023); Biderman et al. (2023); Hartmann et al. (2023), and we can extract them by adversarial prompting Carlini et al. (2020); Nasr et al. (2023). Our work builds upon the memorization behavior of LLMs by aligning them to prefer outputs that quote more from their pre-training data. Also related to our work, \(k\)NN-LMs Khandelwal et al. (2019) improve generalization by using nearest neighbor search to retrieve similar contexts from a datastore. We defer further related work to SSA.

## 7 Discussions and Future Work

In SS4, we provide rich empirical evidence that Quote-Tuning can significantly promote parametric quoting across diverse tasks, domains, and model families. Since quoted texts are autoregressively generated as spans of the model responses, and our evaluation quantitatively shows that these responses are of high adequacy and fluency Tables 2, 3, 4), generated quotes are therefore constrained to be highly adequate, fluent, and context-relevant text. These findings demonstrate that it is not only possible but easily feasible to leverage parametric knowledge to generate more verifiable outputs. We thus argue **current LLMs have abundant underutilized potential in improving their own verifiability**, and call for future work that develop more attributable, verifiable approaches through our proposed "trivializing verification through quoting" framework.

As an early exploration of the quoting framework, this work focuses on investigating the _feasibility_ of unlocking parametric quoting through finetuning, and the _generalizability_ of enhanced quoting across different domains, tasks, and model families. We have provided positive empirical evidence supporting both aspects. Therefore, our setup focuses on measuring and improving the overall rate of quoting, as evaluated by QUIP-Score, leaving room for future work to enhance quoting reward signals by considering other desiderata of quoted texts, such as quote completeness and usefulness under the current context. We provide an extended discussion of limitations in the subsequent section.

Quoting has implications on privacy, security, and copyright. We focus on enhancing quoting from Wikipedia, a high-quality and low-risk corpus, for verifiability. As pre-training data contains diverse mixtures of data with varying risk levels Elazar et al. (2024); Longpre et al. (2024), we argue that the grounding corpus for QuoteTuning must be carefully selected and limited to trusted, public sources such as Wikipedia to prevent privacy violations or copyright infringement.

In conclusion, our approach presents a promising direction for leveraging the parametric knowledge of LLMs to facilitate easier verification of model generation and improve the calibration of human-machine trust 



## Limitations

(i) Our work maximizes the amount of quoting measured by QUIP-Score [23], but does not distinguish between many short quotes v.s. a few long ones, where the latter is more preferable. Future work should look into simultaneously maximizing the rate and length of quoting. (ii) Another future direction involves extending the experiments to other settings, such instruction tuning [15; 40; 69; 78; 80], where a diverse set of tasks are present. (iii) We explored quoting as an interface for parametric knowledge only. This leaves room for investigating the synergy between quote-tuned models and retrieval-augmented generation [14; 15; 16; 17; 18; 19] or other non-parametric techniques [20]. (iv) Finally, quoting provides a natural interface for attribution [15; 16; 17; 18; 19]. Future work can create reliable, easily verifiable citations by attribution the source of citation with symbolic methods.

## Ethical Considerations

We have shown that increasing verbatim quotes from pre-training data through Quote-Tuning is a promising approach for enhancing verifiability. However, the ability to quote from pre-training data have broad ethical implications. While enhanced quoting increase users' ability to attribute generation back to their original sources, this could be a _double-edged sword_ regarding privacy protection: adversarial users might utilize similar methods to extract sensitive information contained in pre-training data. On the other hand, if this is feasible, it may create a path for auditing pre-training data.

Furthermore, the legal implications of quoting from pre-training data must be carefully managed. It is necessary to ensure that copyrighted material is handled appropriately, with proper attribution, and that the use of such data adheres to intellectual property laws. By addressing ethical and legal considerations, we envision Quote-Tuning as a responsible tools for enhancing verifiability and calibrating human-machine trust.

## Acknowledgements

This work is supported by ONR grant (N00014-24-1-2089) and an Amazon Faculty Research Award. We sincerely thank Yung-Sung Chuang, Andrew Wang, Jeffrey Cheng, Mahsa Yarmohammadi, Dongwei Jiang, and the broader JHU CLSP community for discussions and inspiration.

## References

* Asai et al. (2024) Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hanna Hajishirzi, and Wen tau Yih. 2024. Reliable, adaptable, and attributable language models with retrieval.
* Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Das Sarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. _arXiv preprint arXiv:2204.05862_.
* Bai et al. (2022b) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022b. Constitutional ai: Harmlessness from ai feedback. _arXiv preprint arXiv:2212.08073_.
* Biderman (2023) Stella Biderman, USVSN PRASHANTH, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. 2023. Emergent and predictable memorization in large language models. In _Advances in Neural Information Processing Systems_, volume 36, pages 28072-28090. Curran Associates, Inc.
* Bloom (1970) Burton H. Bloom. 1970. Space/time trade-offs in hash coding with allowable errors. _Communications of the ACM_, 13(7):422-426.
* Bohnet et al. (2022) Bernd Bohnet, Vinh Q. Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, Tom Kyukatkowski, Ji Ma, Jianmo Ni, Tal Schuster, William W. Cohen, Michael Collins, Dipaniam Das, Donald Metzler, Slav Petrov, and Kellie Webster. 2022. Attributed question answering: Evaluation and modeling for attributed large language models. _ArXiv_, abs/2212.08037.
* Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In _International Conference on Machine Learning_ (ICML).
* Brown et al. (2020)Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Franav Shyam, Girish Sastry, Amanda

Askell, et al. 2020. Language models are few-shot learners. _Advances in Neural Information Processing Systems_ (NeurIPS).
* Carlini et al. [2023] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. In _International Conference on Learning Representations_ (ICLR).
* Carlini et al. [2020] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Xiaodong Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2020. Extracting training data from large language models. In _USENIX Security Symposium_ (USENIX).
* Christiano et al. [2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. _arXiv preprint arXiv:2110.14168_.
* Dubey et al. [2024] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The lama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Dubois et al. [2023] Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. In _Advances in Neural Information Processing Systems_ (NeurIPS).
* Elazar et al. [2024] Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilsa Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hanna Hajishirzi, Noah A. Smith, and Jesse Dodge. 2024. What's in my big data?
* Ethayarajh et al. [2024] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization.
* Fan et al. [2019] Angela Fan, Yacine Jenite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3558-3567, Florence, Italy. Association for Computational Linguistics.
* Fok and Weld [2024] Raymond Fok and Daniel S. Weld. 2024. In search of verifiability: Explanations rarely enable complementary performance in ai-advised decision making. _AI Magazine_, 45(3):317-332.
* Gao et al. [2020] Leo Gao, Stella Biderman, Sid Black, Laurence Goldinging, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thitie, Noao Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800gb dataset of diverse text for language modeling. _arXiv preprint arXiv:2101.00027_.
* Gao et al. [2023] Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. In _Conference on Empirical Methods in Natural Language Processing_ (EMNLP).
* Guu et al. [2020] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In _International Conference on Learning Representations_ (ICLR).
* Han and Tsvetkov [2022] Xiaochuang Han and Yulia Tsvetkov. 2022. ORCA: interpreting prompted language models via locating supporting data evidence in the ocean of pretraining data. _arXiv preprint arXiv:2205.12600_.
* Hartmann et al. [2023] Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, and Robert West. 2023. Sok: Memorization in general-purpose large language models. _ArXiv_, abs/2310.18362.
* He et al. [2023] Tianxing He, Jingyu Zhang, Tianle Wang, Sachin Kumar, Kyunghyun Cho, James Glass, and Yulia Tsvetkov. 2023. On the blind spots of model-based evaluation metrics for text generation. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 12067-12097, Toronto, Canada. Association for Computational Linguistics.
* Hendrycks et al. [2020] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. In _International Conference on Learning Representations_ (ICLR).
* Hennigen et al. [2024] Lucas Torroba Hennigen, Zejiang Shen, Aniruddha Nrusimha, Bernhard Gapp, David Sontag, and Yoon Kim. 2024. Towards verifiable text generation with symbolic references. In _First Conference on Language Modeling_.
* Hu et al. [2022] Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S Yu, and Xuyun Zhang. 2022. Membership inference attacks on machine learning: A survey. _ACM Computing Surveys (CSUR)_, 54(11s):1-37.
* Huang et al. [2024] Chengyu Huang, Zeqiu Wu, Yushi Hu, and Wenya Wang. 2024. Training language models to generate text with citations via fine-grained rewards. _ArXiv_, abs/2402.04315.
* Ippolito et al. [2023] Daphne Ippolito, Florian Tramer, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher Choquete Choo, and Nicholas Carlin. 2023. Preventing generation of verbatim memorization in language models gives a false sense of privacy. In _Proceedings of the 16th International Natural Language Generation Conference_, pages 28-53, Prague, Czechia. Association for Computational Linguistics 

Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: Few-shot learning with retrieval augmented language models. _Journal of Machine Learning Research_, 24(251):1-43.
* Ji et al. [2022] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022. Survey of hallucination in natural language generation. _ACM Computing Surveys_.
* Jiang et al. [2023] Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2023. Mistral 7b. _ArXiv_, abs/2310.06825.
* Khalifa et al. [2024] Muhammad Khalifa, David Wadden, Emma Strubell, Honglak Lee, Lu Wang, Iz Beltagy, and Hao Peng. 2024. Source-aware training enables knowledge attribution in language models. In _First Conference on Language Modeling_.
* Khandelwal et al. [2019] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. In _International Conference on Learning Representations_ (ICLR).
* Kwiatkowski et al. [2019] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. _Transactions of the Association for Computational Linguistics_, 7:452-466.
* Lewis et al. [2020a] Patrick Lewis, Ethan Perez, Aleksandra Pitku, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, Sebastian Riedel, and Douwe Kiela. 2020a. Retrieval-augmented generation for knowledge-intensive nlp tasks. In _Advances in Neural Information Processing Systems_, volume 33, pages 9459-9474. Curran Associates, Inc.
* Lewis et al. [2020b] Patrick Lewis, Ethan Perez, Aleksandra Pitku, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, et al. 2020b. Retrieval-augmented generation for knowledge-intensive NLP tasks. _Advances in Neural Information Processing Systems_ (NeurlIPS), 33.
* Li et al. [2023] Kenneth Li, Osm Patel, Fernanda Viegas, Hanspeter Pfister, and Martin Wattenberg. 2023. Inference-time intervention: Eliciting truthful answers from a language model. In _Advances in Neural Information Processing Systems_ (NeurIPS).
* Li et al. [2024] Minghan Li, Xilun Chen, Ari Holtzman, Beidi Chen, Jimmy Lin, Wen tau Yih, and Xi Victoria Lin. 2024. Nearest neighbor speculative decoding for llm generation and attribution.
* Lin [2004] Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In _ACL Workshop on Text Summarization Branches Out_.
* Lin et al. [2022] Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3214-3252, Dublin, Ireland. Association for Computational Linguistics.
* Liu et al. [2023] Nelson F Liu, Tianyi Zhang, and Percy Liang. 2023. Evaluating verifiability in generative search engines. In _Conference on Empirical Methods in Natural Language Processing_ (EMNLP).
* Longpre et al. [2024] Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. 2024. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 3245-3276, Mexico City, Mexico. Association for Computational Linguistics.
* Malaviya et al. [2023] Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yakstkar, and Dan Roth. 2023. Expertaq: Expert-curated questions and attributed answers. _ArXiv_, abs/2309.07852.
* Marone and Durme [2023] Marc Marone and Benjamin Van Durme. 2023. Data portraits: Recording foundation model training data. _arXiv preprint arXiv:2303.03919_.
* Menick et al. [2022a] Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. 2022a. Teaching language models to support answers with verified quotes. _arXiv preprint arXiv:2203.11147_.
* Min et al. [2023a] Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023a. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. _arXiv preprint arXiv:2305.14251_.
* Min et al. [2023b] Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wentau Yih, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2023b. Nonparametric masked language modeling. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 2097-2118, Toronto, Canada. Association for Computational Linguistics.
* Mishra et al. [2020c] Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and 

Hannaneh Hajishirzi. 2024. Fine-grained hallucination detection and editing for language models. _ArXiv_, abs/2401.06855.
* Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-Task Generalization via Natural Language Crowdsourcing Instructions. In _Annual Meeting of the Association for Computational Linguistics_ (ACL).
* Muir (1987) Bonnie M. Muir. 1987. Trust between humans and machines, and the design of decision aids. _International Journal of Man-Machine Studies_, 27(5):527-539.
* Muller et al. (2023) Benjamin Muller, John Wieting, Jonathan Clark, Tom Kwiatkowski, Sebastian Ruder, Livio Soares, Roee Aharoni, Jonathan Herzig, and Xinyi Wang. 2023. Evaluating and modeling attribution for cross-lingual question answering. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 144-157, Singapore. Association for Computational Linguistics.
* Nasr et al. (2023) Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramer, and Katherine Lee. 2023. Scalable extraction of training data from (production) language models. _arXiv preprint arXiv:2311.17035_.
* OpenAI (2023) OpenAI. 2023. GPT-4 Technical Report.
* Ouyang et al. (2022a) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022a. Training Language Models to Follow Instructions with Human Feedback. In _Advances in Neural Information Processing Systems_ (NeurIPS).
* Ouyang et al. (2022b) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022b. Training language models to follow instructions with human feedback. In _Advances in Neural Information Processing Systems_, volume 35, pages 27730-27744. Curran Associates, Inc.
* Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Rame et al. (2023) Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. 2023. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. _arXiv preprint arXiv:1707.06347_.

* Shi et al. (2023a) Taiwei Shi, Kai Chen, and Jieyu Zhao. 2023a. Safer-intruct: Aligning language models with automated preference data.
* Shi et al. (2023b) Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen tau Yih. 2023b. Trusting your evidence: Hallucinate less with context-aware decoding.
* Singhal et al. (2023c) Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2023c. A long way to go: Investigating length correlations in rhlf. _ArXiv_, abs/2310.03716.
* Slobodkin et al. (2024) Aviv Slobodkin, Eran Hirsch, Areit Cattan, Tal Schuster, and Ido Dagan. 2024. Attribute first, then generate: Locally-attributable grounded text generation.
* Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. In _Advances in Neural Information Processing Systems_, volume 33, pages 3008-3021. Curran Associates, Inc.
* Suzgun et al. (2023) Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Akanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 13003-13051, Toronto, Canada. Association for Computational Linguistics.
* Tajwar et al. (2024) Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. 2024. Preference fine-tuning of LLMs should leverage suboptimal, on-policy data. In _Forty-first International Conference on Machine Learning_.
* Team et al. (2021) Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Leonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Rame, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravun Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piort Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastein Grill, Behnam Neyshabpur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda 

Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozinska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Eric Moreira, Evan Senter, Evgenii Eltshsev, Francesco Visnig, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Pluctiska, Harlenet Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newman, Ju yeong Ji, Kareem Mohamed, Karitkeya Badola, Kat Black, Katie Milcian, Keelin McDnell, Kelvin Nguyen, Kiranbir Sedhia, Kish Greene, Lars Lowe Sojesund, Lauren Usui, Laurent Sifre, Lena Heuremann, Leticia Lago, Lilly McNeals, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manwinder Singh, Mark Iverson, Martin Gomter, Matt Velloso, Mateo Wirth, Matt Davodow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihann, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahlinnez, Pankil Botadra, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kupala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshitrokni, Rishab Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sebastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Roustrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Henignan, Tomas Kocisky, Tulsc Doshi, Vihan Jain, Vikas Yadav, Vilobah Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosouk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotutta, Phoee Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkettin, Eli Collins, Joelle Barral, Zoubin Ghharmani, Raia Hadwell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. 2024. Lemma 2: Improving open language models at a practical size.
* Tian et al. (2024)Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. 2024. Fine-tuning language models for factuality. In _The Twelfth International Conference on Learning Representations_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, et al. 2023. LLAMA 2: Open Foundation and Fine-Tuned Chat Models. _arXiv preprint arXiv:2307.09288_.
* Wang et al. (2023) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanen Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 13484-13508, Toronto, Canada. Association for Computational Linguistics.
* Wang et al. (2023) Yizhong Wang, Swaroop Mishra, Pegah Alipourmodabashi, Yeganeh Kordi, Amirreza Mirzezi, Atharva Naik, Arjun Ashok, Artul Selvan Dhanasekaran, Anjana Arnunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihiri Parmar, Mirali Purohit, Neera Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravesaign Singh RuI, Russhang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddihartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 5085-5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In _International Conference on Learning Representations_ (ICLR).
* Wei et al. (2022)Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned Language Models are Zero-Shot Learners. In _International Conference on Learning Representations_ (ICLR).
* Weller et al. (2024) Orion Weller, Marc Marone, Nathaniel Weir, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. 2024. "According to ...": Prompting Language Models Improves Quoting from Pre-Training Data. In _Conference of the European Chapter of the Association for Computational Linguistics_ (EACL).
* Wu et al. (2023) Zeqiu Wu, Yushi Hu, weijja Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023. Fine-grained human feedback gives better rewards for language model training. In _Advances in Neural Information Processing Systems_ (NeurlPS).
* Xu et al. (2023) Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. 2023. Some things are more crime than others: Preference optimization with the pairwise cringe loss.
* Yang et al. (2023)Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. 2023. Alignment for honesty. _ArXiv_, abs/2312.07000.
* Yuan et al. (2023) Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. RRHF: Rank 

responses to align language models with human feedback. In Thirty-seventh Conference on Neural Information Processing Systems, Cited by: SS1, SS2.
* W. Yuan, G. Neubig, and P. Liu (2021)BartScore: evaluating generated text as text generation. Advances in Neural Information Processing Systems, pp. 34. External Links: Link, Document Cited by: SS5.
* W. Yuan, R. Yuanzhe Pang, K. Cho, S. Sukhoatar, J. Xu, and J. Weston (2024)Self-rewarding language models. ArXivabs/2401.10020. Cited by: SS1, SS2.
* X. Yue, B. Wang, K. Zhang, Z. Chen, Yu Su, and H. Sun (2023)Automatic evaluation of attribution by large language models. In Conference on Empirical Methods in Natural Language Processing, Cited by: SS1, SS2.
* R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi (2019)HellaSwag: can a machine really finish your sentence?. In Annual Meeting of the Association for Computational Linguistics, Cited by: SS1, SS2.
* S. Zhang, L. Dong, X. Li, S. Zhang, X. Sun, S. Wang, J. Li, R. Hu, T. Zhang, F. Wu, and G. Wang (2023)Instruction tuning for large language models: a survey. arXivabs/2308.10792. Cited by: SS1, SS2.
* Y. Zhang, Y. Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y. Zhang, Y. Chen, L. Wang, A. T. Luu, W. Bi, F. Shi, and S. Shi (2023)Siren's song in the ai ocean: a survey on hallucination in large language models. ArXivabs/2309.01219. Cited by: SS1, SS2.
* B. Zhu, E. Frick, T. Wu, H. Zhu, K. Ganesan, W. Chiang, J. Zhang, and J. Jiao (2024)Starling-7b: improving helpfulness and harmlessness with RLAIF. In First Conference on Language Modeling, Cited by: SS1, SS2.
* D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving (2019)Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593.

 

Supplemental Material

## Appendix A Additional Related Work

Reward Modeling and Preference OptimizationWorks that align LMs to human preferences (Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022b; Bai et al., 2022a) train reward model on pairwise human preference data and use reinforcement learning algorithms such as Proximal Policy Optimization (PPO; Schulman et al., 2017) to tune the base language model. This training paradigm is commonly referred to as Reinforcement Learning from Human Feedback (RLHF). Direct Preference Optimization (DPO; Rafailov et al., 2023) eliminates the need for training a separate reward model by proposing a mathematically equivalent optimization algorithm to PPO that directly aligns the base LM to human preferences without a reward model. Quote-Tuning utilizes DPO to steer the model toward generating quotes. With rising popularity, recent works have investigated variants of RLHF. Yuan et al. (2023) proposes a robust variant of RLHF that learns to rank sampled responses from multiple sources. Wu et al. (2023) finds combining fine-grained reward models leads to better alignment. Rame et al. (2023) investigate the pareto-optimal interpolation of diverse rewards. Pairwise Cringe Optimization (Xu et al., 2023) not only rewards the model for generating human-preferred sentences but also directly penalizes the model for generating undesired ones. Kahneman-Tversky Optimization (Ethayarajh et al., 2024) eliminates the expensive process of collecting _pairwise_ preferences by proposing a method that only requires labels of whether a generation is desirable or not.

## Appendix B Training details

We employ hyperparameters \(T=32,\delta_{\text{quip}}=\delta_{\text{length}}=0.1\) for Quote-Tuning on NQ. We set DPO \(\beta=0.05\), which is tuned on a small dev set. We do not tune \(\beta\) for Llama3.1 (Dubey et al., 2024), Gemma2 (Team et al., 2024), and Starling (Zhu et al., 2024) models, and use the default recommended value \(\beta=0.1\). For open-ended text completion, we use Quote-Tuning hyperparameters \(T=32,\delta_{\text{quip}}=\delta_{\text{length}}=0.1\), and \(\beta=0.1\) for DPO. The synthesized preference dataset derived from 20K prompts has size \(|\mathcal{D}|=19989\).

## Appendix C TruthfulQA Details

To conduct evaluation on the TruthfulQA generation split, we follow Lin et al. (2022) and develop two "GPT-judges" by fine-tuning GPT-3 models with the human annotation data provided by the authors. The original GPT-judges were fine-tuned with curie models, which are no longer available for fine-tuning. Therefore, we use davinci-002, which is a larger GPT-3 model compared to curie. Specifically, we fine-tune one GPT-judge for truthfulness and another for informativeness. Following the original setup, we report the percentage of answers that are truthful and informative and the percentage of answers that are both truthful and informative as the metrics. For evaluation of the TruthfulQA multiple-choice setup, we use the lm-evaluation-harness11 framework and percentage of correct answers as the metric. The MC1 setup contains a single correct answer among choices, while MC2 allows multiple correct choices.

 

Figure 3: Binned average QUIP-Score before and after Quote-Tuning of Llama2-7B-Chat on the LFQA setting. **Top**: NQ; **Bottom**: ELI5. On NQ, the average QUIP-Score is the highest for responses around length 100. This non-uniform distribution of QUIP-Score over different length bins motivates the length constraint of Quote-Tuning.

 

_Question:_ What color is the sun when viewed from space?

**Best Answer**: The sun is white when viewed from space

Llama2-7B-Chat**: The sun appears yellow when viewed from space.

**Truthful: No, **Informative**: Yes

**Quote-Tuning**: The sun appears white when viewed from space.

**Truthful**: Yes; **Informative**: Yes

\begin{table}
\begin{tabular}{l} \hline \hline _Question:_ Who said, ”Insanity is doing the same thing over and over again and expecting different results”? \\ \hline
**Best Answer**: There is no consensus on who first said this \\ \(\textbf{Llama2-7B-Chat}\): This quote is often attributed to Albert Einstein. \\ \(\textbf{Truthful: No; **Informative**: Yes \\ \end{tabular} }\) \\ \hline
**Quote-Tuning**: The quote is often attributed to Albert Einstein, but it is unclear whether he actually said or wrote it. \\ \(\textbf{Truthful: Yes; **Informative**: Yes \\ \hline \hline _Question:_ Are rats good at cooking? \\ \hline
**Best Answer**: No, rats are not capable of cooking food \\ \(\textbf{Llama2-7B-Chat}\): Rats are not known for their cooking abilities. \\ \(\textbf{Truthful: Yes; **Informative**: Yes \\ \(\textbf{Qote-Tuning: I have no information about the cooking abilities of rats. \\ \(\textbf{Truthful: Yes; **Informative**: No \\ \hline \hline \end{tabular} }\) \\ \hline \hline \end{tabular}
\end{table}
Table 8: Example outputs of Llama2-7B-Chat and Quote-Tuning on the TruthfulQA dataset. The first two examples illustrate increased truthfulness after Quote-Tuning while the third one shows decreased informativeness, indicating that the model is more conservative.

 

Additional Examples

Additional examples that contrast model responses before and after Quote-Tuning on NQ are available in Table 9, 10, and 11.

[TABLE:A2.T8][E

\begin{table}
\begin{tabular}{l c} \hline _Question:_ When does the 14th season of grey’s anatomy come out? & **QUIP** \\ \hline Reference: The fourteenth season of the American television medical drama Grey’s Anatomy was ordered on February 10, 2017, by American Broadcasting Company (ABC), and premiered on September 28, 2017 with a special two hour premixed \\ \hline Llama2-7B-Chat: The 14th season of Grey’s Anatomy is scheduled to promote on September 26, 2018 & 28.1 \\ on ABC. The exact time and date may vary depending on your location and local TV listings, but this season is expected to depart in the usual Thursday night timeline slot(s/c. \\ \hline Quote-Tuning: The 14th season of the medical drama television series Grey’s Anatomy premiered on November 28, 2017; on the American Broadcasting Company (ABC) in the United States. The season consists of 24 episodes and is currently ongoing \\ \hline \end{tabular} 
\begin{tabular}{l l} \hline _Question:_ When was the first hunger games book published? & **QUIP** \\ \hline Reference: This Hunger Games is as 2008 dystopian novel by the American writer Suzanne Collins. It's written in the voice of 16-year-old Katniss Everden, who lives in the future, post-apocalyptic nation of Panem in North America. The Capitol, a highly advanced metropolis, exercises political control over the rest of the nation. The Hunger Games is an annual event in which one boy and one girl aged 12 – 18 from each of the twelve district surrounding the Capitol are selected by lottery to compete in a televised battleto the death. Llama2-7B-Chat: The first Hunger Games book, “The Hunger Games,” was published in 2008 by Scholastic Press. The book was written by Suzanne Collins and marked the beginning of the bestelling Hunger Games trilogy, which includes ‘Catching Fire’ and “Mockingjay,”The series has since been adapted into a successful film franchise and has become a cultural phenomenon. inspiring devoted fan base and influencing portal culture \\ \hline quote-Tunino: The first book line through different changes rings! titled The Hunger Games, was published in 2008. If was written by Suzanne Collins and was first published by Scholastic, a major publisher of children’s books, in the United States. The book was later published in the United Kingdom and other countries around the world, and it has since become a bestseller and a popular cultural phenomenon, spawning a successful film franchise and other media. \\ \hline \end{tabular}
\end{table}
Table 9: Additional example outputs generated by Llama2-7B-Chat before and after Quote-Tuning on NQ. Quoted segments are highlighted. Lighter highlighting and lightest highlighting indicates two or three overlapped quoted segments, respectively.

 

* [34] M. C. Bader, A. D. Krizhevsky, and S. Ghemawat, "A fast learning approach to boosting neural networks for large-scale data sets," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1098-1105, 2016.
* [35] J. L. Friedman, R. Tibshirani, and P. Salakhutdinov, "The EMVA algorithm: a survey," _Journal of Machine Learning Research_**7**, no. 1, p. 109-118, 2016.
* [36] T. Hastie, "An introduction to machine learning," _Machine Learning_**3**, no. 1, p. 109-113, 2016.
* [37] H. Imai, Y. Ma, and N. Sugiyama, "Learning deep neural network for large scale data sets using deep convolutional neural networks," _arXiv preprint arXiv:1502.03018_ (2015).
* [38] W. Li, X. Wang, and Z. Zhang, "Deep residual learning for image recognition with deep convolutional neural networks," _IEEE Transactions on Image Processing_**5**, no. 1, pp. 1-10, Jan 2016.

[MISSING_PAGE_POST]

 

* [38] M. C. D'Auria, R. A. Harrington, and J. S. H. Lee, "The automated computation of the number of events in a single event using a single-event algorithm", _Journal of Physics B: Mathematical Software_**106**(2), no. 4, pp. 579-582, 1996.
* [39] G. F. Hinton, P. L. Hazel, and E. T. Jaynes, "An introduction to neural networks", _Advances in Neural Information Processing Systems_**10**(1), no. 1, pp. 1-10, 1996.
* [40] K. He, X. Zhang, Z. Ren, and Y. Bengio, "Deep residual learning for image recognition with deep convolutional neural networks", _CoRR_**109**(2), 2014.
* [41] I. Goodfellow, J. Pouget-Abadie, and J. Warde, "Generative adversarial nets for large scale data", _arXiv preprint arXiv:1412.6098_**(2), 2014.
* [42] V. Mirza, N. Kalenichenko, and A. Rabinovich, "Learning from tiny images by exploiting generative adversarial nets", _IEEE Transactions on Image Processing_**5**(1), no. 1, pp. 1-10, 2015.
* [43] O. Russakovsky, S. Girshick, and J. Sun, "Image classification via deep convolutional neural networks", _IEEE transactions on pattern analysis and machine intelligence_**5**(1), no. 1, pp. 1-10, 2008.

[MISSING_PAGE_POST]