Learning to Generate Answers with Citations via Factual Consistency Models (2024)

Abstract: Large Language Models (LLMs) frequently hallucinate, impeding their reliability in mission-critical situations. One approach to address this issue is to provide citations to relevant sources alongside generated content, enhancing the verifiability of generations. However, citing passages accurately in answers remains a substantial challenge. This paper proposes a weakly-supervised fine-tuning method leveraging factual consistency models (FCMs). Our approach alternates between generating texts with citations and supervised fine-tuning with FCM-filtered citation data. Focused learning is integrated into the objective, directing the fine-tuning process to emphasise the factual unit tokens, as measured by an FCM. Results on the ALCE few-shot citation benchmark with various instruction-tuned LLMs demonstrate superior performance compared to in-context learning, vanilla supervised fine-tuning, and state-of-the-art methods, with an average improvement of 34.1, 15.5, and 10.5 citation F1 points, respectively. Moreover, in a domain transfer setting we show that the obtained citation generation ability robustly transfers to unseen datasets. Notably, our citation improvements contribute to the lowest factual error rate across baselines.

heading: 1 Introduction
text: Large language models (LLMs; Brown et al., 2020) have showcased remarkable capabilities in information-seeking tasks. However, concerns persist regarding their trustworthiness, stemming from their opaque nature and factual hallucinations in their responses (Zhang et al., 2023; Mallen et al., 2023). One proposed solution involves the production of citations to retrieved text passages alongside LLM-produced statements (Liu et al., 2023; Rashkin et al., 2023). By integrating citations into generated responses, readers can easily verify
* Work done while interning at Amazon Web Services.
LLMs statements. The ability to accurately produce citations enables LLMs to generate responses more closely aligned with cited sources, alleviating hallucinations (Gao et al., 2023b; Yue et al., 2023).
Despite its significance, accurate citation generation proves to be challenging. State-of-the-art LLMs, such as ChatGPT (OpenAI, 2023), and commercial generative chat engines, such as Bing Chat, produce accurate citations only for less than 60% of generated statements (Gao et al., 2023b; Liu et al., 2023). Figure 1 illustrates typical citation errors, including hallucinated statements or citations associated with incorrect claims. Hence, there is a necessity to train LLMs to generate citations accurately. This paper focuses on teaching LLMs to generate citations for retrieval-augmented longform question answering (LFQA), tackling two main challenges: the scarcity of high-quality labeled data at scale and the risk of compromising original language and generalization capacities during fine-tuning for citation generation.
11876
To address these challenges, we present CaLF (Citation Learning via Factual Consistency Models), a fine-tuning strategy that enables LLMs to learn citation generation without sacrificing their language capacities. As illustrated in Figure 2, the cornerstone of our approach is factual consistency models (FCMs; Kryscinski et al., 2020, inter alia), initially introduced as a neural measure of consistency between a claim and its context. We use FCMs to gauge whether cited passages support a generated statement. Our method incorporates FCMs in two designs. Firstly, we propose a weakly-supervised training strategy, where an LLM generates diverse responses with citations, an FCM filters high-quality citation data, and the LLM is fine-tuned on the filtered data. Secondly, we utilize focused learning to adjust the loss contribution of each answer token based on its factual relevance, as measured by an FCM. The intuition is to have the LLM concentrate on tokens related to factual knowledge during fine-tuning, minimizing the impact on its original language capacities.
We evaluate CaLF on various LLMs, including Llama2 (Touvron et al., 2023), Mistral-Instruct, and MistralOrca (Jiang et al., 2023). On the ALCE automatic few-shot citation evaluation benchmark (Gao et al., 2023b), CaLF enhances citation metrics over the in-context learning and baseline finetuning, with an average improvement of 34.1 and 15.5 F1, respectively, while maintaining fluency and correctness. All LLMs trained via CaLF , outperform the state-of-the-art model Self-Rag (Asai et al., 2024) and ChatGPT (OpenAI, 2023), with an average improvement of 24.8 and 10.5 citation F1 points, respectively. Domain transfer experiments, testing citation quality on a dataset different from the training dataset, highlight CaLF’s ability to generalize across tasks and domains. Additionally, on the FactScore biography generation benchmark (Min et al., 2023), CaLF demonstrates an improvement in factuality. Finally, human evaluation results indicate that CaLF yields more preferable answers compared to the fine-tuning baseline.

heading: 2 Related Work
text: LFQA with Citations To produce citations alongside a response, the generation can be conditioned on a few high-quality in-context examples with embedded citations (Gao et al., 2023b; Li et al., 2023). In contrast, Gao et al. (2023a); Bohnet et al. (2022) propose to edit an already generated

heading: Diverse Answer Generation
text: response post-hoc to attribute to retrieved sources, causing computational overhead during inference. Alternatively, proprietary work has explored human preference learning for citation production (Menick et al., 2022; Thoppilan et al., 2022; Nakano et al., 2021). Reinforcement learning from human feedback is expensive and typically more brittle than supervised fine-tuning. Very recently, Asai et al. (2024) incorporate critique tokens, which serve as a feedback and citation mechanism, into GPT-4 obtained instruction-tuning data. These tokens allow for flexible retrieval-augmented generation with citations. In contrast to previous work, CaLF incorporates openly available FCMs as critique models into the fine-tuning process of already instructiontuned LLMs while not modifying inference, maintaining efficiency.
Factual Consistency Models FCMs assess whether all the factual information in a claim is consistent w.r.t to the information conveyed in its grounding text. The task shares strong similarities to natural language inference (NLI) (Bowman et al., 2015; Dagan et al., 2005). However, in contrast to NLI, factual consistency is not evaluated on subjec-
tive or opinionated statements. Work that explores FCMs for improving the generation is mainly constrained to summarization. Aharoni et al. (2023) filter samples from a large-scale dataset according to an FCM while Muller et al. (2023) use an FCM to rerank and select source passages for crosslingual summarization. Tian et al. (2023) incorporate FCMs to improve factuality via direct preference optimization (Rafailov et al., 2023). Instead of using FCMs, Deng et al. (2023) improve the factuality of generations by measuring the cosine similarity between a token’s embedding and relevant knowledge to adjust a token’s loss contribution. Our work is the first to explore the use of model explainability mechanisms, namely Shapley values (Shapley, 1953), for incorporating FCMs directly into the fine-tuning process of an LLM.

heading: 3 Preliminaries
text: Task description. Given an information-seeking question q, such as shown in Figure 1, the task is to generate a long-form answer ŷ = {s1, · · · , sn}, consisting of sentences si, conditioned on passages P retrieved from a knowledge base. A long-form answer with citations needs to ensure that one or multiple relevant passages Ci ⊆ P are cited in each generated sentence si (indicated by brackets with a passage index, e.g. “[1]” for p1), such that the generated information in si follows from the cited passages Ci. This task definition strictly requires all facts to originate from the retrieved passages, ensuring that ŷ is fully verifiable by P . We further assume the availability of few-shot training samples (q, P, y) ∈ D to learn citation production. Generation with LLMs. In this work, we are interested in using LLMs to generate long-form answers with citations. An answer ŷ is generated autoregressively, computing the next token distributions conditioned on the question q, retrieved passages P , and the answer generated so far ŷ<t:∏|ŷ|
t=1 pθ(ŷt | q, P, ŷ<t), with θ being the parameterization of the LLM. Consequently, a model is updated on a gold answer y by minimizing the negative log-likelihood loss (NLL):
LNLL = − 1
| y |
|y|∑
t=1
log pθ(yt | q, P, y<t), (1)
Factual Consistency Models. An FCM ϕ measures the factual consistency between a sentence si and a collection of citations Ci: o = ϕ(si, Ci),
with o ∈ {0, 1} being the binary consistency prediction of the FCM. While FCMs are often modelled via modified NLI models that output a binary prediction directly (Gekhman et al., 2023; Utama et al., 2022), AlignScore (Zha et al., 2023) produces a single calibrated continuous value o = pϕ(consistent | si, Ci) ∈ [0, 1]. In our scenario, such a scalar is beneficial for computing Shapley values and for controlling the consistency’s strictness. We refer to Mosca et al. (2022) for background on Shapley values and the SHAP framework in the context of NLP.

heading: 4 Citation Learning via FCMs
text: CaLF is a fine-tuning strategy for producing longform answers with citations. CaLF’s main assumption is that an FCM ϕ can be leveraged as a supervision signal to improve citation quality of an LLM F via an iterative training procedure, as illustrated in Figure 2 and Algorithm 1. CaLF alternates between two modes in each iteration k. First, it generates weakly-supervised training data D̂k to enrich the training corpus by sampling diverse answers from a fine-tuned LLM Fk−1 and filters them via the FCM ϕ (Sec. 4.1). Second, it fine-tunes the LLM Fk−1 on D̂k +D with a modification of the NLL objective, which re-weights the loss contribution of individual tokens according to their importance for ensuring factual consistency, as measured by the FCM ϕ (Sec. 4.2). The number of iterations is determined dynamically by stopping when the proportion of filtered examples over the candidates does not improve between iterations or once the maximum number of iterations is reached.

heading: 4.1 Answer Generation for Training
text: To generate weakly supervised training data (x, P, ŷ) ∈ D̂k with answer ŷ containing citations to passages P , we assume the availability of a collection of information-seeking questions x ∈ X and a list of atomic facts A expected in an answer to x. For questions X , an LLM generates a collection of answer candidates Ŷ , conditioned on retrieved passages P , selected by an out-of-thebox retrieval system R. As indicated in Algorithm 1, we produce answer candidates using either the fine-tuned model Fk−1 from the previous iteration, or, in the case k = 0, we use in-context prompting with the few-shot examples D. We focus on the generation of diverse answer candidates to enrich the weakly supervised training data. First, we
Algorithm 1 The training procedure of CaLF. Input: LLM F ; FCM ϕ; Retriever R; questions X and an-
swer facts A; few-shot examples D; Iterations K. Output: Fine-tuned LLM FK ; Citation Data D̂K . 1: U0 ← {⊕si∈y(Norm(SHAPϕ(si))) | (x, P, y) ∈ D} 2: W0 ← {Align(U, Tϕ(ŷ), TF (ŷ) | U ∈ U0} 3: P ← {R(x,KB) | x ∈ X} 4: k ← 0 5: while k ≤ K and |D̂k−1||Ŷk−1| ≥ | D̂k−2 Ŷk−1
| do 6: if k = 0 then ▷ Data Generation (§4.1) 7: Ŷk ← Diverse Sampl.IC(F ,X , P,D) 8: else 9: Ŷk ← Diverse Sampl.(Fk−1,X , P )
10: end if 11: D̂k ← {(x, P, ŷ) | ŷ ∈ Ŷk ∧Q(ŷ, A, P ) = 1} 12: Uk ← { ⊕
si∈ŷ Norm(SHAPϕ(si))) | (x, P, ŷ) ∈ D̂k}
13: Wk ← {Align(Tϕ(ŷ), TF (ŷ) | U ∈ Uk} 14: Fk ← Update F via FL ▷ Focused Learning (§4.2) 15: ∇LFL(D + D̂k,W0 +Wk) 16: k ← k + 1 17: end while
use sampling strategies such as nucleus sampling (Holtzman et al., 2020), temperature scaling (Guo et al., 2017), and diverse beam search (Vijayakumar et al., 2018). Second, we consider citation replacements in answer sentences si in ŷ to diversify the answer candidates beyond the output of the LLM. Specifically, for generated citations Ci in a sentence si, we generate two citation replacements, sampled according to the passage probability measured via the retriever R since the direct computation of ϕ(si, Ci) over all citation options is infeasible.
Each answer candidate ŷ ∈ Ŷ is subsequently filtered via an answer quality assurance function Qϕ : ŷ → {0, 1}, measured via the FCM ϕ, to obtain D̂t with weakly-supervised cited answers ŷ:
Q =    1 if Citation-Recall(ŷ, C, ϕ) > Θ ∧ Citation-Precision(ŷ, C, ϕ) > Θ ∧ Correctness(ŷ, A, ϕ) > Θ
0 else,
with C being the citations assigned to sentences in ŷ, and Θ being a dynamically determined quality threshold that adjusts such that the size D̂t is above a minimum viable size. Citation-Recall(ŷ, C) = 1 n ∑ si∈ŷ ϕ(si, Ci) measures the factual coverage of citations, Citation-Precision(ŷ, C) = 1 |C| ∑ Ci∈C 1 |Ci| ∑ ci,j∈Ci max(ϕ(si, ci,j), 1 − ϕ(Ci\{ci,j},si)) measures the relevance of citations, and Correctness(ŷ, A) measures the proportion of facts A covered in ŷ. These defini-
tions largely align with the ones in the benchmark of Gao et al. (2023b) for evaluating citations.

heading: 4.2 Focused Learning for Factual Consistency
text: To emphasize the learning of producing citations, we measure the relevance for each token in an answer for ensuring the factual consistency between y and the retrieved passages P . We subsequently modify the NLL loss computation (see Eq. 1) for the instruction-tuning of the LLM F by re-weighting the loss contribution of the t−th token according to relevance weights wt ∈ W :
LFL = − 1
| y |
|y|∑
j=1
wtlog pθ(yt | q, P, y<t). (2)
In contrast to NLL where the loss is computed as the arithmetic mean over the token-level loss, our focused learning loss LFL emphasizes tokens which are considered of higher importance for ensuring
factual consistency between a generated statement and the cited passages according to an FCM ϕ.
The mechanism for obtaining the relevance weights W is illustrated in Figure 3. We first compute and normalize Shapley values for the factual consistency prediction between y and its cited passages using the FCM ϕ, obtaining U . These normalized relevance values for the tokens of the FCM are subsequently mapped to tokens of the LLM F via an alignment algorithm. As seen in Figure 3, given a generated statement such as “In the 1968 film Planet of the Apes, Galen was played by Wright King.”, the relevance weights W consider factual tokens such as Wright and King more important than of and was, emphasizing units of information important for factually accurate citation.
Computation of Token Relevance. We first compute relevance weights U over the FCM ϕ by computing Shapley values for the factual consistency prediction between an answer’s sentence si and its citations Ci: oi = ϕ(si, Ci). Shapley values assign importance scores wt to each feature (here token) in si concerning prediction oi. Since Shapley values distribute the prediction score oi along all the tokens of sentence si, the value of oi and the length of si impacts the scale of assigned values, as shown in Figure 3, potentially biasing the loss towards shorter sentences and amplifying idiosyncrasies of the FCM. Thus, we normalize the assigned Shapley values for each sentence via min-max normalization Norm(Ui): { ut−min(Ui)max(Ui)−min(Ui) | ut ∈ Ui}, with Ui being token weights for sentence si. Thus, the token with the highest and lowest Shapley value is assigned a relevance score of 0 and 1, respectively.1 The computation of weights U for a given response y can subsequently be summarized as:
U = ⊕ni=1Norm(SHAPϕ(si)) (3)
with ⊕ being a concatenation operator. Since citation tokens do not bear any semantic meaning themselves, they are excluded from the computation of ϕ(si, Ci) and are thus not yet contained in U . Therefore, we further insert a weight of 1 for each citation token of y into U .
Feature Importance Mapping. Since the obtained relevance weights U are specific to the tokenizer used for ϕ, an LLM that uses a different
1The exploration of alternative normalization functions is left to future work.
tokenization may not directly apply U to re-weight the loss LFL (c.f. Eq. 2). To this end, we define an alignment function that maps the FCM token weights U to LLM token weights W for the same sequence y. The alignment function first maps the shortest possible token span yϕ,l:l+m from the FCM’s tokenizer Tϕ to the span yF ,p:p+q from the LLM’s tokenizer TF , with j, l ≥ 1.2 The relevance score for each LLM token in yF ,p:p+q is computed as the average relevance score over the aligned FCM span yϕ,l:l+m: Wp:p+q = { ∑ l≤t<m(ut) |yϕ,l:l+m| | yt ∈ yF ,p:p+q} In the example of Figure 3, the weight for the LLM’s token McDowall is computed as the average over the weights for the three aligned FCM tokens (McD, ow, all). Further, since both LLM tokens K and ing are aligned to the single token King of the FCM, both tokens are set to the same relevance weight according to our algorithm.

heading: 5 Evaluation
text: Datasets & Metrics. We conduct experiments on long-form QA datasets of the ALCE citation benchmark (Gao et al., 2023b), namely their version of ASQA (Stelmakh et al., 2022) and ELI5 (Fan et al., 2019). We further evaluate models on BIO (Min et al., 2023). For our domain transfer experiments, we further consider Hagrid (Kamalloo et al., 2023) as a source for training. Since Hagrid’s answers are generated by GPT-4 without annotations for factual coverage, we do not use it for evaluation as a target. For training, ALCE contains | D |= 4 cited gold instances. For Hagrid, we randomly sample 4 instances for CaLF. Details are in Appendix A.1.
We follow the official metrics and nomenclature of Gao et al. (2023b) to measure correctness (via EM Recall), fluency (via MAUVE (Pillutla et al., 2021)), and citation F1 (via an NLI-trained T5-11B model (Honovich et al., 2022)). We further measure Rouge-L (Lin, 2004) and introduce a more strict variation of their correctness metric: passage-grounded correctness (Correct. in P), which only considers information from responses which are supported by the retrieved passages P . Subsequently, this metric ignores factual content produced by an LLM’s parametric memory if not explicitly derivable from the retrieved passages, isolating factual grounding from a model’s parametric memory. We use FactScore (Min et al., 2023) to evaluate the biographies of BIO. Detailed metric
2Note that we implement this mapping function tailored to the specific tokenizers used by our models, see Appendix A.2.
descriptions (e.g. citation recall and precision) and results are in Appendix A.3.
Experimental Setup. Following recommendations for weakly supervised learning (Zhu et al., 2023) and few-shot learning (Alex et al., 2021), we do not consider a validation set for hyperparametertuning, representing real-world scenarios more accurately. Due to computational constraints, we use LoRA (Hu et al., 2022) for parameter-efficient finetuning of CaLF and our fine-tuning baselines. We use Alignscore as our FCM ϕ in all experiments unless otherwise mentioned. A threshold is used to map Alignscore’s output o into a binary prediction for Q. Alignscore is substantially different from the FCM model used for evaluating citations, using a different architecture and training data (c.f. App. A.2). ASQA, Hagrid, and BIO use Wikipedia as their underlying knowledge base while ELI5 uses CommonCrawl. We use the same retrievers as Gao et al. (2023b) and Asai et al. (2024) to maintain comparability. Further implementation details are in Appendix A.2.
Baselines. We compare CaLF on identical instruction-tuned LLMs against both in-context prompting and few-shot fine-tuning (Few-shot FT) via Eq. 1, trained on D. In our domain transfer experiments, the entire 335 training samples of Hagrid are used to train the fine-tuning baseline (FT). Furthermore, we consider state-of-the-art models, namely the most powerful in-context prompted baselines from Gao et al. (2023b), ChatGPT (gpt-3.5-turbo-0301) and GPT-4 (gpt-4-0613; 8K context window). Due to context length limi-
tations, in-context prompting uses 2 randomly sampled instances. We further compare against the best results of AGREE (Ye et al., 2024) which use PaLM 2’s text-bison-001. Finally, we evaluate against open-source models, including SelfRAG 7B (Asai et al., 2024), based on Llama2, and Blueprint (BP) (Fierro et al., 2024), based on T53B.

heading: 5.1 Main Results
text: Table 1 shows the main in-domain results, with mean and standard deviation computed over three seeds. CaLF improves citation F1 across datasets and models by 34.1 and 15.5 points over both in-context learning (In-context) and baseline finetuning (Few-shot FT), respectively. Impressively, all tested LLMs with CaLF outperform Self-RAG, ChatGPT, and GPT-4 with an average improvement of 24.8, 10.5, and 12.9 citation points, respectively. Moreover, CaLF achieves high citation scores while the overall quality of the response remains high. We observe modest improvement in correctness but substantial gains in grounded correctness, indicating that CaLF is better at including verifiable facts in its responses than our baselines. This contrasts observations for other models: GPT4 trades off improvements in correctness at the cost of citation quality, resulting in ChatGPT having overall higher citation scores than GPT-4. Notably, CaLF improves on correctness over GPT-4 while also producing higher-quality citations than ChatGPT, beating both models at their best-performing metric. Similarly to GPT, BP produces accurate citations but has low correctness, especially on ELI5.
Moreover, training via CaLF also leads to an overall improvement in ROUGE-L and MAUVE across models over the fine-tuning baseline by an average of 1.0 and 5.5, respectively.

heading: 5.2 Domain Transfer
text: We run domain transfer experiments to evaluate the generalization of CaLF’s citation production, by training an LLM with CaLF on a source dataset and measuring performance on a different target dataset. Table 2 shows our domain transfer results. In every source-target configuration and across instruction-tuned models, CaLF outperforms zeroshot in-context learning, Few-shot FT, FT, and SelfRAG in both citation quality and correctness. CaLF (using MistralOrca-7B) exhibits small variability regarding the training source D, with a citation F1 difference ∆∗→ASQA of 0.5 and ∆∗→ELI5 of 2.7, respectively. While CaLF’s citation quality is comparable in-domain versus in a transfer setting on ASQA (−∆3.9, see Table 1), we observe larger differences on ELI5 (−∆11.9), since for ELI5 the knowledge source and question scope greatly differ from the Wikipedia-based source datasets.

heading: 5.3 Factuality
text: We evaluate CaLF’s factual precision using FactScore. Results are shown in Table 3 for state-of-the-art methods taken from Asai et al. (2024), our fine-tuning baseline, and CaLF with MistralOrca-7B. CaLFscores the highest with 83.4, indicating that the improved citation quality translates to higher factual accuracy. Interestingly, while citation recall can be considered a stricter measure than the FactScore, the former is much higher for
CaLF. We postulate the difference is caused by retrieval inaccuracies. Subsequently, we adjust predictions to abstain from answering if none of the passages’ titles match with the BIO entity.3 As seen in Table 3, the FactScore improves to 86.1 and 88.9 for our fine-tuning baseline and CaLF, respectively. While these results are promising they also highlight the importance of accurate retrieval and high-quality knowledge bases so that citation production can translate into improved factuality, an observation also made in Menick et al. (2022); Kryscinski et al. (2020).

heading: 6 Discussion
text: Ablation. Table 4 shows an ablation for the two mechanisms introduced in CaLF, the generation of weakly-supervised training data (WS) and the focused learning loss (LFL). By adding WS, we see an improvement of 18.3 and 16.7 citation recall and precision points on ASQA, respectively. By
3Abstaining is explicitly incorporated into FactScore.
adding LFL, we further improve on top of WS by 6.3 and 8.2 recall and precision points, respectively. On ELI5, we observe similar improvements.
Iterative Training. Performance of CaLF across iterations is shown in Figure 4 on ASQA. We see a majority of citation performance improvements within the first three iterations after which citation F1 stabilizes. We further observe consistent improvements to MAUVE, ROUGE-L, and grounded correctness across iterations. Importantly, we do not observe erratic or unstable performance, indicating the robustness of our iterative training procedure. The proportion of filtered examples D̂k over Ỹk as our dynamic stopping criterion matches the citation performance well, improving efficiency by early stopping once saturated (here iteration 4).
Selection of FCM. We further investigate the extent to which the quality of an FCM translates to improved citation production for CaLF. To this end, we replace the FCM with: (i) a DeBERTav3 model adjusted for factual consistency (Steen et al., 2023), (ii) the current T5-based state-of-the-art on the TRUE benchmark (Gekhman et al., 2023), (iii)
the FCM Honovich et al. (2022) used for citation evaluation. We constrain D̂t to 32 samples and run without LFL since for decoders the computation of Shapley values over their vocabulary (prediction) is more involved. Table 5 shows results on the TRUE benchmark (summarization subset) and the citation F1 on ASQA with CaLF. We generally observe that better scores on the TRUE benchmark translate to higher citation scores when incorporated into CaLF. The only exception is the FCM Honovich et al. (2022), which is used in both training and evaluation, scoring disproportionately high in terms of citation F1, most likely due to model biases leaking through evaluation. While Gekhman et al. (2023) performs the best on TRUE, it is much more expensive to run at scale for CaLF than AlignScore.
Adversarial Baselines. Long-form QA is notoriously difficult to automatically evaluate (Xu et al., 2023; Krishna et al., 2021). Thus, to assess whether the automatic evaluation metrics of Gao et al. (2023b) can be exploited, we designed several adversarial baselines, such as copying retrieved passages as the answer or citing all passages for every generated sentence. Results are shown in Table 6. Every baseline that attempts to trick one metric performs poorly on another. For instance, copying passages (Copy) results in very poor MAUVE while citing all passages (Ours w/ cite all) results in very low citation precision. A more detailed discussion is shown in Appendix A.3.
Human Evaluation. We further conduct a human evaluation to compare the quality of generated responses produced by CaLF with Few-shot FT (using MistalOrca-7B). Human subjects are tasked to judge the models’ generations regarding citation quality, informativeness, coherence, and fluency, following human evaluation in Gao et al. (2023b) and recommendations in Zhong et al. (2022). The latter three metrics are measured using a five-point Likert scale, with five being the highest score. We
randomly sample 30 instances for each model from ELI5, ASQA, and BIO, resulting in 180 instances. Results are shown in Table 7. CaLF achieves substantially higher citation quality, fluency, and answer coherence ratings than across datasets with an F1 of 91.5 versus 72.7 for the baseline. While average informativeness ratings are comparable between CaLF and the baseline , on ELI5 particularly we observe low informativeness with CaLF. This is caused by frequent retrieval errors since ELI5 uses a weak retriever (BM25) to efficiently traverse over its large and noisy knowledge base. Instead of using its parametric memory when passages contain little relevant information, we observe that CaLF still produces accurately grounded responses, however, at the cost of less informative content.
Efficiency At inference time, CaLF matches the efficiency of the few-shot FT baseline, contrasting previous methods that cause significant overhead, including in-context learning (increased input length), Self-RAG (tree-decoding with critique tokens), and post-hoc editing. The training complexity of CaLF can be described as O(K × |X )|),
with K and X being the number of iterations and the collection size of questions we generate weaklysupervised data of, respectively. The generation of diverse answer candidates is computationally the most involved while the filtering of answer candidates and the computation of Shapley Values is efficient due to the small size of the FCM we use (Alignscore, 355M parameters). To quantify the computational cost, we measured training times using a single A100 40GB GPU. We measure a training time of 13h51min for CaLF and 1h2min for the few-shot FT baseline. While there’s a notable disparity in training time, it’s essential to note that achieving comparable performance to CaLF via regular fine-tuning would necessitate training on significantly more data, resulting in additional training and, importantly, data annotation cost.

heading: 7 Conclusion
text: This paper presented CaLF, a fine-tuning method for LLMs to produce accurate citations alongside generated text. It focuses on using FCMs as a training signal by filtering candidate answers with citations and by re-weighting the LLM’s objective function according to the tokens’ factual importance. CaLF outperforms all baselines in terms of citation quality and passage-grounded correctness while ensuring that the overall quality of responses remains high, measured via both automated and human evaluation. In a domain transfer setting we further validate the generalizability of CaLF. Moreover, we discuss the benefits of accurate citation production for improving factuality and highlight the importance of each component of CaLF through a systematic ablation. Future work looks at incorporating a learned mechanism into CaLFto abstain from answering if none of the retrieved passages are considered relevant to the question.

heading: Limitations
text: The assumption that every generated sentence requires a citation is an oversimplification we adopt from Gao et al. (2023b). However, real-world dialogue agents commonly introduce their response with non-factual phrases or sentences (e.g. Of course I can help!). A potential solution is to introduce an extrapolatory label as described in Yue et al. (2023). Furthermore, CaLF is not entirely model-agnostic out-of-the-box, since the tokenization alignment algorithm was designed with the idiosyncrasies of the particular LLMs and FCMs in mind. Moreover, inaccuracies or biases of the factual consistency models could negatively affect the citation quality of CaLF (c.f. Sec. 6). A common strategy to mitigate biases involves an alignment stage, where models are explicitly optimized on based on preference data. It would be interesting to explore alignment in the context of CaLF applied to highly domain-specific questions, such as those compiled in the recent dataset of Malaviya et al. (2024).
Finally, our paper focuses exclusively on improving generation, yet as highlighted in Section 6, high-quality retrieval systems are vital for citations to be effective. When retrieved passages contain non-factual information or lack useful content, produced answers might not be informative. While the reliance of retrieved passage is answer production poses a limitation, language model’s intrinsic memory for content generation forfeits the transparency and verifiability benefits inherent in citation production. Balancing the use of citation production and intrinsic memory while preserving the advantages of both remains an ongoing challenge. Future directions might consider a joint fine-tuning procedure of generator and retriever and even capturing their rich interactions, such as abstaining from answering questions when retrieved passages are irrelevant. The action of abstaining to answer could serve as a signal to the retrieval system to seek alternative sources of information.

heading: Ethics Statement
text: Our paper improves the ability of LLMs to produce accurate citations to sources alongside their generated answers to improve the verifiability of LLMs’ responses. As noted in Section 5.3, we caution against equating improved citation quality with improved factuality since even correctly cited passages can be factually incorrect or misleading
without additional context about the passage and its source. Citations can therefore invoke a false sense of trust and amplify observations made in Si et al. (2023). It remains important for users to stay critical and reflect on generated responses. By improving citation accuracy our paper simplifies the verification process but does not eliminate it. Furthermore, Huang and Chang (2024) argue that the incorporation of citations can help to address intellectual property and associated ethical issues in the deployment of LLMs due to the increased verifiability of responses made. Finally, we recognize a potential risk of dual-use by adversarial actors: similarly to how humans cherry-pick data to support a strong bias or prior, we cannot guarantee that an LLM will not exhibit similar behaviour when manipulated with passages that contain misinformation or miss relevant context.

heading: Acknowledgements
text: The authors would like to thank the anonymous reviewers for their time and detailed feedback on our paper. We further thank Tony Hu for being part of the human evaluation.

heading: A Appendix

heading: A.1 Datasets
text: The four datasets considered for evaluation cover different question-answering tasks and domains: ASQA is a disambiguation task built on top of AmbigQA (Min et al., 2020), ELI5 contains realworld highly open-ended questions and answers from an online forum (Reddit), Hagrid contains entity-specific questions, and BIO is a biography generation task containing simple person-related questions. Hagrid’s training set as provided by Kamalloo et al. (2023) consists of 1,922 GPT-4 generated answers, out of which 335 instances are human-labeled as both attributable and informative. Only these 335 instances are used as training data. For Hagrid, we randomly sample 4 instances for CaLF and use all its 335 training samples for our fine-tuning baseline. We use each datasets’ original training set to sample questions X and atomic facts A. Since Hagrid does not have any annotations for factual coverage, we compute the quality assurance function Q only over the Citation conditions. BIO consists of two evaluation sets. As recommended by the authors and following Self-RAG, we use the second, and more difficult, evaluation set in our experiments. The test set of ALCE-ASQA and ALCE-ELI5 consists of 1000 randomly sampled instances from their respective development set. For more details on the ALCE benchmark, we refer to Gao et al. (2023b).

heading: A.2 Experimental Setup
text: Token alignment algorithm We design the token alignment algorithm around the idiosyncrasies of the LLMs’ and FCM’s tokenizers. Llama2, Mistral-Instruct, and MistralOrca use the Llama tokenizer4, a BPE model (Sennrich et al., 2016) based on sentencepiece5. Similarly, the FCM, Alignscore, uses a RoBERTa tokenizer which is derived from the GPT-2 tokenizer, also using BPE. Both Llama and RoBERTa tokenizers treat spaces as parts of the tokens. In contrast to RoBERTa tokenizer, the Llama tokenizer indicates each space via an underscore ("_"). We exclude special tokens of either tokenizer (e.g. "
" or "<0x0A>") from the alignment procedure.
4https://huggingface.co/docs/transformers/ main/en/model_doc/llama#transformers. LlamaTokenizer
5https://github.com/google/sentencepiece?tab= readme-ov-file
Given a sentence, tokenized both by FCM and LLM, we we first strip the FCM tokens (i.e. remove spaces) and remove underscores for tokens of the LLM, essentially removing all spaces to unify their representations. The algorithm then checks whether the current tokens are equal or subsets of one another, with and without considering the memory of tokens not yet aligned but iterated over by the pointers. For instance, consider the FCM tokens: ’Maw’, ’syn’, ’ram’, and LLM tokens for the same word: is _M’, ’aws’, ’yn’, ’ram’. For the first two FCM tokens, none of the LLM tokens can be directly aligned to it. Our algorithm ensures to find the smallest sequence of tokens in both the LLM and FCM that can be aligned to each other (here the spans are ’Mawsyn’ and ’ram’) and assigns the relevance score as described in Section 4.2. We find the smallest sequence by keeping a list of tokens for each FCM and LLM that could not yet be aligned and increment the respective pointers according to which sequence needs to continue (e.g. Checking that [M] + [was] continues [Maw] potentially, requires incrementing the FCM pointer, since an ’s’ is missing in [Maw]. This algorithm is efficient and runs in linear time.
Generating Cited Answers for Training. While datasets such as ELI5 have over 250K training instances we could use to generate weaklysupervised training data, we constrained the size of X to 1000 samples, since it is computationally infeasible to run our weakly-supervised data generation procedure on all training samples of the dataset. The threshold Θ is determined dynamically. Starting with a value of 0.9, if the number of samples in D̂ is below 3, we consider the threshold too high for the given LLM and reduce it by 0.1 until the requirement is met.
Passage Retrieval. For the retriever R we use GTR (Ni et al., 2021) for ASQA and Hagrid (Wikipedia), BM25 for ELI5 (CommonCrawl6), and Contriever-MS MARCO (Izacard et al., 2022) for BIO (Wikipedia), to maintain comparability with Gao et al. (2023b) and Asai et al. (2024). We use |P | = 3 passages throughout due to context length limitations. We use the same indices as provided by ALCE, subsequently, each passage has a length of 100 tokens.
Answer Generation. We use the same instructions/prompts across all models, specifically the
6http://commoncrawl.org
default prompts from ALCE. In contrast to the code of ALCE7, we used the chat templates across all models and baselines8 to render the inputs, since we observed crucial tokens missing during the finetuning procedure otherwise. The only exception is ELI5, where we observed that chat templates resulted in worse performance and used ALCE’s strategy plus relevant formatting tokens instead across all models and baselines. For inference, we use MAP with a beam size of 1, instead of sampling with temperature scaling as done in ALCE, assuming that this results in more factual outputs.9
Factual Consistency Model Our choice of using AlignScore as our FCM ϕ was further motivated by its dissimilarity to the citation evaluation model (Honovich et al., 2022) (in contrast to e.g. TrueTeacher (Gekhman et al., 2023) which in principle performed better but is more similar to the evaluation model). While AlignScore is an encoderbased RoBERTa model, TRUE (Honovich et al., 2022) is an encoder-decoder T5-11B model. Moreover, their training data is different. AlignScore was trained on 15 datasets of various tasks, including natural language inference, summarization, information retrieval, and paraphrasing while TRUE has only seen 6 natural language inference-related datasets.
Training & Hyperparameters We set the learning rate to 3−4 and train for a total of 100 steps across all models and experiments. The maximum generation length is set to 256 tokens, however, most generations stay far below this limit (see Appendix A.3. We use adamw (Loshchilov and Hutter, 2019) as the optimizer. We use a batch size of 1 during training with gradient accumulation, resulting in an effective batch size of 4. For LoRA, we use a rank r = 4 and apply it to all parts of the attention mechanism. For fine-tuning, we exclude tokens of the prompts from the loss computation that are not part of the gold answer, so we are not fine-tuning the instructions, only the answers that follow after the instruction. The iterative training process stops after at most 8 iterations due to computational constraints. We further down-weight the
7https://github.com/princeton-nlp/ALCE 8https://huggingface.co/docs/transformers/
main/en/chat_templating 9Note that we do not use constrained decoding for citation generation despite its apparent suitability here. Yet, the position of citation markers within a sentence can be rather flexible, which we consider a desired property for naturally produced text.
loss contribution of the EOS token to 0.02 since models tend to otherwise produce very short singlesentence responses.
Implementation Details We use the Huggingface checkpoints for LLama2-7B10, MistralOrca7B11, and Mistral-Instruct-7B,12 as well as for all FCMs we considered in our experiments, namely AlignScore (RoBERTa-large, 355M parameters) 13, TRUE (T5, 11B parameters)14, (Steen et al., 2023)15 (DeBERTaV3, 304M parameters), and TrueTeacher (T5, 11B parameters)16. The Mistral models are licensed under Apache2.0, AlignScore and (Steen et al., 2023) are licensed under MIT, TrueTeacher is licensed under cc-by-nc-4.0, and Llama2 is licensed under the llama license17. Subsequently, our research is consistent with the licenses’ intended use. The models are intended to be used for English. We use the ALCE data and prompts from their repository18, and Huggingface’s Datasets for Hagrid19. For running the experiments, we used a combination of A100 40GB and A10G with 23GB GPUs. We use the Python package rouge-score20 for computing the ROUGE, and the package mauve-text21 for computing the MAUVE score. We adopt the code of ALCE for the citation and correctness metrics and define passagegrounded correctness ourselves. We use NLTK (Bird and Loper, 2004) for some pre-and postprocessing steps.
Baselines. Results from state-of-the-art methods are taken from their respective papers. The only exception is Self-RAG on ELI5, which we have run by ourselves using the authors’ repository and
10https://huggingface.co/meta-llama/ Llama-2-7b-chat-hf
11https://huggingface.co/Open-Orca/ Mistral-7B-OpenOrca
12https://huggingface.co/mistralai/ Mistral-7B-Instruct-v0.1
13https://huggingface.co/yzha/AlignScore, and their repository https://github.com/yuh-zha/ AlignScore
14https://huggingface.co/google/t5_xxl_true_ nli_mixture
15https://huggingface.co/juliussteen/ DeBERTa-v3-FaithAug
16https://huggingface.co/google/t5_11b_ trueteacher_and_anli
17https://github.com/facebookresearch/llama/ blob/main/LICENSE
18https://github.com/princeton-nlp/ALCE 19https://huggingface.co/datasets/miracl/hagrid 20https://pypi.org/project/rouge-score/ 21https://pypi.org/project/mauve-text/
their models22. At the point of submission, their repository produces results which are worse than reported in their paper, as explained by the authors due to a bug23, which potentially impacts scores reported of Self-RAG on ELI5. While the authors have stated their intentions to fix this issue, the problem was not resolved as of our submission date. We will update their scores once a fix is available.

heading: A.3 Evaluation
text: Detailed descriptions for each evaluation metric are shown in Table 8. Results shown in Table 2 are computed with default random seed 42.
Ablation Table 9 Table 10 show the ablation results for CaLFon Mistral-Instruct 7B and Llamav27B-chat, respectively. The results largely align with observations made for MistralOrca-7B in table 4. We observe that passage-grounded correctness either slightly decreases or remains comparable to the baseline when using the weakly-supervised training without our focused learning objective LFL. Once the objective is added,vcorrectness improves consistently across datasets and models over the baseline.
Adversarial Baselines The automated metrics proposed by Gao et al. (2023b) do not explicitly control for the generation of irrelevant information (i.e. factual precision, such as FactScore)24. Subsequently, their metrics favour longer responses for coverage-based measures (i.e. correctness), as also pointed out by Asai et al. (2024). This raises the question of whether the automated metrics deployed can be tricked with trivial responses or certain response patterns.
Table 6 shows results across metrics for several such baselines on ASQA. Our approach has an average response length of 71.2 tokens, comparable to typical fine-tuning with 61.0 tokens, both being substantially shorter than the dataset’s average gold answer length with 113 tokens. In contrast, the retrieved passages themselves are 521.8 tokens long. Considering these as the answer themselves, we indeed achieve a higher correctness score (50.6) while maintaining perfect citation, however, we observe a substantial decrease in both ROUGE-L and
22https://github.com/AkariAsai/self-rag 23https://github.com/AkariAsai/self-rag/issues/4 24The exclusion of such metric in ALCE is likely due to gold answers not being designed to be factually comprehensive.
MAUVE scores. This is intuitive since the retrieved passages are very dissimilar in style from the gold answers. Moreover, MAUVE has an explicit length bias (Pillutla et al., 2021). When explicitly biasing our CaLF to generate long sequences by removing the EOS token during fine-tuning and setting the generation limit to 256 tokens, we also observe a substantial increase in correctness while maintaining much of the attribution performance. Yet, again we observe a substantial decrease in ROUGE-L and MAUVE, highlighting the importance of maintaining all performance metrics high while optimizing attribution, as achieved by CaLF. Finally, we consider replacing the citations made in generated responses from our model with: (i) citations to exclusively the first passage, (ii) citations to all passages. As seen in the table, citation scores are substantially worse than our approach. While in principle (ii) should present an upper bound on attribution recall, we observe lower scores than our model even here. This can be explained by inaccuracies in the evaluation model, being biased towards information at the beginning of the premise (scoring much worse for citations to the last passage).

heading: A.4 Human Evaluation
text: Human subjects are tasked to judge the models’ generations regarding: (i) citation recall: judgement whether a generated sentence is fully supported by citations, (ii) citation precision: whether a citation partially or fully supports a sentence, (iii) informativeness: whether the generation helps to answer the question, (iv) coherence: whether generated sentences are semantically and syntactically well-connected, (v) fluency: whether all sentences are grammatically correct and well-readable. While (i), (ii), and (iii) are adopted from the human evaluation in Gao et al. (2023b), we further measure (iv) and (v) as recommended in Zhong et al. (2022). We randomly sample 30 instances for each model for each dataset (ELI5, ASQA, BIO), resulting in 180 instances which are judged via the above criteria. The instances were judged by four subjects, distributed equally (45 samples per subject). Each subject was provided with detailed annotation guidelines, providing examples for each possible annotation option with explanations. The subjects were partially authors and partially volunteers. All subjects were informed how the data would be used and provided consent. All subjects are male under 65 years of age and either from the USA or the UK.

heading: A.5 Qualitative Examples
text: To qualitatively compare the produced answers by CaLF and our few-shot FT baseline we show a randomly selected output for each evaluation dataset. Results are shown in Figure 5, Figure 6, and Figure 7 for ASQA, ELI5, and BIO, respectively.

heading: Future Work
text: Future directions might consider a joint fine-tuning procedure of generator and retriever and even capturing their rich interactions, such as abstaining from answering questions when retrieved passages are irrelevant. The action of abstaining to answer could serve as a signal to the retrieval system to seek alternative sources of information