Improving Attributed Text Generation of Large Language Models via Preference Learning (2024)

Abstract: Large language models have been widely adopted in natural language processing, yet they face the challenge of generating unreliable content. Recent works aim to reduce misinformation and hallucinations by resorting to attribution as a means to provide evidence (i.e., citations). However, current attribution methods usually focus on the retrieval stage and automatic evaluation that neglect mirroring the citation mechanisms in human scholarly writing to bolster credibility. In this paper, we address these challenges by modeling the attribution task as preference learning and introducing an Automatic Preference Optimization (APO) framework. First, we create a curated collection for post-training with 6,330 examples by collecting and filtering from existing datasets. Second, considering the high cost of labeling preference data, we further propose an automatic method to synthesize attribution preference data resulting in 95,263 pairs. Moreover, inspired by the human citation process, we further propose a progressive preference optimization method by leveraging fine-grained information. Extensive experiments on three datasets (i.e., ASQA, StrategyQA, and ELI5) demonstrate that APO achieves state-of-the-art citation F1 with higher answer quality. 1

heading: 1 Introduction
text: Large Language Models (LLMs) have demonstrated emergent abilities and have gained widespread application in Natural Language Processing (NLP) (Brown et al., 2020; Wei et al., 2022; OpenAI, 2022; Anil et al., 2023). For example, LLMs have shown remarkable in-context learning capabilities across a variety of domains and tasks (Dong et al., 2023). Although LLMs have been widely adopted, a prominent issue is that they
*Corresponding author. 1Code is released in https://github.com/HITsz-TMG/
ATG-PO
produce hallucinations in certain situations (Ye et al., 2023a; Zhang et al., 2023). In other words, they generate information that sounds plausible but is nonfactual, thereby limiting their applicability in the real world. To mitigate hallucinations, researchers have resorted to grounding statements in responses generated by LLMs to supported evidence, either by providing rationales or by adding citations to the statements (Li et al., 2023a; Liu et al., 2023).
Recent works have utilized external knowledge sources such as retrieved documents and knowledge graphs for attribution (Shuster et al., 2021; Li et al., 2023c). Generally, these works are divided into two types: 1) the model generates an answer with citations based on the retrieved documents (Li et al., 2023b); 2) an answer is first generated, then modified again to add attribution references by retrieving with query and initial answer (Gao et al., 2023a). However, these works focus mainly on the retrieval stage (Ye et al., 2023b) and the evaluation process (Yue et al., 2023). Considering the selection of the model’s desired responses and behavior from its very broad knowledge and capabilities, it is more necessary to optimize the generation process, not only reducing the hallucination of the original answer but also avoiding the hallucination of the attribution process. On the other hand, fine-tuning LLMs after pre-training can also significantly improve performance for users’ downstream tasks. First, given positive examples of correct behavior, supervised fine-tuning can be performed using standard likelihood-based training. Secondly, given positive and negative examples (binary feedback or pairwise feedback), methods such as unlikelihood training on negative examples (Welleck et al., 2020) or RLHF-PPO (Ziegler et al., 2019) can be used for learning. However, these methods usually suffer from expensive data collection process, reward model training, sparse reward and text degeneration problems, making them difficult to use
5079
in practical applications (Azar et al., 2023). In this paper, inspired by the citation mechanisms in human scholarly writing (Brooks, 1986; Teplitskiy et al., 2022), we address these challenges by conceptualizing the attribution task for LLMs as preference learning and proposing an Automatic Preference Optimization (APO) framework, as shown in Figure 1. Initially, we assemble a curated dataset comprising 6,330 examples sourced and refined from existing datasets for posttraining. This step makes the LLMs know the basic format and requirements of attribution. Considering the substantial cost and extremely timeconsuming of preference pair annotations, we thus introduce an automated approach to generate attribution preference data, yielding 95,263 pairs. Furthermore, drawing inspiration from the human process of citation and direct preference optimization (Rafailov et al., 2023), we propose a progressive preference optimization method with experience replay bypassing the need for explicit reward modeling or reinforcement learning. We conduct the extensive experiment on three datasets (i.e., ASQA, StrategyQA, and ELI5). The experiment results demonstrate that APO surpasses compared baselines across all datasets with improved citation F1 along with higher response quality. Our contributions are summarized as follows:
• To the best of our knowledge, we are the first to apply preference learning for attribution tasks. We also show that our method can be applied under synthesized preference scenarios.
• We establish a full data collection pipeline for attribution tasks and will open-source our all authorized data after publication for future research.
• We propose a progressive preference optimization method to alleviate the sparse reward problem by leveraging fine-grained information. We further benchmark existing direct preference optimization methods and provide insights for attribution tasks.

heading: 2 Related Work

heading: 2.1 Text Generation for Verification
text: Prior works have studied methods and evaluations for verification that identify supporting sources for model outputs. For instance, Rashkin et al. (2021)
introduce the concept of Attributable to Identified Sources (AIS) which transforms model outputs into standalone, interpretable propositions. The response s can be attributed to a source P if they meet the intuitive criterion “According to P, s”. Bohnet et al. (2022) adapt the AIS framework for QA scenarios. Further, Gao et al. (2023b) extrapolate AIS to evaluate generated text of LLMs with citations. Additionally, several works focus on building and using automated AIS evaluations (Honovich et al., 2022; Gao et al., 2023a; Liu et al., 2023). For a comprehensive overview, please refer to Li et al. (2023a). In contrast to existing approaches, our work broadens the scope of attribution beyond just verifiable text generation and devises a methodology to enhance these attributions which frames it as a preference learning problem.

heading: 2.2 Preference Optimization Methods
text: Preference Optimization (PO) methods significantly improve generate quality to align with human values (Christiano et al., 2017; Ziegler et al., 2019; Stiennon et al., 2020; Bai et al., 2022). It usually first collects pairs of generations under the same context and a pairwise human preference to indicate which generation is better. Then the PO is used to optimize generating policy to generate better candidates from the pair. For example, Reinforcement Learning from Human Feed-
back (RLHF) is a model-based algorithm to optimize preference learning (Ouyang et al., 2022). However, the RLHF process is complex, timeconsuming, and unstable. The direct PO uses an off-policy algorithm to directly optimize the generating policy, eliminating the need for a reward model (Rafailov et al., 2023; An et al., 2023; Kang et al., 2023; Zhao et al., 2023). These approach are more data-efficient and stable. For example, DPO uses the Bradley-Terry model (Bradley and Terry, 1952) and log-loss, which can lead to over-fitting to the preference data, especially when preference is deterministic and ignores the KL-regularization term. The IPO algorithm (Azar et al., 2023) addresses this issue by using a root-finding MSE loss to solve the problem of ignoring KL-regularization when preference is deterministic. However, these methods fail to fully account for more fine-grained preferences and that is exactly what we want to do.

heading: 3 Preliminary
text: The main pipeline of preference learning usually consists of: 1) pretraining and Supervised FineTuning (SFT), where SFT is not a must; 2) preference data collection; 3) preference optimization.
Pretraining and SFT Phase Preference learning typically starts with a pretrained LLMs or LLMs fine-tuned on high-quality data using maximum likelihood estimation. The final policy πref after this phase is represented as
πref ≈ argmax π Ex,y∼Dref log π(x) log(y|x), (1)
where Dref denotes the training data distribution.
Preference Data Collection Phase After pretraining and SFT phase, πref is prompted by context x, and generate two responses yw, yl ∼ πref(·|x). Then x, yw, yl is labeled by humans to judge which response is preferred and denote yw ≻ yl|x if yw is preferred, and yl ≻ yw|x if yl is preferred. We define a new symbol I = I[yw ≻ yl|x], and all <x, yw, yl, I> consist the preference dataset Dp:
⟨x, yw, yl, I⟩ ∼ Dp. (2)
Preference Optimization Phase In the final phase, the prevailing method uses reinforcement learning algorithm to learn an explicit or implicit reward from the preference data, and then using on-policy or off-policy policy gradient algorithm to maximize the reward. Recently, some methods
have derived the optimal policy using reward maximization under KL-regularization and also derive a loss with optimal policy as its solution, then learn the optimal policy by minimizing the derived loss on empirical dataset.
Reinforcement Learning from Human Feedback (RLHF) The RLHF uses standard two-phase reward model-based reinforcement learning to maximize the reward. It contains two steps: 1) reward estimation from preference data 2) reward maximization using PPO algorithm. It aims to maximize reward with a KL constraint on the reference model πref (inputs x omitted):
π∗ = argmax π
Ey∼π [ r(y)− β log π(y)
πref(y)
] , (3)
where β is the regularization weight and r(y) is the reward function learned using the Bradley-Terry model on the preference dataset of generating y.
Direct Preference Optimization (DPO) DPO eliminates the training of reward model. It derives a loss on the current policy πθ (yw, yl omitted):
Ldpo = − log σ ( β log πθ(yw)
πref(yw) − β log πθ(yl) πref(yl)
) , (4)
i.e., the binary cross entropy with
p̂θ(yw ≻ yl) = σ ( β log πθ(yw)
πref(yw) − β log πθ(yl) πref(yl)
) ,
(5)
and target p(yw ≻ yl) = 1. We describe more PO methods in details in Appendix B.

heading: 4 Methodology

heading: 4.1 Problem Formulation
text: Formally, consider a query q and a corpus of text documents D. The goal is to produce an output S, where S is a collection of n distinct statements: s1, s2, . . . , sn. Each statement si is associated with a set of citations Ci. This set Ci is defined as Ci = {ci,1, ci,2, . . .}, where each ci,j is a document from the corpus D. For application purposes, the output from LLMs can be divided into individual statements using sentence boundaries. This approach is utilized because a single sentence typically encapsulates a coherent statement while maintaining brevity, facilitating easy verification. Regarding the citation format, citations are typically presented in square brackets, e.g., The sun is formed approximately 4.6 billion years ago [1][2]. However, it should be noted that these citations can
be attributed to specific phrases as well, not just at the end of sentences.
Moreover, in this paper, we define generation hallucination refers to a situation where the model generates content that is not based on factual information and attribution hallucination means that the statement corresponding to one citation is unfaithful or not supported by the referred source content.

heading: 4.2 Overall Framework
text: As shown in Figure 2, we introduce the APO framework to apply preference learning for attribution task. The APO framework consists of the posttraining procedure to ground the base model for attribution (§4.3), and the preference optimization procedure to address both generation hallucination and attribution hallucination (§4.4).

heading: 4.3 Post-training
text: The goal of post-training procedure is to ensure that given a specific question q and a corpus of text documents D, the model can be successfully instructed to generate answer S and add citation Ci for each statement si in its response when necessary.
Data Collection We construct the post-training data from training sets using existing attribution datasets including EVIGSE (Liu et al., 2023), ExpertQA (Malaviya et al., 2023) and HARGID (Kamalloo et al., 2023). We select these datasets because they are high-quality attribution datasets with diverse domains and sources annotated by human experts or powerful LLMs. After preprocessing and formatting, the final post-training data collection includes 6,330 samples. The pre-processing
Algorithm 1 Preference data sampling and labeling 1: Input Queries Q, Critic Mc, Generator Mg , Retriever R 2: Output Output initialized preference dataset Pinit 3: Pinit = {} 4: for q ∈ Q do 5: Retrieve top-k passages D using R given q 6: Predicts relevant label Lrel ∈ {0, 1} using critic Mc
for each d in D given q, d 7: Generate S constructed by statements {s1, s2, .., sn},
using generator Mg given (Ipost, q,D). 8: for si ∈ S do 9: Predicts supported label Lsup ∈ {0, 1} for each
ci,j using critic Mc given q, d ↔ ci,j , si 10: end for 11: Add augmented (q,D,S, C,Lrel,Lsup) to Pinit 12: end for
details are shown in Appendices C and E, and the statistics of training data are shown in Table 1.
Training After that, instruction Ipost, documents D and question q are formatted to be the input while answer S composed of multiple statements is formatted as output. We tune the model using autoregressive language modeling objectives, resulting in initial generator Mg.

heading: 4.4 Preference Optimization
text: In this section, we describe our preference optimization procedure to enable a model-agnostic approach for improving the quality of generated responses. First, considering the cost of labeling preference data, we devise an automatic data collection algorithm motivated by errors where previous models may have misattributed. Second, we propose a progressive preference optimization approach to amplify the preference signal by using synthesized preference pairs. We further apply the experience replay to alleviate the over-fitting and text degradation phenomenon due to the distribution shift introduced by automatic data generation.
Automatic Data Collection In general, attributed text generation should be both relevant and supported (Asai et al., 2023b). Being relevant needs the reference document in the answer to be helpful in handling the question. It is used to measure whether C provides useful information to solve q. Being supported asks the generated text be grounded on the reference documents. It is used to measure whether all of the verification-worthy statements in S are supported by C.
Following the requirements above, we first get initial responses and related labels for each query with the Algorithm 1. The query comes from multiple open domain tasks or high-quality instruc-
tion data sets shown in Table 1. The source of retrieved documents is English Wikipedia. The retriever R we use here is gtr-t5-large2. The objective is to generate the attributed text with relevant and supported labels for related documents using the critic model Mc. Here, we use pre-trained selfrag_llama2_7b3 as Mc in Asai et al. (2023b) because it can give fine-grained feedback using reflection tokens.
After that, we generate preference pairs using an automatic collection algorithm. Specifically, we determine whether the citations Ci of each statement si of query q are all related to it based on the relevant tags. If it is all relevant, we add the current statement and its preceding statements Si to the set Ptmp for subsequent processing. For example, if s2 meets the requirement, we add {s1, s2} to Ptmp. The motivation here is that we want to select the statements that can answer the question based on the document as the initial set. Then, for each entry in Ptmp, we first retrieve another topm (m ≫ k) documents and filter them into 10 irrelevant documents Dir scored by relevant logits predicted by Mc. If all documents in Dir are relevant, we use the last 10 documents as Dir. After that, we generate the positive and negative pair for each statement si ∈ Ptmp. There are two situations: the statement si is fully supported by Ci and otherwise. For the first situation, we first expand Ci with supported document by second judgment in D using Mc. Then, we generate one positive statement using q, Si−1 and new Ci and two negative statements using q, Si−1, Dir and q, Si−1, new Ci, error instruction e respectively. Thus, there are two preference pairs in this context. For the second situation, we generate one positive statement using q, Si−1 and new Ci and one negative statement using q, Si−1, D − Ci, error instruction e. The full procedure is shown in Algorithm 2.
In the generation of negative samples, we use the error instruction e ∈ E , which defines two types: irrelevant but supported means the generated text si is grounded on unhelpful reference documents Ci, while relevant but unsupported further has three fine-grained subtypes: 1) fabricated statement refers to the generated text contains facts or information that cannot be derived from reference documents; 2) mistaken synthesis means that several reference documents are used, but facts or
2huggingface.co/sentence-transformers/ gtr-t5-large
3huggingface.co/selfrag/selfrag_llama2_7b
logics are mistakenly intermingled. The generated text thus contains factual error or logic error; 3) unintentional omission means that reference documents are used, but the key points are incomplete. There are no factual errors in generated text, but some information is omitted. The irrelevant but supported error derives from attribution hallucination, whereas the relevant but unsupported error is the result of generation hallucination. Note that irrelevant and unsupported errors are not included, since it is more like easy negatives. The details of error instructions are in Appendix F.
Progressive Preference Optimization To reinforce the preference feature and alleviate sparse reward problem (Zheng et al., 2023; Lightman et al., 2023), we propose a progressive preference optimization method. Considering generations can be separated into several consecutive statements, each statement may contain hallucinations at all. The entire response-level reward preference modeling performs in the global context and potentially oversights the fine-grained deterministic preferences we constructed. Hence, we use fine-grained statementlevel reward to perform preference optimization to update the model in a more effective and efficient way. Formally, assuming that deterministic preference is performed at statement-level, we can rewrite the preference optimization loss in Eqn. (4) as follows (− log σ omitted):
L ≜ β log πθ(yw) πref (yw) − β log πθ(yl) πθ(yw)
= β log
∑ i πθ(s
w i |sw:i−1)∑
i πref (s w i |sw:i−1)
− β log ∑ j πθ(s l j |sl:j−1)∑
j πref (s l j |sl:j−1)
= β ∑
i
log πθ(s
w i |sw:i−1)
πref (swi |sw:i−1) − β
∑
j
log πθ(s
l j |sl:j−1)
πref (slj |sl:j−1)
= β ∑
i
( log πθ(s w i |sw:i−1)
πref (swi |sw:i−1) − log πθ(s
l i|sl:i−1)
πref (sli|sl:i−1)
) .
(6)
The progressive preference optimization loss can be further written as follows (− log σ omitted):
L ≜ E (swi ,s l i∼D)
( β log πθ(s w i )
πref (swi ) − β log πθ(s
l i)
πref (sli)
)
= E (yw,yl∼D)
1 n
∑
i
( β log πθ(s w i )
πref (swi ) − β log πθ(s
l i)
πref (sli)
) .
(7)
The main difference between vanilla preference optimization in Eqn. (4) and progressive preference optimization is that the latter contains an implicit mean pooling procedure when implementing the preference optimization loss.
Furthermore, the directed preference optimization may face the challenges of overfitting to some deterministic preference due to weak KL constraint (Azar et al., 2023). Hence, we propose to leverage experience replay (Rolnick et al., 2019) as learning with rehearsal to alleviate the over-fitting phenomenon. The idea of replaying experience typically stores a few old training samples within a small memory buffer. Therefore, we iteratively add post-training autoregressive language modeling loss to the preference optimization procedure in a fixed interval, resulting in final generator Mp.

heading: 4.5 Inference and Refinement
text: During inference, for query q, D is first retrieved and then sent to Mp output to the final answer Sinit consists of n statements. As there may not be all statements correctly attributing documents, we additionally perform the post-hoc refinement after the original generation. We maintain a collection of citations Ctmp. Starting from the last statement of Sinit, if the current si has the citations, update the Ctmp to the citations of the current si; if the current si does not have a citation, add the current citation set Ctmp to this statement until all n statements have been traversed. Then we concatenate these n statements together as the final answer S.

heading: 5 Setup

heading: 5.1 Datasets and Evaluation Metrics
text: Dataset We mainly focus on attributable longform question-answering (QA) task using ASQA dataset and ELI5 subsets from Gao et al. (2023b). In addition to these factoid long-form QA tasks, we test the generation quality on StrategyQA dataset (Geva et al., 2021) which focuses on opendomain QA where the required reasoning steps are implicit in the question. We use the official test set as our evaluation set.
Metrics Following Gao et al. (2023b), we report citation recall, precision, and F1 which uses TRUE (Honovich et al., 2022) as the attribution evaluation model ϕ to automatically examine whether the cited documents entail the model generation. For ASQA dataset, we report the recall of correct short answers (EM-R) by checking whether the short answers (provided by the dataset) are exact substrings of the generation. For ELI5 dataset, we report the claim recall (Claim) to check whether the model output entails the sub-claims, that are generated by text-davinci-003 (Ouyang
et al., 2022). For StrategyQA dataset, we report the accuracy for task performance.

heading: 5.2 Competitive Methods
text: We compare APO with several baselines. For each baseline, we use gtr-t5-large as our retriever.
In-Context Learning (ICLCITE): We prompt LLMs with few-shot examples, each consisting of a query, a set of retrieved documents and an answer with inline citations. The LLMs can in-context learn from the examples and generate grounded responses for the test query and retrieved documents.
Post-Hoc Cite (POSTCITE): Given query q, we first instruct LLMs to answer q without retrieved documents. Then, we use the attribution evaluation model ϕ to link each statement to the most relevant document retrieved by the query.
Post-Hoc Attribute (POSTATTR): Instead of citing the most relevant document, for each statement, we further retrieve a set of k documents and then use the ϕ to link to the document that maximally supports the statement by threshold.
Self-RAG (Asai et al., 2023b): Self-RAG is the state-of-the-art (SoTA) method that adaptively retrieves documents on-demand. It generates with reflection on retrieved documents and its generations by special token control.
AGREE (Ye et al., 2023b): AGREE leverages test-time adaptation to reinforce unverified statements which iteratively improves the responses of LLMs. It tunes a pre-trained LLM to self-ground its response in retrieved documents using automatically collected data.

heading: 5.3 Implementation Details
text: If not specified, we retrieve the top 5 documents as the related documents to q and we set the decoding temperature to 0.01 during inference. For the post-training, we tune the model for 2 epochs with a learning rate of 5e-5. For the preference optimization, we tune the model with LoRA (Hu et al., 2022) for 1 epoch, and we set alpha to 2 and lora ranks to 16. We set m to 100. We use llama-2-13b-base (Touvron et al., 2023) for fair comparison. We run all the experiments on NVIDIA A100 80G GPUs.

heading: 6 Results

heading: 6.1 Main Result
text: Table 2 shows the comparison results of APO with other baselines on three datasets. In terms of cor-
rectness and citation quality, our method outperforms the baselines on all three datasets. It shows that APO has better overall generation performance in various scenarios. Specifically, our method outperforms Self-RAG by 8.8 points on the EM-R metric. We speculate that this inconsistency stems from the difference between coherent generation and step-wise generation in Self-RAG. Our method also shows consistent improvements over AGREE across multiple benchmarks which suggests that APO can more effectively exploit the power of LLM to enhance retrieval. APO can be used to complement these active or adaptive retrieval-based methods and we leave it for future work. Compared to the post-training baseline, the preference optimization shows further improvement with an 8.0 average increased citation F1. Furthermore, we observe a trade-off between correctness and citation quality in several baselines including Self-RAG and AGREE, possibly due to the generation hallucination and attribution hallucination defined in §4.1. In contrast, APO helps to deal with these hallucinations and performs well in terms of both correctness and citation quality.

heading: 6.2 Ablation Study
text: We evaluate the effectiveness of each predefined error type and the results are shown in Table 3. Specifically, we perform progressive PO on the model after post-training and remove data corresponding to a predefined type. We observe that without data corresponding to hallucinated statement error, citation F1 drops significantly which suggests that our approach improves the groundedness of the model. Mistaken synthesis error seems to contribute little to performance improvement, but we observe that it can help improve groundedness under human evaluation (§6.5). Without unintentional omission error, the model shows poor generation quality. This means that the model may generate incomplete answers.
Moreover, we perform an ablation study on the training strategy of preference optimization. We find that the model can also be improved under the response-level preference optimization method such as vanilla DPO, but the improvement is slightly less. In addition, we ablate the PO by removing the ASQA questions from our preference data. Note that we construct the preference data based on the training set of ASQA, and use its test set for evaluation. We have verified and guaranteed that there is no data overlap between the two. We find that the generation quality and citation quality have decreased. We attribute it to high-quality indomain questions in ASQA as a long-form question answering dataset.

heading: 6.3 Different Prompting Strategy
text: We explore applying APO to four prompting strategies (Gao et al., 2023b): 1) VANILLA that provides the top-5 retrieved documents for each question. It is our default setting. 2) SUMM that provides summaries instead of the full text of the top-10
retrieved documents for each question. 3) SNIPPET that provides snippets instead of the full text of the top 10 retrieved documents for each question. 4) ORACLE that provides 5 gold documents for each question. We use llama-2-13b-chat as the comparison method because it has impressive instruction following ability and moderate size. As shown in Table 4, we find that in most cases, APO achieves better performance than baseline. For example, APO under VANILLA and ORACLE settings performs best in Citation F1 on ASQA, while it under SUMM and SNIPPET settings in ELI5 has improved Citation F1. It shows that the format of the context has an impact on attribution task.

heading: 6.4 Different PO Methods
text: Table 5 illustrates the results of different direct preference optimization methods adopted by Mp. We include a SFT baseline to tune the Mg using the positive part in the chosen preference pairs that we created. We observe that our method can be transferred to several different preference optimization methods, but the performance swings in several metrics. All preference optimization methods have performance boosts compared with the posttraining baseline and the SFT baseline. It shows that preference optimization can help improve the

heading: Error Type # Proportion (%)
text: generation quality to some extent.

heading: 6.5 Error Analysis
text: We conduct human evaluation of model response on ASQA dataset. Specifically, we collect 50 samples that contain errors judged by the attribution evaluation model ϕ. We then perform a detailed manual review of these samples to identify error types. Our evaluation results are shown in Table 6. We find that nearly half of the errors are of fabrication error. We reveal that the model either generated text not supported by the reference documents or incorrectly attributed information to irrelevant documents. In certain instances, hallucinations are due to the documents with low quality. For example, some documents are truncated, and the model attempts to complete or extrapolate the incomplete text. Additionally, we notice omission errors on both generated text and citation where the model fails to generate necessary citations to substantiate its statements. Although synthesis errors are less common, we observe some cases which model conflated information from multiple documents and generated counterfactual statements. The case study is shown in Appendix G.

heading: 7 Conclusion
text: This paper introduces the APO framework for attributed text generation. We treat attribution as a preference learning task, utilizing curated posttraining collections and an automated synthesis algorithm to reduce manual labeling costs. Experiments on three datasets demonstrate the effectiveness of APO which achieves leading citation F1 and improved response quality. Future work can explore extending APO to real-world applications.

heading: Limitation
text: We aim to improve the credibility and reliability of content generated by LLMs using the APO framework. However, it faces limitations such as the narrow scope of datasets used, which may not fully represent the diversity of real-world applications (Liu et al., 2023). The generalization capabilities of the model are also a concern, as the automatic generated data may not cover all scenarios of hallucination. While addressing the high cost of data labeling, the scalability and economic feasibility in larger datasets remain unexplored. The approximation of human citation processes may not capture all the complexities of scholarly writing, and its reliance on external sources raises concerns about the quality and availability of these sources. Potential biases in training data and synthesized data could lead to biased outputs (Wang et al., 2023; Hu et al., 2015). The robustness of the framework against deliberate hallucination and its adaptability to rapidly evolving NLP fields are not fully assessed, highlighting areas for future improvement and research in enhancing LLM reliability.

heading: Ethical Statement
text: The ethical considerations surrounding the use of LLMs that generate citations encompass a range of concerns, including the risk of increased trust without verification, challenges in time-critical decision-making, the assumption of inherent trustworthiness, and copyright issues. Ethically, it is crucial to encourage users to critically engage with and verify machine-generated content to mitigate misinformation and hallucinations. Additionally, recognizing the limitations and potential legal challenges related to copyright when using such attributions is essential. Addressing these ethical issues and educating users on the potential pitfalls becomes increasingly important to ensure the re-
sponsible and informed use of text generated by LLMs.

heading: Acknowledgments
text: We thank the anonymous reviewers for their comments and suggestions. This work is supported by grants: Natural Science Foundation of China (No. 62376067) and Guangdong Basic and Applied Basic Research Foundation (2023A1515110078). Xuebo Liu was sponsored by CCF-Tencent RhinoBird Open Research Fund.

heading: A Further experiments
text: We preform the results on ASQA and ELI5 datasets with more metrics across 3 seeds in Table7.

heading: B Details about Preference Optimization Methods
text: Reinforcement Learning form Human Feedback (RLHF) (Ouyang et al., 2022) uses rewardmodel-based reinforcement learning algorithm to learn the optimal policy. It first learns a reward model from the preference data, then uses an onpolicy PPO algorithm (Schulman et al., 2017) to maximize the learned reward. The reward is learned to use Bradley-Terry model (Bradley and Terry, 1952), which assumes the preference score can be approximated by substituted with point-wise reward. This assumption may lead to an approximation error when preference is deterministic. The PPO algorithm is used on data sampled from generating policy, which may have a different support or distribution drift from preference data, the learned reward model inference on the out-of-distribution data may reduce the accuracy. The process of RLHF needs to train reward model and on-policy PPO algorithm which is complex, time-consuming, and unstable.

heading: Direct Preference Optimization (DPO)
text: (Rafailov et al., 2023) combines off-policy algorithm and Bradley-Terry model to directly learn the generating policy from preference data. The off-policy algorithm is based on KL-regularization reward maximization from off-RL community, which is data efficient, stable and eliminating the need for a reward model. When preference is deterministic which occurs in most cases, the reward of Bradley-Terry model is undefined, which leads to ignoring the KL-regularization term and overfitting the preference dataset.
Identity-mapping Preference Optimization (IPO) (Azar et al., 2023) claims when preferences are deterministic or near deterministic, DPO will lead over-fitting to the preference dataset at the expense of ignoring the KL-regularation term. To optimize the objective, IPO derives an off-policy loss on empirical dataset:
hπθyw,yl = log πθ(yw) πref(yw) − log πθ(yl) πref(yl) , (8)
LIPO(θ, yw, yl) = ( hπyw,yl − 1
2β
)2 . (9)
That means IPO loss will always regularize πθ towards πref by controlling the gap between the loglikelihood ratios log πθ(yw|x)πθ(yl|x) and log πref(yw|x) πref(yl|x) .
Kahneman-Tversky Optimization (KTO) (Kawin et al., 2023) directly maximizes the utility of LLM generations instead of maximizing the log-likelihood of preferences by introducing a Kahneman-Tversky Optimization loss. KTO does not need preference pairs and only knowledge of whether output is desirable or undesirable for a given input.

heading: Sequence Likelihood Calibration (SLiC)
text: (Zhao et al., 2023) uses calibrated likelihood of model-generated sequences to better align with reference sequences in the model’s latent space. It tries to alleviate the problem of MLE that gives probability mass to sparsely observed target sequences, which is used to calculate reward in DPO.

heading: C Details about Pre-processing
text: For ExpertQA dataset, we remove samples whose 1) citations attribute to empty references; 2) documents contain different document IDs but same context. For EVIGSE dataset, we remove samples whose 1) citation attribute to “None” references; 2) do not have reference documents. We further normalize the “supported” label and the citation format for these datasets. The details of each dataset we used for post-training procedure after pre-processing are shown in Table 8.

heading: D Details about Preference Data Creation
text: For each error type, we set the weight to be 1:1:1 to make sure the error samples are balanced. The quantitative assessment about preference dataset is shown in Table 9.

heading: E Post-training Templates
text: The post-training template we used follows the question answering template used by Gao et al. (2023b) since we find that preposition question before document can result in a performance boost when trying ICLCITE method in the preliminary experiments. The concrete templates are shown in Table 10.

heading: F Details about the Instruction
text: The templates employed for generating preference data are detailed in Table 11 for positive instances,
Table 12 for statements exhibiting hallucination errors, Table 13 for statements with synthesis errors, and Table 14 for statements characterized by omission errors.

heading: G Case Study
text: In this section, we perform a detailed case study and demonstrate several examples of each type of error we defined. As shown in Table 15, we classify it as a fabrication error since it uses an undefined entity. In Table 16, we classify it as a synthesis error since it mixes up facts from document 4 and document 5, which results in a factual error. In Table 17, we classify it as a omission error since it used facts from document 4 and document 5, but document 4 is not attributed.

heading: H Related Works of Retrieval Augmentation of LLMs
text: Retrieval augmentation has emerged as a prominent technique aimed at enhancing the accuracy and veracity of LLMs (Gao et al., 2023c; Asai et al., 2023a). Specifically, Sun et al. (2023) couples LLMs with long-term and short-term memories, resulting in improved claim and citation generation. Meanwhile, in order to effectively incorporate external knowledge into LLMs, SearChain proposes a global reasoning chain strategy that facilitates retrieval augmentation generation at each node within the chain (Xu et al., 2023). In another line of research, the self-reflection is leveraged for retrieval
verification during the retrieval-augmented generation process (Li et al., 2023b; Asai et al., 2023b). Despite these advancements, prior studies have not adequately addressed the issue of attribution hallucination (Zuccon et al., 2023). In contrast, we focus on making the model better answer the query and align with the reference.
Algorithm 2 Automatic preference data collection algorithm 1: Input Input Pinit, Critic Mc, Generator Mg, Retriever R, Error instructions E 2: Output Output augmented preference dataset Psyn 3: Ptmp = {} 4: for (q,D,S, C,Lrel,Lsup) in Pinit do 5: for each statement si in S do 6: if the referenced passages Ci are all relevant to q then 7: add (q,D,S:i, C:i,Lrel,L:i,:sup) to Ptmp 8: end if 9: end for
10: end for 11: for (q,D,S:i, C:i,Lrel,L:i,:sup)in Ptmp do 12: Retrieve top-m passages Dir using retriever R given q. 13: Subsequently delete passage d from Dir, if di is predicted as relevant to q using critic Mc. 14: for each statement si in S and its relative attributed passages Ci do 15: if si is supported by Ci then 16: Predicts supported for si using critic Mc given q, di, si, where di ∈ D 17: Add supported passages to the relative attributed passages Ci of statement si. 18: Generate ss∧ri using Mg given q, s:i−1, new Ci. 19: Generate ss∧r̃i using Mg given q,s:i−1, Dir. 20: Generate ss̃∧ri using Mg given q, s:i−1, new Ci and pre-defined error type e. 21: add (q, ss∧ri , s s̃∧r i , D) and (q, s s∧r i , s s∧r̃ i , D) to Psyn. 22: else 23: Generate ss∧ri using Mg given q, s:i−1, new Ci. 24: Generate ss̃∧ri using Mg given q, s:i−1, D− new Ci and pre-defined error type e. 25: add (q, ss∧ri , s s̃∧r i , D) to Psyn. 26: end if 27: end for 28: end for

heading: Input

heading: Input

heading: Input

heading: Input

heading: Input

heading: Output

heading: Question

heading: Output

heading: Question

heading: Output

heading: Question

heading: Future Work
text: The robustness of the framework against deliberate hallucination and its adaptability to rapidly evolving NLP fields are not fully assessed, highlighting areas for future improvement and research in enhancing LLM reliability