Venue: NAACL 2025

# PA-RAG: RAG Alignment via Multi-Perspective Preference Optimization

 Jiayi Wu1, Hengyi Cai2, Lingyong Yan3, Hao Sun4, Xiang Li\({}^{*}\), Shuaiqiang Wang3, Dawei Yin3, Ming Gao1

###### Abstract

The emergence of Retrieval-augmented generation (RAG) has alleviated the issues of outdated and hallucinatory content in the generation of large language models (LLMs), yet it still reveals numerous limitations. When a general-purpose LLM serves as the RAG generator, it often suffers from inadequate response informativeness, response robustness, and citation quality. Past approaches to tackle these limitations, either by incorporating additional steps beyond generating responses or optimizing the generator through supervised fine-tuning (SFT), still failed to align with the RAG requirement thoroughly. Consequently, optimizing the RAG generator from multiple preference perspectives while maintaining its end-to-end LLM form remains a challenge. To bridge this gap, we propose Multiple Perspective **P**reference **A**lignment for **R**etrieval-**A**ugmented **G**eneration (PA-RAG), a method for optimizing the RAG generator to align with RAG requirements comprehensively. Specifically, we construct high-quality instruction fine-tuning data and multi-perspective preference data by sampling varied quality responses from the generator across different prompt documents quality scenarios. Subsequently, we optimize the generator using SFT and Direct Preference Optimization (DPO). Extensive experiments conducted on four question-answer datasets across three LLMs demonstrate that PA-RAG can significantly enhance the performance of RAG generators. Our code and datasets are available at [https://github.com/wujwji/PA-RAG](https://github.com/wujwji/PA-RAG).

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 9091–9112

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

Pre-trained large language models (LLMs) demonstrate impressive question-answering capabilities but also reveal certain drawbacks, such as generating outdated or hallucinatory information [68, 7]. The emergence of retrieval-augmented generation (RAG) [14] has alleviated this issue by incorporating retrieved documents relevant to the question into the prompt, thereby providing the LLMs with information they may not know.

However, while RAG systems help improve factual grounding, the role of the generator in previous work is usually far from well-aligned. Despite their impressive pre-training, off-the-shelf LLMs are not optimally suited for the specific requirements of RAG tasks without proper task alignment.

1) **Helpfulness for RAG application (response informativeness)**[12]: The generator should possess a refined ability to identify and utilize valuable information from the provided context. When presented with a mix of high-quality and less relevant information, it should actively and effectively leverage the valuable content while disregarding the rest. 2) **Harmlessness for RAG application (response robustness)**[13] has alleviated this issue by incorporating retrieved documents relevant to the question into the prompt, thereby providing the LLMs with information they may not know.

However, while RAG systems help improve factual grounding, the role of the generator in previous work is usually far from well-aligned. Despite their impressive pre-training, off-the-shelf LLMs are not optimally suited for the specific requirements of RAG tasks without proper task alignment.

1) **Helpfulness for RAG application (response informativeness)**[12]: The generator should possess a refined ability to identify and utilize valuable information from the provided context. When presented with a mix of high-quality and less relevant information, it should actively and effectively leverage the valuable content while disregarding the rest. 2) **Harmlessness for RAG application (response robustnes)**[13] have shown that RAG is capable of producing an accurate representation of the generated entity. This is because the RAG is able to provide a better representation of the entities, which is more suitable for representing entities.

Figure 1: PA-RAG retains the end-to-end form of the generator while enabling the generator to learn multi-perspective preference information.



2024; Zhu et al., 2024): A well-aligned generator should demonstrate robust noise resistance, capable of filtering out misleading information without being adversely affected. This ability ensures the generated responses remain accurate and coherent even when the input context is imperfect. 3) **Honesty for RAG application (citation quality)**Gao et al. (2023); Sun et al. (2023): The generator should be capable of producing responses that are firmly rooted in the provided documents while maintaining appropriate attribution. This involves explicitly citing the source documents when necessary, and ensuring that the generated content is not only factually accurate but also traceable to the retrieved documents.

To satisfy the aforementioned RAG requirements, existing RAG systems mainly fall into two categories: end-to-end architecture and pipeline architecture. These approaches attempt to bridge the gap between off-the-shelf LLMs and the specific demands of RAG, but both face significant limitations. End-to-end architectures primarily rely on supervised fine-tuning (SFT) of general LLMs Yoran et al. (2024); Fang et al. (2024). These approaches focus on constructing high-quality responses in RAG scenarios. However, through further analysis, we found that the requirements for the generators in RAG tasks are highly context-dependent and often interwoven. These varying and intertwined demands make it challenging for a model to meet all RAG objectives through standard SFT alone, as it does not incorporate the necessary preference information required for adapting to different retrieval scenarios. On the other hand, pipeline architectures introduce additional steps beyond generation, such as re-ranking retrieved documents, filtering irrelevant information, or employing post-hoc verification to ensure that citations support the claims Dong et al. (2024); Yu et al. (2024); Wang et al. (2024); Sun et al. (2023). However, these additional steps can only satisfy certain RAG requirements and still have a considerable gap in aligning with the global RAG requirements. Moreover, these additional steps introduce extra computational costs and time consumption, making it less practical for large-scale deployment. This further inspires us to think: _Is it possible to fully align the generator with the diverse RAG requirements while retaining the simplicity and efficiency of an end-to-end architecture?_

To this end, in this paper, we propose Multi-Perspective **P**reference **A**lignment for **R**etrieval-**Augmented **G**eneration (PA-RAG), aiming to optimize the generator of RAG systems to align comprehensively with specific RAG requirements. As illustrated in Figure 1, PA-RAG maintains the end-to-end architecture of the generator while enabling it to learn multi-perspective preference information. The training of PA-RAG is divided into two phases. The first phase is foundational capability training, where the generator acquires the basic ability to utilize and cite documents through instruction fine-tuning. To construct high-quality instruction fine-tuning data, we utilize ChatGPT to generate complete and correct answers and employ a _citation rewrite_ mechanism to ensure citation quality. The second phase is the multi-perspective preference optimization phase, in which the generator is trained using Direct Preference Optimization (DPO) Rafailov et al. (2023) to learn preference information from different perspectives. This phase encompasses three sub-stages, sequentially enhancing the generator's response informativeness, response robustness, and citation quality. To construct high-quality preference data, we sample responses of varying quality from the generator across scenarios with different document qualities in prompt to build preference data for informativeness and robustness. Additionally, we employ the _citation rewrite_ mechanism to construct preference data for citation quality.

We conducted extensive experiments on four QA datasets and three LLMs, demonstrating that PA-RAG significantly enhances the generator's performance. The improvement achieved by PA-RAG far surpasses the performance gains using only SFT or additional steps. LLMs trained with PA-RAG show an average absolute improvement of 13.97% in correctness, 49.77% in citation recall, and 39.58% in citation precision. Our contributions can be summarized as follows:

* We propose PA-RAG, which achieves comprehensive alignment of LLMs with specific RAG requirements through a multi-stage, multi-perspective training while preserving its end-to-end architecture.
* We publicly release our training data, which includes 58.9k instruction fine-tuning data and 48.7k preference optimization data.
* Through extensive experiments, we demonstrate the effectiveness of PA-RAG, showing that the preference optimization from each perspective is effective 

Preliminaries

In this section, we describe the optimization objectives of the RAG generator, the motivation for multi-perspective optimization, and the rationale for choosing preference optimization over mere instruction fine-tuning.

### Optimization Objectives of the RAG Generator

As illustrated in Figure 1, when answering a question, the RAG generator outputs the corresponding response after receiving the question and multiple relevant documents retrieved by the retriever. Our optimization goal is to enable the generator to fully utilize valuable documents and accurately cite the references corresponding to claims.

The ability to fully utilize valuable documents corresponds to the correctness of the response, meaning that all answers contained within the documents should be included in the generator's output. Formally, let \(G\) represent the generator, \(x\) denotes the input, \(D=\{d_{1},d_{2},...,d_{n}\}\) represent the documents retrieved by the retriever, \(A=\{a_{1},a_{2},...,a_{n}\}\) represent the short answers contained in the documents, and \(y\) denote the response generated by the generator. The correct response by the generator can be expressed as:

\[\begin{split} y&=G(x,D)\\ \text{s.t.}\forall a_{i}&\in A,C(y,a_{i})=\text{True }\end{split} \tag{1}\]

where \(C(y,a_{i})=\text{True}\) implies that the answer \(a_{i}\) is included in \(y\).

Based on the generator's ability to correctly answer questions, it is necessary to enhance its capability to accurately cite documents corresponding to claims, referred to as citation quality. Each response \(y\) contains multiple statements comprising a claim and multiple citations. We aim for each claim to be fully supported by the cited documents, while avoiding irrelevant citations. Formally, we denote the statement as \(s\), the claim as \(c\), and the citation as \(t\). The correct citation of references corresponding to claims can be expressed as follows:

\[y=\{s_{1},s_{2},...,s_{n}\} \tag{2}\]

\[s_{i}=\{\text{``claim''$:$c_{i}$,``citation'':$t_{i}=[t_{i1},t_{i2},...,t_{in}]$}\} \tag{3}\]

\[\forall s_{i}\in y,\phi(\text{concat}(t_{i}),c_{i})=1 \tag{4}\]

\[\begin{split}\forall s_{i}\in y,\forall t_{ij}\in t_{i},\\ \phi(t_{ij},c_{i})=1\vee\phi(\text{concat}(t_{i}\setminus t_{ ij}),c_{i})=0\end{split} \tag{5}\]

Here, \(\text{concat}(t_{i})\) denotes the concatenation of all cited documents, and \(\phi(t,c)=1\) indicates that the claim \(c\) is fully supported by the citation \(t\). Formula 4 indicates that all claims can be fully supported by their citations, while Formula 5 ensures that no irrelevant documents are cited.

### Preference Optimization Under Different Document Qualities

As described in SS 2.1, enhancing the correctness of the RAG generator is the primary objective. However, optimizing correctness is highly complex. Specifically, the probability of a model correctly answering a question is \(P(y|x)\). In the RAG framework, this probability depends on the distribution of the retrieved documents and can be formulated as \(P(y|x)=\sum_{D}[P(D|x)\cdot P(y|D,x)]\). However, the retriever in a RAG system is not perfect, leading to variability in document quality under different circumstances, which results in a non-uniform distribution of \(P(D|x)\). Therefore, it is essential to optimize the generator under different document quality conditions.

We categorize document quality into two types. The first type involves scenarios where all documents contain relevant answers. In this scenario, we expect the generator to fully utilize the valuable documents and produce complete answers. Therefore, when the documents are of high-quality, we need to focus on optimizing response informativeness. The second type includes scenarios where the document set contains noisy documents. In this case, we expect the generator to avoid the interference of noisy documents and still maintain the completeness of its answers. Thus, when dealing with low-quality documents, the emphasis shifts to optimizing response robustness. However, optimizing for informativeness and robustness pulls the generator in different directions. The former encourages referencing more documents, while the latter demands ignoring more. Standard instruction fine-tuning is insufficient to teach the model how to balance these competing demands (documents encouraged to be referenced are valuable, while those that should be ignored are considered noisy, see SS 4.5 for details), making it difficult to optimize the generator effectively. To address this challenge, we introduce preference information and perform multi-perspective preference optimization to guide the generator in focusing on valuable documents while ignoring noisy ones 



## 3 Methodology

After defining the optimization objectives, we trained the generator in two phases. The first phase enabled a general LLM to acquire basic RAG capabilities. The method for constructing the training data for this phase is detailed in SS 3.1. The second phase involved multi-perspective preference optimization, further enhancing the generator's response informativeness, response robustness, and citation quality. The method for constructing the training data for this phase is described in SS 3.2.

### Instruction Fine-tuning for Basic RAG Capabilities

During the instruction fine-tuning phase, we aimed to equip the generator with fundamental abilities to utilize and cite documents following the optimization objectives outlined in SS 2.1. When constructing the training data, we employ ChatGPT-3.5 (GPT-3.5-Turbo-1106) and introduce a _citation rewrite_ mechanism to create near-perfect responses. An overview of the instruction fine-tuning data construction is illustrated in Figure 2.

Firstly, we need to acquire high-quality documents after obtaining the questions and short answers provided by existing datasets. We use the RAG retriever to retrieve the top 100 most relevant documents from the retrieval corpus, and then filter to retain the golden documents that contain the short answer. Subsequently, we select up to five documents from all the golden documents as prompt documents, ensuring that all short answers are included in the prompt documents.

Secondly, we need to construct responses that include all short answers and exhibit high citation quality. Specifically, we use ChatGPT-3.5 to generate responses. To enhance the quality of the responses, we include instructions and short answer hints in the ChatGPT prompt. Detailed prompt can be found in Appendix D.

Although ChatGPT-3.5 is quite powerful and the prompt includes short answer hints, the quality of the citations in the answers is unsatisfactory. Therefore, we introduce a _citation rewrite_ mechanism to improve the quality of substandard citations, which is divided into the following steps: **First**, verify the citation. We use a Natural Language Inference (NLI) model to validate whether the document corresponding to each citation (premise) in the statement can satisfy the claim (hypothesis). **Second**, construct the citation. If the citation in the first step cannot support the claim, we traverse the power set of all prompt documents as citations to explore a feasible citation scheme. **Third**, simplify the citation. If the citation can support the claim, we check one by one whether the citation is irrelevant to the claim and remove irrelevant citations.

After generating answers with ChatGPT-3.5 and performing _citation rewrite_, we filter to retain responses that include all short answers and contain only accurate citations to serve as the training data for the instruction fine-tuning of the generator, which is then subsequently fine-tuned.

### Preference Optimization of the Generator through DPO

Once the generator has acquired the fundamental ability to utilize and reference documents, we sequentially optimize the generator from the perspectives of response informativeness, response robustness, and citation quality during the preference optimization phase. Preference optimization requires constructing data with preference information, including an input, a superior answer (chosen output), and an inferior answer (rejected output). We will progressively explain the methods for constructing preference data from each perspective. An overview of the preference data construction is illustrated in Figure 3.

Figure 2: Overview of instruction fine-tuning data construction.





#### 3.2.1 Response Informativeness

Response informativeness refers to the completeness of the answer in the response. The optimization objective from this perspective is to make the generator fully utilize the document that contains the short answer.

**Input Construction:** Similar to constructing the input component of the instruction fine-tuning data, the input includes the instruction, the question, and the high-quality prompt documents containing all short answers from up to 5 golden documents.

**Chosen Output Construction:** The construction method is consistent with the output component of the instruction fine-tuning data, using ChatGPT-3.5 and the _citation rewrite_ mechanism to construct a response that includes all short answers and the accurate citation.

**Rejected Output Construction:** To simulate the scenario where the generator ignores parts of the golden documents, we delete some golden documents from the prompt, generate responses with the generator, and filter to retain inferior answers that are incomplete. Since some documents are removed from the prompt, we need to adjust the citation numbers after obtaining the inferior answers. Specific examples of citation number adjustments are shown in Appendix E.

#### 3.2.2 Response Robustness

Response robustness refers to the ability of the generator to resist interference. The optimization objective from this perspective is to enable the generator to avoid interference from noisy documents.

**Noisy document construction:** We categorize noisy documents into two types. The first type is documents related to the question but does not contain the answer, from which we randomly select two documents that do not contain short answers from the top 100 relevant retrieved documents. The second type is irrelevant documents to the question, from which we randomly select two documents that do not contain short answers from the retrieval documents of other questions.

**Input construction:** The input includes the instruction, the question, and the low-quality prompt documents that mix up to five golden documents and four noisy documents.

**Chosen output construction:** To simulate a scenario where the generator ignores all noisy documents, we use the chosen output generated in the absence of noisy documents in SS 3.2.1. Since there are no noisy documents, we need to adjust the citation numbers. Specific examples of citation number adjustments are shown in Appendix E.

**Rejected output construction:** Use inputs containing low-quality documents, generate responses with the generator, and filter to retain inferior answers that are incomplete.

#### 3.2.3 Citation Quality

Citation quality refers to the generator's ability to cite documents correctly. The optimization objective from this perspective is to enable the generator to cite documents related to the claim correctly and

Figure 3: Overview of preference data construction.



to avoid citing irrelevant documents.

Data construction is divided into two steps: First, we use the generator to generate responses and filter to retain those containing all short answers. Second, we use the _citation rewrite_ mechanism to identify incorrect citations that fail the NLI model verification or cite irrelevant documents as rejected output and than correct them as chosen output.

**Construction of input:** The input includes the instruction, the questions, the prompt documents, and the response up to the incorrect citation.

**Construction of chosen output:** Correct citation after citation rewrite.

**Construction of rejected output:** Incorrect citation that fails the NLI model verification or cites irrelevant documents.

#### 3.2.4 Staged Preference Optimization

Given the substantial discrepancies in preference information from different perspectives, we divide the preference optimization process into multiple sub-stages to independently optimize the RAG preferences from each perspective. Specifically, the first and second sub-stages optimize response informativeness and response robustness, respectively. The rejected outputs in these stages are generated by the generator that has undergone instruction fine-tuning. The third sub-stage optimizes citation quality, with preference data generated by the generator that has undergone optimization in the first and second sub-stages.

## 4 Experiment

### Experimental Setup

#### 4.1.1 Dataset and Evaluation Methodology

DatasetsThe questions used for constructing our training data are sourced from the ASQA (Stelmakh et al., 2022), WebQuestions (Berant et al., 2013), and Natural Questions (Kwiatkowski et al., 2019) training splits. Detailed statistics for the training set can be found in Appendix C. The evaluation data is sourced from the test splits of the three datasets mentioned above, along with TriviaQA Joshi et al. (2017), which serves as an unseen dataset to assess the **out-of-domain** generalizability of the generator.

Evaluation MetricsTo align with the optimization objectives discussed in SS 2.1, we evaluate the generator's performance regarding correctness and citation quality. Following ALCE Gao et al. (2023a) and VtgSun et al. (2023), for correctness, we assess whether the short answers (provided by the dataset) are exact substrings of the generation to calculate the **exact match** (EM) score. For citation quality, we use **citation recall** to evaluate whether the output is fully supported by the cited documents and **citation precision** to assess whether irrelevant documents are cited.

#### 4.1.2 Implementation Details

We selected three general LLMs as the base RAG generator: Llama2-7b-chat, Llama2-13b-chat, and Llama3-8b-Instruct. For the NLI model of citation rewrite and citation quality evaluation, we follow the works of Gao et al. (2023a), Sun et al. (2023), and Huang et al. (2024), utilizing the NLI model TRUE Honovich et al. (2022), a T5-11B model that is fine-tuned on a collection of NLI datasets. Meanwhile, ALCE has verified through extensive human evaluations that this NLI model correlates with human judgment. For retrieval, we employed the Wikipedia dump from December 20, 2018, as our retrieval corpus and used GTR Ni et al. (2022) as our dense retriever.

We performed full fine-tuning on the generators, with the training hyperparameters detailed in Appendix A. All generators were trained on the same dataset, in which the RAG generator used to generate preference data was the Llama2-7b-chat at different stages of fine-tuning.

#### 4.1.3 Baselines

We adopt the following baseline systems. 1) **Base generator**, the general LLM without further optimization. 2) **RetRobust-13B**Yoran et al. (2024), a method that enhances the robustness of the RAG generator through SFT. 3) **Self-RAG-13B**Asai et al. (2024), a method that improves the performance of the RAG generator through a rank-then-generate pipeline. 4) **SFT on chosen**, to compare preference optimization and SFT, we use the chosen output of preference data for SFT on the Llama2-7b-chat. All the methods above were evaluated using the same prompt documents used for PA-RAG to facilitate a fair comparison.

### Main Results

The evaluation results of PA-RAG and the baseline across four QA datasets are presented in Table 1. We have made the following observations: First, PA-RAG consistently and significantly improves the performance of various LLMs in RAG scenarios, with an average increase of 13.97% in EM 

score, 49.77% in citation recall, and 39.58% in citation precision. This demonstrates that our training data is applicable to different LLMs, showcasing strong generalizability. Meanwhile, PA-RAG does not compromise the fluency of the generator's outputs.1 Second, PA-RAG significantly outperforms the baselines, indicating that compared to end-to-end generators trained via SFT and pipeline generators, the end-to-end generator trained through preference optimization aligns better with RAG preferences, resulting in superior performance.

Footnote 1: A detailed analysis of response fluency is presented in Appendix H.

### Ablation Study

The evaluation results for each training phase are presented in Table 2. We observed that each training phase generally contributes to improvements in both correctness and citation quality, indicating that optimization of RAG requirements in each perspective is necessary. Furthermore, correctness and citation quality are complementary to each other. Detailed Analysis of why optimizing citation quality might have a negative impact on citation recall is presented in Appendix G.

To more intuitively demonstrate the impact of preference optimization on the generator, we present detailed LLM evaluation in Appendix J and case studies in Appendix K.

### Impact of Preference Optimization Order

In the main experiment, we first optimized response informativeness, followed by response robustness. We further investigated the impact of altering the

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline \hline
**Datasets** & \multicolumn{3}{c}{**ASQA**} & \multicolumn{3}{c}{**WebQuestions**} & \multicolumn{3}{c}{**Natural Questions**} & \multicolumn{3}{c}{**TriviaQA**} \\ \cline{2-16}  & **Correct** & **Citation** & **Correct** & **Citation** & **Correct** & **Citation** & **Correct** & **Citation** & **Correct** & **Citation** \\ \cline{2-16}

- & 33.71 &
- & 43.08 &
- & 73.13 &
- \\ Self-RAG-13B & 31.61 & 58.77 & 68.88 & 38.64 & 60.46 & 71.75 & 48.09 & 63.65 & 72.75 & 66.95 & 63.62 & 65.24 \\ SFT on Chosen & 34.49 & 72.61 & 69.29 & 39.07 & 80.81 & 77.66 & 50.30 & 73.77 & 70.76 & 64.59 & 66.15 & 62.75 \\ \hline \multicolumn{10}{c}{**PA-RAG (Ours)**} \\ \hline \multirow{3}{*}{Llama2-7h+PA-RAG} & **46.16** & **77.66** & **76.82** & **48.37** & **79.36** & **78.16** & **63.27** & **76.60** & **76.49** & **72.52** & **70.86** & **69.89** \\  & **46.53** & **80.68** & **77.73** & **48.72** & **80.90** & **78.41** & **62.71** & **78.72** & **75.89** & **73.56** & **71.96** & **68.95** \\ \cline{1-1}  & **47.03** & **81.97** & **79.33** & **48.59** & **84.31** & **79.84** & **62.29** & **81.26** & **79.08** & **73.79** & **74.07** & **71.05** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Main results of PA-RAG and baselines on four QA datasets. The best performances within the same backbone generator are marked in **bold**.

\begin{table}
\begin{tabular}{l l r r r r r r r r r r r r r r} \hline \hline
**Datasets** & \multicolumn{3}{c}{**ASOA**} & \multicolumn{3}{c}{**WebQuestions**} & \multicolumn{3}{c}{**Natural Questions**} & \multicolumn{3}{c}{**TriviaQA**} \\ \cline{2-16}  & **Correct** & \multicolumn{3}{c}{**Citation**} & \multicolumn{3}{c}{**Correct**} & \multicolumn{3}{c}{**Citation**} & \multicolumn{3}{c}{**Correct**} & \multicolumn{3}{c}{**Citation**} & \multicolumn{3}{c}{**Correct**} & \multicolumn{3}{c}{**Citation**} & \multicolumn{3}{c}{**Correct**} & \multicolumn{3}{c}{**Citation**} \\ \cline{2-16}
**Generators** & **EM** & **Rec** & **Prec** & **F1** & **EM** & **Rec** & **Prec** & **F1** & **EM** & **Rec** & **Prec** & **F1** & **EM** & **Rec** & **Prec** & **F1** \\ \hline \multicolumn{10}{c}{**Llama2-7h-CHAT**} \\ \hline Base Generator & 35.61 & 20.51 & 34.51 & 25.73 & 40.91 & 24.36 & 40.27 & 30.36 & 48.82 & 20.06 & 34.71 & 25.43 & 67.69 & 22.81 & 37.27 & 28.30 \\ +IFT & 37.98 & 75.69 & 70.71 & 72.83 & 42.76 & 75.85 & 70.50 & 73.08 & 54.59 & 75.44 & 68.59 & 71.54 & 68.51 & 65.41 & 60.34 & 62.77 \\ +IFT+RI & 43.95 & **77.68** & 68.32 & 72.70 & 47.75 & 78.93 & 70.37 & 53.81 & 67.19 & 76.19 & 66.22 & 70.85 & 71.61 & 69.00 & 60.23 & 64.32 \\ +IFT+RI+RR & **46.37** & 75.25 & 71.58 & 73.37 & 48.03 & 79.00 & 74.31 & 76.58 & 62.99 & 75.55 & 71.12 & 73.27 & 72.36 & 69.23 & 64.74 & 66.91 \\ +IFT+RT+IR+CR & 46.16 & 77.66 & **76.82** & **77.74** & **48.37** & **79.36** & **78.16** & **78.76** & **63.27** & **76.60** & **76.49** & **76.54** & **72.52** & **70.86** & **69.89** & **70.37** \\ \hline \multicolumn{10}{c}{**Llama2-13b-CAAT**} \\ \hline Base Generator & 27.24 & 16.02 & 42.87 & 19.49 & 25.45 & 13.75 & 21.03 & 16.63 & 34.47 & 14.58 & 22.57 & 17.72 & 54.55 & 15.31 & 23.31 & 18.49 \\ +IFT & 38.23 & 76.28 & 70.82 & 73.45 & 42.04 & 78.77 & 73.05 & 75.85 & 54.85 & 75.75 & 69.91 & 72.71 & 69.54 & 68.69 & 61.81 & 64.25 \\ +IFT+RI & 42.46 & 80.24 & 70.05 & 74.80 & 46.66 & **81.74** & 71.59 & 76.33 & 58.68 & 78.33 & 68.79 & 72.85 & 71.5 & 70.26 & 60.88 & 65.24 \\ +IFT+RI+RR & 45.50 & 77.72 & 75.57 & 76.63 & 48.16 & 80.23 & 77.08 & 78.79 & 61.58 & 76.82 & 74.17 & 75.47 & 72.78 & 68.79 & 66.57 & 67.66 \\ +IFT+RI+RR+CO 

optimization order on performance, with results shown in Table 3. Our findings are as follows: First, optimizing response robustness before response informativeness resulted in lower performance than the main experimental setup. Second, skipping informativeness and directly optimizing robustness or merging the data to optimize simultaneously can lead to a decline in performance. We believe this is because optimizing informativeness enables the generator to learn how to effectively utilize retrieved documents, which is the fundamental capability in the RAG system. In contrast, optimizing robustness requires the generator to learn to reject irrelevant documents, which is a more advanced skill.2 Learning skills at different levels requires a reasonable arrangement in learning order, which is similar to curriculum learning Bengio et al. (2009). Neglecting the foundational skills or mixing different skill levels during learning may hurt the model's original capabilities.

Footnote 2: Detailed analysis of why rejecting irrelevant documents is a more advanced skill is showcased in Appendix F.

### Further Comparison of DPO and SFT

As mentioned in SS 2.2, there is a significant disparity in the optimization directions of response informativeness and robustness. We further explore the differences between DPO and SFT when optimizing these two directions sequentially. As shown in Table 4, we observe that, when using SFT for training, optimizing response informativeness first can enhance the generator's performance. However, subsequent optimization for response robustness leato a significant performance decline, exposing SFT's vulnerability to catastrophic forgetting French and Chater (2002) when undergoing two relatively different optimizations consecutively. In contrast, DPO handles this situation well, consistently improving the generator's performance.

## 5 Related Work

### Retrieval-Augmented Generator

To satisfy the requirements of RAG system, existing retrieval-augmented generators are primarily categorized into end-to-end and pipeline frameworks. In end-to-end frameworks, models such as RetRobust Yoran et al. (2024a) and RAAT Fang et al. (2024) enhance the robustness of the generator by performing SFT on high-quality data. In pipeline frameworks, models like DPA-RAG Dong et al. (2024a), REAR Wang et al. (2024), Self-RAG Asai et al. (2024), and RankRadg Yu et al. (2024) enhance the robustness of the generator by explicitly re-ranking the retrieved documents. Furthermore, Vtg Sun et al. (2023) improves citation quality by introducing explicit citation verification and modification. However, these methods still struggle to align the generator with the RAG requirements fully.

### Fine-tuning Approaches for LMs

The mainstream approaches for fine-tuning language models in generative tasks include SFT Radford (2018) and reinforcement learning from human feedback (RLHF) Christiano et al. (2017). SFT involves constructing input-output pairs to teach the model how to complete tasks. RLHF methods like DPO Rafailov et al. (2023) incorporate training data that includes both superior and inferior outputs, allowing the model to learn preference information. RLHF is more effective than SFT in aligning the model with task preferences.

## 6 Conclusion

In this work, we propose PA-RAG, a method for optimizing the generator of RAG systems to align with specific RAG requirements comprehensively. The training process includes instruction fine-tuning and multi-perspective preference optimization. We conducted extensive experiments on four QA benchmarks and three LLMs, demonstrating that PA-RAG can significantly enhance the generator's response informativeness, response robustness, and citation quality.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Generators** & **ASQA** & **WebQ** & **NQ** & **TOA** \\ \hline   lllama-7b-Chat+IFT & 37.98 & 42.76 & 54.59 & 68.51 \\  +SFT RI(Chosen) & 38.93 & 43.69 & 54.84 & 69.10 \\  +SF RT H+RR(Chosen) & 32.17 & 37.02 & 47.01 & 62.56 \\  +DPO RI & 43.95 & 47.75 & 60.11 & 71.61 \\  +DPO RI+RR & 46.37 & 48.03 & 62.99 & 72.36 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Further comparison of DPO and SFT under response informativeness (RI) and response robustness (RR). Results of the EM score.

\begin{table}
\begin{tabular}{l l l l l} \hline \hline
**Generators** & **ASQA** & **WebQ** & **NQ** & **TQA** \\ \hline   llama-7b-Chat+IFT & 37.98 & 42.76 & 54.59 & 68.51 \\  +RF RI(Chosen) & 38.93 & 43.69 & 54.84 & 69.10 \\  +SFT RI+RR(Chosen) & 32.17 & 37.02 & 47.01 & 62.56 \\  +DPO RI & 43.95 & 47.75 & 60.11 & 71.61 \\  +DPO RI+RR & 46.37 & 48.03 & 62.99 & 72.36 \\ \hline \hline \end{tabular}
\end{table}
Table 3: EM score results of different preference optimization orders. Best scores are highlighted in **bold**.

 



### Limitations

Our method requires fine-tuning in four stages, including one instruction fine-tuning stage and three preference optimization stages. This results in a cumbersome search for the optimal hyperparameter settings during training. We have presented the hyperparameter details in Appendix A to prevent other researchers from duplicating the search for the best hyperparameter settings.

## Ethics Statement

This work was conducted in strict compliance with the ACL Ethics Policy. All datasets and models used for experiment are publicly available. Furthermore, our work aims to explore a RAG generator training method. We do not foresee any negative ethical impacts arising from our work.

## Acknowledgement

This work was supported by the Shanghai "Science and Technology Innovation Action Plan" Project (No.23511100700).

## References

* Asai et al. [2024] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Bengio et al. [2009] Yoshua Bengio, Jerome Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In _Proceedings of the 26th annual international conference on machine learning_, pages 41-48.
* Berant et al. [2013] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL_, pages 1533-1544. ACL.
* Christiano et al. [2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. _Advances in neural information processing systems_, 30.
* Dong et al. [2024a] Guanting Dong, Yutao Zhu, Chenghao Zhang, Zechen Wang, Zhicheng Dou, and Ji-Rong Wen. 2024a. Understand what LLM needs: Dual preference alignment for retrieval-augmented generation. _CoRR_, abs/2406.18676.
* Dong et al. [2024b] Qian Dong, Yiding Liu, Qingyao Ai, Zhijing Wu, Haitao Li, Yiqun Liu, Shuiaiqiang Wang, Dawei Yin, and Shaoping Ma. 2024b. Unsupervised large language model alignment for information retrieval via contrastive feedback. In _Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024_, pages 48-58. ACM.
* Fang et al. [2024] Feiteng Fang, Yuelin Bai, Shiwen Ni, Min Yang, Xiaojun Chen, and Ruifeng Xu. 2024. Enhancing noise robustness of retrieval-augmented language models with adaptive adversarial training. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024_, pages 10028-10039. Association for Computational Linguistics.
* French and Chater [2002] Robert M. French and Nick Chater. 2002. Using noise to compute error surfaces in connectionist networks: A novel means of reducing catastrophic forgetting. _Neural Comput._, 14(7):1755-1769.
* Gao et al. [2023a] Tianyu Gao, Howard Yen, Jiantong Yu, and Danqi Chen. 2023a. Enabling large language models to generate text with citations. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_, pages 6465-6488. Association for Computational Linguistics.
* Gao et al. [2023b] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2023b. Retrieval-augmented generation for large language models: A survey. _CoRR_, abs/2312.10997.
* Honovich et al. [2022] Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansky, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: re-evaluating factual consistency evaluation. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022_, pages 3905-3920. Association for Computational Linguistics.
* Huang et al. [2024] Chengyu Huang, Zeqiu Wu, Yushi Hu, and Wenya Wang. 2024. Training language models to generate text with citations via fine-grained rewards. In _Proceedings of the 62nd Annual Meeting of the Association for Computation Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024_, pages 2926-2949. Association for Computational Linguistics.
* August 4, Volume


_1: Long Papers_, pages 1601-1611. Association for Computational Linguistics.
* Kwiatkowski et al. (2018) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2018. Natural questions: a benchmark for question answering research. _Trans. Assoc. Comput. Linguistics_, 7:452-466.
* Li et al. (2023) Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percry Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval).
* Ni et al. (2022) Jianmo Ni, Chen Qu, Jing Lu, Zhuyu Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, and Yinfei Yang. 2022. Large dual encoders are generalizable retrievers. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_, pages 9844-9855. Association for Computational Linguistics.
* Pillutla et al. (2021) Krishna Pillutla, Swohba Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. MAUVE: measuring the gap between neural text and human text using divergence frontiers. In _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_, pages 4816-4828.
* Radford (2018) Alec Radford. 2018. Improving language understanding by generative pre-training.
* 16, 2023_.
* Stelmakh et al. (2022) Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022. ASQA: factoid questions meet long-form answers. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_, pages 8273-8288. Association for Computational Linguistics.
* Sun et al. (2023) Hao Sun, Hengyi Cai, Bo Wang, Yingyan Hou, Xiaochui Wei, Shuaiqiang Wang, Yan Zhang, and Dawei Yin. 2023. Towards verifiable text generation with evolving memory and self-reflection. _arXiv preprint arXiv:2312.09075_.
* Townhidul Islam Tonmay et al. (2024) S. M. Towhidul Islam Tonmay, S. M. Mehedi Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Amad Chadha, and Amitava Das. 2024. A comprehensive survey of hallucination mitigation techniques in large language models. _CoRR_, abs/2401.01313.
* 16, 2023_.
* Wang et al. (2024) Yuhao Wang, Ruiyang Ren, Junyi Li, Wayne Xin Zhao, Jing Liu, and Ji-Rong Wen. 2024. Rear: A relevance-aware retrieval-augmented framework for open-domain question answering. _arXiv preprint arXiv:2402.17497_.
* Moran et al. (2024a) Ori Moran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024a. Making retrieval-augmented language models robust to irrelevant context. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Moran et al. (2024b) Ori Moran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024b. Making retrieval-augmented language models robust to irrelevant context. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Yu et al. (2024) Yue Yu, Wei Ping, Zihan Liu,oxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Rankrag: Unifying context ranking with retrieval-augmented generation in llms. _arXiv preprint arXiv:2407.02485_.
* Zhu et al. (2024) Junda Zhu, Lingyong Yan, Haibo Shi, Dawei Yin, and Lei Sha. 2024. Atm: Adversarial tuning multi-agent system makes a robust retrieval-augmented generator. _arXiv preprint arXiv:2405.18111_.

## Appendix A Training Hyperparameters

We utilized full fine-tuning for all training stages and employed the same hyperparameter settings for all models.

During the instruction fine-tuning phase, we set the batch size to 128, the learning rate to 2e-5, and trained for one epoch.

In the preference optimization phase, we set the batch size to 64 and trained for one epoch for all stages. For the optimization stages of response informativeness and response robustness, the learning rate is 2e-6. In the citation quality optimization stage, the learning rate is 2e-7 



## Appendix B Compute Resources

Our experiments were conducted on a server equipped with 1TB of memory and 4 Nvidia A800 80G GPUs.

## Appendix C Training Data Statistics

Detailed information on the data size and composition for each training stage can be found in Table 5.

### Distribution of Citation Complexity in Instruction Fine-tuning

In the instruction fine-tuning training data, the distribution of the number of citations per claim is shown in Table 6.

### The Disparity Between "Chosen" and "Rejected" in Preference Data.

As shown in Table 7, the disparity in correctness and citation quality between the "Chosen" and "Rejected" in the preference data are presented. We can observe that the preference differences within the preference data are substantial enough to provide relatively rich preference information.

## Appendix D Prompt

[TABLE:A2.T9][ENDTAU]

**Instruction:** Write an accurate, engaging, and concise answer for the given question using only the provided search results (some of which might be irrelevant) and cite them properly. Use an unbiased and journalistic tone. Always cite for any factual claim. When citing several search results, use [1][2][3].cite at least one document and at most three documents in each sentence. If multiple documents support the sentence, only cite a minimum sufficient subset of the documents.

**Question:** {Question}

The final answer should contain the following short answers: {Short answers}

**Documents:** {Documents}

**Answer:**

## Appendix E Examples of Adjusting the Citation Number

### Rejected Output for Response Informativeness

When constructing the DPO training data for response informativeness, a question and five golden documents were given, numbered [1][2][3][4][5]. When constructing the rejected output, we would remove some golden doc

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & **1** & **2** & **3** & **4** & **5** \\ \hline ASQA & 70.98\% & 24.17\% & 4.24\% & 0.46\% & 0.05\% \\ WebQ & 71.43\% & 21.54\% & 6.26\% & 0.65\% & 0.05\% \\ NQ & 71.87\% & 21.44\% & 6.00\% & 0.52\% & 0.16\% \\ \hline \hline \end{tabular}
\end{table}
Table 6: Distribution of Citation Complexity

\begin{table}
\begin{tabular}{c l c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multicolumn{1}{c}{**Preference**} & \multicolumn{2}{c}{**Correct**} & \multicolumn{2}{c}{**Citation**} \\ \cline{2-5}  & \multicolumn{2}{c}{**EM**} & \multicolumn{1}{c}{**Rec**} & \multicolumn{1}{c}{**Prec**} \\ \hline \multicolumn{5}{|c|}{**Response Informativeness**} \\ ASQA & Chosen & 100 & 100 & 100 \\  & Rejected & 38.33 & 77.59 & 75.51 \\ \hline WebQ & Chosen & 100 & 100 & 100 \\  & Rejected & 21.14 & 72.90 & 71.00 \\ \hline NQ & Chosen & 100 & 100 & 100 \\  & Rejected & 9.07 & 70.53 & 69.62 \\ \hline \multicolumn{5}{|c|}{**Response Robustness**} \\ ASQA & Chosen & 100 & 100 & 100 \\  & Rejected & 37.30 & 76.62 & 69.19 \\ \hline WebQ & Chosen & 100 & 100 & 100 \\  & Rejected & 21.14 & 74.71 & 67.51 \\ \hline NQ & Chosen & 100 & 100 & 100 \\  & Rejected & 6.08 & 73.19 & 67.40 \\ \hline \multicolumn{5}{|c|}{**Citation Quality**} \\ ASQA & Chosen & - & 100 & 100 \\  & Rejected & - & 27.42 & 12.13 \\ \hline WebQ & Chosen & - & 100 & 100 \\  & Rejected & - & 25.88 & 13.17 \\ \hline NQ & Chosen & - & 100 & 100 \\  & Rejected & - & 27.15 & 13.13 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Preference disparity within the preference data

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & **1** & **2** & **3** & **4** & **5** \\ \hline ASQA & 70.98\% & 24.17\% & 4.24\% & 0.46\% & 0.05\% \\ WebQ & 71.43\% & 21.54\% & 6.26\% & 0.65\% & 0.05\% \\ NQ & 71.87\% & 21.44\% & 6.00\% & 0.52\% & 0.16\% \\ \hline \hline \end{tabular}
\end{table}
Table 5: Trianing data statistics

uments, such as documents [1] and [3], leaving documents [2], [4], and [5]. To maintain a unified input format, the documents will be renumbered starting from 1. The mapping of the new and old document numbers is shown as Table 8.

The generator outputs an answer based on the question and the remaining documents: Claim1[2]. Claim2[1][3].

Since the citation numbers in the answer generated by the generator are the new document numbers, and the documents in the input part of the DPO training data are the old numbers, we need to revert the new numbers to the old ones using to the table above. The restored answer is as follows: Claim1[4]. Claim2[2][5].

### Chosen Output for Response Robustness

When constructing the training data for response robustness, given a question and documents numbered [1][2][3][4][5][6][7][8], where [1][3][4][7] are golden documents and [2][5][6][8] are noisy documents. When constructing the chosen output, we will remove all the noisy documents. The mapping of the new and old document numbers is shown as Table 9.

ChatGPT outputs an answer based on the question and the remaining documents: Claim1[1][2]. Claim2[3][4].

Restore the new numbers to the old ones according to the table above. The restored answer is as follows: Claim1[1][3]. Claim2[4][7].

## Appendix F Detailed Analysis of Why Rejecting Irrelevant Documents is a More Advanced Skill

As shown in Table 3, in our experiments, skipping the training for informativeness (learning to utilize relevant documents) and directly conducting the training for robustness (learning to ignore irrelevant documents) leads to a decline in model performance. This is because the model may not realize that the documents it needs to ignore are the irrelevant ones, resulting in indiscriminate neglect of most documents. In contrast, performing training for informativeness first and then training for robustness improves performance. This is because the model has already developed a basic ability to utilize documents and can recognize that it should ignore only the irrelevant documents. Therefore, ignoring irrelevant documents is a more advanced skill than fully utilizing every relevant document. If the results were the opposite, it would be counterintuitive and inexplicable.

Appendix G Detailed Analysis of Why Optimizing Citation Quality Might Have a Negative Impact on citation Recall.

It is challenging to simultaneously improve both the precision and recall of citation quality. While enhancing recall, the model may become stricter in assessing the relevance of documents, which can lead to a decrease in precision. Therefore, we introduced the F1 score to evaluate citation quality from a more balanced perspective. Our experimental results indicate that optimizing response informativeness and response robustness can enhance answer correctness, while optimizing citation quality can improve the citation F1 score.

## Appendix H Further analysis of whether PA-RAG affects the fluency of responses

We follow ALCE Gao et al. (2023a), using MAUVE Pillutla et al. (2021) to evaluate the fluency of the model's responses. The fluency of the text output by models at various training stages on the ASQA dataset is shown in the Table 10:

We found that PA-RAG does not affect the fluency of the model's output.

\begin{table}
\begin{tabular}{c c} \hline \hline
**Old ID** & **New ID** \\ \hline
[2] & [1] \\
[4] & [2] \\
[5] & [3] \\ \hline \hline \end{tabular}
\end{table}
Table 8:

\begin{table}
\begin{tabular}{c c} \hline \hline
**Old ID** & **New ID** \\ \hline
[1] & [1] \\
[3] & [2] \\
[4] & [3] \\
[7] & [4] \\ \hline \hline \end{tabular}
\end{table}
Table 9: 



## Appendix I Specific Criteria and Examples of Inferior Answers that are Incomplete

Inferior answers refer to responses that do not meet the criteria for correct answers. The criteria for the correct answers are shown in SS 2.1. Transform these into criteria for inferior answers is as follows: Let \(G\) represent the generator, \(x\) denotes the input, \(D=\{d_{1},d_{2},...,d_{n}\}\) represent the documents retrieved by the retriever, \(A=\{a_{1},a_{2},...,a_{n}\}\) represent the short answers contained in the documents, and \(y\) denote the response generated by the generator. The correct response by the generator can be expressed as:

\[y=G(x,D),\exists a_{i}\in A,C(y,a_{i})=\text{False} \tag{6}\]

where \(C(y,a_{i})=\text{False}\) implies that the answer \(a_{i}\) is not included in \(y\).

Specific examples of inferior answers are as follows:

```
PromptforClisting5
**Question:** What is the name of the pirate in SpongeBob? Goldenshortanswwords: ['Patchy', 'Burger Beard'] Inferiorlonganswer:** The pirate in SpongeBob is named Patchy the Pirate[1]. Patchy the Pirate is the president of the SpongeBob SquarePants fan club and is portrayed as a tyrannical pirate in the plot of the SpongeBob SquarePants film series[1][2].
```

In this inferior answer, the answer "Burger Beard" is missing. We determine whether an answer is inferior by checking if any golden short answer is not included in the long answer.

## Appendix J LLM Evaluation

As an accuracy evaluation metric, Exact Match (EM) cannot directly reflect the response informativeness and response robustness. We follow AlpacaEval Li et al. (2023) and use the LLM (GPT-3.5-Turbo) evaluation to qualitatively judge whether the response informativeness and response robustness increase after RAG preference alignment. The prompt for the LLM judgement is as follows:

``` PromptforClisting5
**Example:** I want you to create a leaderboard of different of large-language models. To do so, I will give you the instructions (prompts) given to the models, and the responses of two models. Please rank the models according to the correctness of their answers and whether they make use of valuable documents and avoid being interfered with by irrelevant documents. All inputs and outputs should be python dictionaries. Here is the prompt: {instruction} Here are the outputs of the models: [ ["model": "model_1",answer": '{output_1]^*}, {"model": "model_2",answer": '{output_2}"} Now please rank the models by the quality of their answers, so that the model with rank 1 has the best output. Then return a list of the model names and ranks, i.e., produce the following output: [ {'model': `model-name>, 'rank': <model-rank>}, {'model': <model-name>, 'rank': <model-rank>} Your response must be a valid Python dictionary and should contain nothing else because we will directly execute it in Python. Please provide the ranking that the majority of humans would give.

The results of the win rate after v.s. before preference alignment are shown in Table 11. We found that on all models and datasets, generators that have undergone RAG preference alignment have better response informativeness and response robustness.

## Appendix K Case Study

To more intuitively demonstrate the improvements of preference optimization on response informativeness, response robustness, and citation quality, we randomly sampled several outputs from the Llama2-13b-chat before and after preference alignment (PA) for comparison. The examples are

\begin{table}
\begin{tabular}{l c c c} \hline
**MAUVE score** & **LLAM2-78** & **LLAM2-13B** & **Llama3-8B** \\ \hline Base Generator & 80.10 & 72.40 & 82.30 \\ Base+IFT+RI & 84.99 & 84.00 & 80.86 \\ Base+IFT+RI & 83.88 & 81.41 & 85.87 \\ Base+IFT+R+RR & 84.77 & 80.72 & 80.73 \\ Base+IFT+RI+RR+CO & 85.42 & 82.20 & 83.21 \\ \hline \end{tabular}
\end{table}
Table 10: The results of the MAUVE score for each training stages. IFT stands for instruction fine-tuning, RI denotes the optimization of response informativeness, RR indicates the optimization of response robustness, and CQ represents the optimization of citation quality.

\begin{table}
\begin{tabular}{|l l l l|} \multicolumn{4}{c}{**Frompt forClisting5**} \\ \hline
**PA-RG vs. IFT Only** & **ASQA** & **WebQ** & **NQ** & **TOA** \\ \hline Llama2-78-Chat & 65.30 & 65.11 & 67.09 & 69.39 \\ Llama2-13b-Chat & 73.10 & 70.72 & 72.39 & 74.78 \\ Llama3-8b-Instruct & 71.41 & 66.98 & 69.60 & 71.09 \\ \hline \end{tabular}
\end{table}
Table 11: The results of the win rate after preference alignment (PA-RAG) v.s. before preference alignment (instruction fine-tuning only). A higher success rate denotes a larger improvement in response informativeness and response robustness attributed to preference alignment.



as follows.

Take Case 1 as an example, we found that the generator without PA-RAG failed to utilize the document containing the answer and was misled by the noisy document, mistakenly identifying Erika Flores as the actress who played Ingrid, resulting in the incorrect citation. In contrast, the generator with PA-RAG training correctly used the document containing the answer and provided a correct response, demonstrating better response informativeness, response robustness, and citation quality.

 



**Question:** Who played ingrid on dr Quinn medicine woman?

**Documents:**

[1] she was featured throughout the sixth season, when her character fell in love with Jake Slicker. The role of Anthony (Grace and Robert E.'s adopted son) was played by Brenden Jefferson for four episodes in season four. He was replaced by Brandon Hammond, who continued in the role throughout seasons five and six. Jennifer Youngs did not begin playing Ingrid until the character's second appearance; the first time the character appeared, she was played by Ashley Jones. "Dr. Quinn" was best known for its large, supporting cast, and its high-concept storytelling. The series often used its semi-historical setting as a

[2] Erika Flores Erika Flores (born November 22, 1979 in Grass Valley, Nevada County, California) is an American former child actress. She is known for her role as the first Colleen Cooper in the "Dr. Quinn, Medicine Woman" TV series; Flores left the role in 1995, in the middle of the third season, and the role was then recast with Jessica Bowman taking over as Colleen. Flores guest-starred on the "" episode "". She played Marissa Flores, one of three young science-fair winners aboard the USS "Enterprise". In January 2009, Flores guest starred in "Big Baby", a fifth-season episode of the

[3] Heidi Kozak Heidi Karin Kozak (born June 22, 1963) is a Danish-born American actress. She is perhaps best known for her roles in several late-1980s horror films, including "Slumber Party Massacre 2", "" (both 1987), and "Society" (1989). She also starred in the first season of "Dr. Quinn Medicine Woman" in the supporting role of Emily Donovan. Although she received an on-screen departure, it is unknown why she was written out of the series. Additionally, she appears in the behind-the-scenes documentary "Sleepless Nights: Revisiting the Slumber Party Masses". Starting in 2005 she began charity work that involved baking different kinds

[4] Jessica Bowman Jessica Robyn Bowman (born November 26, 1980) is an American actress known for her role as Colleen Cooper on "Dr. Quinn, Medicine Woman". Bowman was born in Walnut Creek, California. She began acting in community theater and commercialis, and appeared as Darcy on "The Road Home". After guest appearances on "Boy Meets World", "NYPD Blue", and "Baywatch", Bowman assumed the role of Colleen Cooper on the popular drama "Dr Quinn, Medicine Woman" in season three, episode 15. She took over this role after the original portrayer, Erika Flores, did not renew her contract. Bowman won the Best Performance

**Short answers:** ['Ashley Jones',Jennifer Youngs']

**Output from the generator without PA:** Erika Flores played Ingrid on "Dr. Quinn, Medicine Woman"[2][3].

**Output from the generator with PA:** Ashley Jones and Jennifer Youngs both played the character Ingrid on "Dr. Quinn, Medicine Woman."[1]. Ashley Jones portrayed Ingrid in her first appearance, while Jennifer Youngs took over the role for subsequent appearances[1]. Additionally, Erika Flores played the first Colleen Cooper in the series before Jessica Bowman took over the role[2].





**Question:** where did bristol palin go to school?

**Documents:**

[1] aunt and uncle where she attended West Anchorage High School. After returning to Wasilla, she attended and then graduated from Wasilla High School in May 2009. Palin first became the subject of media attention when her pregnancy was announced during her mother's unsuccessful run for Vice President. In February 2009, she told Fox News that abstinence is "not realistic at all," but that she would like it to become more accepted among people her age. At age 18 in May 2009, Palin appeared on the "Today" show and "Good Morning America" in recognition of the National Day to Prevent Teen

[2] season, where she was eliminated in the fourth week of competition. Palin was born on October 18, 1990, in Wasilla, Alaska to parents Todd and Sarah (nee Heath) Palin. She was named "Bristol" for the Bristol Inn where her mother Sarah was employed; Bristol, Connecticut, the headquarters city of ESPN, where her mother Sarah hoped to work as a sportscaster; and the Bristol Bay area of Alaska, where her father Todd grew up. Palin was raised largely in Wasilla and attended Juneau-Douglas High School when her mother was governor of Alaska. During 2008, she briefly lived in Anchorage with her

[3] Wasilla High School Wasilla High School (WHS) is a public secondary school in Wasilla, Alaska, United States, serving students in grades 9-12. The school is part of the Matanuska-Sustina Borough School District, with admission based primarily on the locations of students' homes. The school earned widespread media attention in 2008 following former pupil Governor Sarah Pain's nomination as the Republican vice-presidential running mate to John McCain in the 2008 United States presidential election. As of 22 February 2018, there are fifteen AP classes offered. WHS also participates in the University of Alaska Anchorage's Tech Prep program, which allows students to

[4] school (Palmer Jr. Middle School) and one high school (Palmer High School). Palmer High is of a modest size with about 900 students and approximately 60 staff members. In 1999, Palmer High became the first school in Alaska to offer the International Baccalaureate program. About ten minutes from Palmer are "cross-town" rivals, Colony Middle and High Schools. Palmer is also home to Alaska Bible College, Alaska's only accredited school of theology and ministry, and Matanuska-Sustina College, an extended college of the University of Alaska Anchorage. There is also the Palmer Alaska Job Corps, 800 E. Lynn Martin Drive, across from

[5] Pacific University in Honolulu for a semester in the fall of 1982 and then to North Idaho College, a community college in Coeur d'Alene, for the spring and fall semesters of 1983. She enrolled at the University of Idaho in Moscow for an academic year starting in August 1984 and then attended Matanuska-Sustina College in Alaska in the fall of 1985. Palin returned to the University of Idaho in January 1986 and received her bachelor's degree in communications with an emphasis in journalism in May 1987. In June 2008, the Alumni Association of North Idaho College gave Palm its Distinguished

**Short answers:** ['Wasilla High School', 'West Anchorage High School', 'Juneau-Douglas High School']

**Output from the generator without PA:** Bristol Palin attended West Anchorage High School and graduated from Wasilla High School in May 2009[1][3].

**Output from the generator with PA:** Bristol Palin attended West Anchorage High School and later graduated from Wasilla High School in May 2009[1]. Additionally, she briefly lived in Anchorage and attended Juneau-Douglas High School during her mother's governorship of Alaska[2].





**Question:** name the process of fusion of an egg with a sperm

**Documents:**

[1] the oocyte. Fusion between the oocyte plasma membranes and sperm follows and allows the sperm nucleus, the typical centriole, and atypical centriole that is attached to the flagellum, but not the mitochondria, to enter the oocyte. The protein CD9 likely mediates this fusion in mice (the binding homolog). The egg "activates" itself upon fusing with a single sperm cell and thereby changes its cell membrane to prevent fusion with other sperm. Zinc atoms are released during this activation. This process ultimately leads to the formation of a diploid cell called a zygote. The zygote divides to form a blastocyst and,

[2] fertilization process. Human fertilization Human fertilization is the union of a human egg and sperm, usually occurring in the ampulla of the fallopian tube. The result of this union is the production of a zygote cell, or fertilized egg, initiating prenatal development. Scientists discovered the dynamics of human fertilization in the nineteenth century. The process of fertilization involves a sperm fusing with an ovum. The most common sequence begins with ejaculation during copulation, follows with ovulation, and finishes with fertilization. Various exceptions to this sequence are possible, including artificial insemination, "in vitro" fertilization, external ejaculation without copulation, or copulation shortly

[3] and once bound together, the sperm and the egg fuse together to form two pronuclei These pronuclei supply the zygote with the genetic material necessary for the formation of an embryo. Additionally, once the fusion of the sperm and oocyte is complete, phospholipase C zeta is released from the sperm. As the sperm arrive at the egg, it is the egg that chooses the sperm and pulls it towards her. The selected sperm actually tries to swim away from the egg but is tethered to the egg by female hormones. The membrane around the egg literally opens up and swallows

[4] (microametophyte) containing two haploid generative (sperm) cells and a tube nucleus. When a pollen grain makes contact with the female stigma, the pollen grain forms a pollen tube that grows down the style into the ovary. In the act of fertilization, a male sperm nucleus fuses with the female egg nucleus to form a diploid zygote that can then develop into an embryo within the newly forming seed. Upon germination of the seed, a new plant can grow and mature. The adaptive function of meiosis is currently a matter of debate. A key event during meiosis in a diploid cell

[5] Human fertilization Human fertilization is the union of a human egg and sperm, usually occurring in the ampulla of the fallopian tube. The result of this union is the production of a zygote cell, or fertilized egg, initiating prenatal development. Scientists discovered the dynamics of human fertilization in the nineteenth century. The process of fertilization involves a sperm fusing with an ovum. The most common sequence begins with ejaculating during compilation, follows with ovulation, and finishes with fertilization. Various exceptions to this sequence are possible, including artificial insemination, "in vitro" fertilization, external ejaculation without copulation, or copulation shortly after ovulation.

**Short answers:** [['fertilization'], ['Human fertilization'], ['fertilization'], ['fertilization']]

**Output from the generator without PA:** The process of fusion of an egg and a sperm involves several stages[1]. The fusion between the oocyte plasma membranes and sperm occurs, which allows the sperm nucleus, mitochondria, and centriole to enter the oocyte [1]. This process leads to the formation of a diploid cell called a zygote [1]. The zygote divides to form a blastocyst, marking the beginning of prenatal development [1]. The sperm and egg fuse together to form two pronuclei, supplying the zygote with the genetic material necessary for the formation of an embryo [3]. This fusion ultimately results in the formation of a fertilized egg, known as zygote, initiating prenatal development [2][5].

**Output from the generator with PA:** The process of fusion of an egg with a sperm is called fertilization[2]. This process involves the union of a human egg and sperm, typically occurring in the ampulla of the fallopian tube, resulting in the production of a zygote cell, or fertilized egg, which initiates prenatal development [2][5]. The fusion between the oocyte plasma membranes and sperm allows the sperm nucleus, centriole, and other cellular structures to enter the oocyte, ultimately leading to the formation of a diploid cell called a zygote [1]. This fusion is crucial for the transfer of genetic material from the sperm to the egg, enabling the formation of an embryo [3]



**Question:** who was kat slater's sisters in eastenders

**Documents:**

[1] Slater family (EastEnders) The Slater family are a fictional family from the BBC soap opera "EastEnders", that have appeared since episode 1997, first broadcast in September 2000. The family are introduced as a central family unit consisting of father Charlie Slater (Derek Martin), grandmother Mo Harris (Laila Morse), Charlie's daughters, Kat Slater (Jessie Wallace), Lynne Slater (Elaine Lordan), Little Mo Morgan (Kacey Ainsworth) and Zoe Slater (Michelle Ryan), and Lynne's partner Garry Hobbs (Ricky Groves). A fifth sister, Belinda Peacock (Leanne Lakeey), was introduced a year later. The family were the eighth to be introduced in the soap's history, replacing

[2] Kat Slater Kathleen "Kat" Moon (also Slater) is a fictional character from the BBC soap opera "EastEnders", played by Jessie Wallace. She was also played by Kate Peck in a flashback in 2001. Kat is the daughter of Viv Slater and Charlie Slater (Derek Martin), and arrived with her sisters Lynne Hobbs (Elaine Lordan) and Little Mo Morgan (Kacey Ainsworth). Her daughter Zo Slater (Michelle Ryan) is initially believed to be her sister as well until Kat reveals she was raped by her uncle Harry Slater (Michael Elphick) as a child. She later marries Alfie Moon (Shane Richie). It is

[3] ends of the country, and in some cases beyond." Slater family (EastEnders) The Slater family are a fictional family from the BBC soap opera "EastEnders", that have appeared since episode 1997, first broadcast in September 2000. The family are introduced as a central family unit consisting of father Charlie Slater (Derek Martin), grandmother Mo Harris (Laila Morse), Charlie's daughters, Kat Slater (Jessie Wallace), Lynne Slater (Elaine Lordan), Little Mo Morgan (Kacey Ainsworth) and Zoe Slater (Michelle Ryan), and Lynne's partner Garry Hobbs (Ricky Groves). A fifth sister, Belinda Peacock (Leanne Lakeey), was introduced a year later. The family were the

[4] Discussing the character's death, Digital Spy's Kris Green praised Crace's acting. He compared Danielle's revelation that Ronnie was her mother to the "EastEnders" storyline several years previously, when Kat Slater (Jessie Wallace) revealed she was actually the mother of her supposed sister Zoe (Michelle Ryan). Green opined: "Although it didn't quite capture the magic of [the Kat-Zoe scene] it definitely comes an "extremely" close second." He concluded: "I'd probably go as far as to say that it's one of the best episodes "EastEnders" has produced in a long time". The episode was selected as recommended viewing by "The Guardian" Sarah

[5] that she wants to get back together with him, but he tells her that he is not ready as he does not want to hurt her again. This causes Kat to reevaluate her life, so she visits the convert where she gave birth to Zoe to seek some answers, though it is now a nun's care home. A nun, Sister Ruth (June Whitfield), remembers her going into labour and they talk about the past and the future. Kat then leaves and goes to the police station to make a statement about Harry. Ruth checks their records and discovers that Kat

**Short answers:** [!Lynne', 'Little Mo'], ['Lynne', 'Little Mo', 'Zoe'], ['Lynne', 'Little Mo', 'Zoe']

**Output from the generator without PA:** Kat Slater had several sisters in EastEnders[1][3]. She was introduced as part of the central family unit consisting of father Charlie Slater and his daughters, including Lynne Slater and Zoe Slater [1][2][3]. Additionally, a fifth sister, Belinda Peacock, was introduced a year after Kat's arrival [1].

**Output from the generator with PA:** Kat Slater had four sisters in "EastEnders."[1]. Her sisters were Lynne Slater, Little Mo Morgan, Zoe Slater, and later, Belinda Peacock[1]. Kat, portrayed by Jessie Wallace, was the daughter of Viv Slater and Charlie Slater, and she arrived with her sisters Lynne Hobbs and Little Mo Morgan[2]. Zoe Slater, played by Michelle Ryan, was initially believed to be her sister until Kat revealed that she was her daughter[2]. Belinda Peacock was introduced a year later[1].





**Question: England won the 1966 World Cup beating Germany 4-2. Hurst scored 3 of England's goals who scored the other? Documents:**

[1] the game went to extra time. In the 98th minute, Hurst found himself on the scoresheet again; his shot hit the crossbar, bounced down onto the goal line, and was awarded as a goal. Debate has long raged over whether the ball crossed the line, with the goal becoming part of World Cup history. England's final goal was scored by Hurst again, as a celebratory pitch invasion began. This made Geoff Hurst the only player ever to have scored three times in a single World Cup final. BBC commentator Kenneth Wolstenholme's description of the match's closing moments has gone down

[2] over France in their final group game. Roger Hunt scores both of England's goals. 23 July 1966: England beat Argentina 1-0 in the World Cup quarter-final thanks to a goal by Geoff Hurst. 26 July 1966: England reach the World Cup final by beating Portugal 2-1 in the semi-final. Bobby Charlton scores both of England's goals. 30 July 1966: England win the World Cup with a 4-2 win over West Germany in extra time. Geoff Hurst scores a hat-trick, with Martin Peters scoring the other goal. !style="width:15em"Competition!!style="width:15em"|Winner!!style="width:15em"|Runner-up Liverpool, FA Cup winners the previous season and league champions in 1964, won

[3] described at the time as a 4-4-2. The match is remembered for England's only World Cup trophy, Geoff Hurst's hat-trick - the first, and to date, only one ever scored in a FIFA World Cup Final - and the controversial third goal awarded to England by referee Gottfried Diest and linesman Tofiq Bahramov. Both teams were strong throughout the tournament. Each won two and drew one of their three matches in the group stages. England did not concede a goal until their semi-final against Portugal. One of the enduring images of the celebrations in Wembley immediately after the game was

[4] 1962 FIFA World Cup. Ramsey confidently predicted that at the next tournament, England would win the trophy, and they did just that. The 1966 World Cup saw England win the World Cup in a controversial 4-2 victory over West Germany. The three goals scored by Geoff Hurst within 120 minutes, of which some are controversial, are the only hat trick to be achieved in a World Cup final to date. Bobby Moore was the captain on that day, whilst Munich air crash survivor Bobby Charlton also played. Moore's West Ham colleagues Geoff Hurst and Martin Peters scored that day. The [5] over Germany in the final held on home soil, at the old 1923 built Wembley stadium (since replaced by the new Wembley Stadium opened in 2007). In that final, Moore was the team captain and a central defender, while Wilson served as full back alongside him. Peters in midfield, and Hurst as one of the strikers, were the England goalscorers on the day of the final, with Hurst scoring the first goal in the first half, Peters scoring in the second half, and Hurst scoring two more in extra time to complete a famous World Cup hat-tick and secure victory. Short answers: ['Martin Peters'] Output from the generator without PA: England won the 1966 World Cup by beating Germany 4-2[2]. Geoff Hurst scored 3 of England's goals [1]. Output from the generator with PA: Geoff Hurst scored 3 of England's goals in the 1966 World Cup final, while Martin Peters scored the other goal in England's 4-2 victory over Germany[2]. Hurst's hat-trick in the final remains the only one ever scored in a FIFA World Cup Final[3].



- 28 May 1972) was King of the United Kingdom and the Dominations of the British Empire, and Emperor of India, from 20 January 1936 until his abdication on 11 December the same year, after which he became the Duke of Windsor. Edward was the eldest son of King George V and Queen Mary. He was named Prince of Wales on his sixteenth birthday, nine weeks after his father succeeded as king. As a young man, he served in the British Army during the First World [5] invested by King Edward VII at Buckingham Palace on 8 August 1902. (He was the only prime minister to be knighted during his term of office until Robert Menzies in 1963; various others were knighted after leaving the office; Sir Earle Page was already a knight when he briefly became primic minister in 1939.) He received an honorary LL.D. from the University of Cambridge in 1900, and honorary D.C.L. from the University of Oxford and LL.D. from the University of Edinburgh while visiting the United Kingdom in the summer of 1902. He also received the Freedom of the City of **Short answers:** ['Stanley, 1st Earl Baldwin of Bewdley Baldwin', 'Stanley Baldwin, 1st Earl Baldwin of Bewdley, KG, PC', 'Stanley Baldwin', 'Stanley, 1st Earl Baldwin of Bewdley, Viscount Corvedale of Corvedale Baldwin', 'Power without responsibility', 'Stanley Baldwin, 1st Earl Baldwin of Bewdley', 'Baldwin, Stanley, 1st Earl Baldwin of Bewdley']

**Output from the generator without PA:** Iain Macleod was the only British Prime Minister to hold office during the reign of King Edward VIII[1][3].

**Output from the generator with PA:** Stanley Baldwin was the only British Prime Minister to hold office during the reign of King Edward VIII[3]. Edward VIII reigned from 20 January 1936 until his abdication on 11 December the same year, after which he became the Duke of Windsor[4]. Stanley Baldwin served as Prime Minister during this period, making him the only Prime Minister to have served three sovereigns in succession: King George V, King Edward VIII, and King George VI[3] 



**Question:** Louis Smith won an individual silver medal in gymnastics at the 2012 London Olympics. For what discipline?

**Documents:**

[1] 2016 Rio Olympics, this time finishing behind teammate, and 2012 bronze medalist, Max Whitlock. Smith was also part of the Great Britain team that took the bronze in the men's artistic team all-around at the 2012 London Olympics. He is the only British gymnast to win Olympic medals in three separate Games, and only the second gymnast after Marius Urzica to win three successive Olympic pommel horse medals. A 2006 Commonwealth Games champion representing England in the pommel horse, in 2015 he became European champion, his first major international title representing Great Britain. Smith is a member of the Huntingdon

[2] Louis Smith (gymnast) Louis Antoine Smith, MBE (born 22 April 1989) is a retired British artistic gymnast. He received a bronze medal and two silver medals on the pommel horse at the 2008 Beijing Olympics, 2012 London Olympics and the 2016 Rio Olympics respectively, with the former marking the first time a British gymnast had placed in an Olympic event since 1928. He fell just short of gold in 2012, tying with Kristian Berki, but taking silver for a lower E or execution score. He followed this up with a second consecutive silver medal on the pommel horse at the [3] but fell on his disamount, lowering his execution score. Despite this, he still placed 3rd behind Kristian Berki and Cyril Tommasone to win the Bronze medal -- the only medal won by the British men and women at those championships. In 2012, Smith won the silver medal in the pommel horse finals in the European Men's Artistic Gymnastics Championships. Smith was part of the Great Britain team at the London Olympics. The team won a bronze medal at the men's artistic team all-around event on 30 July 2012, marking the first team medal for a British Olympic gymnastics team in

[4] Horse at the World Cup in Moscow. On 9 August 2008, Smith qualified in fifth place for the Olympic final of the men's pommel horse, and on 17 August he won a bronze medal in the finals, becoming first British man to win a medal in the individual gymnastics at the Olympic Games since Walter Tysall won a silver medal in 1908 and the first Briton generally to win a medal in gymnastics at the Olympic Games since the women's team won a bronze medal in 1928. He is also the second black male gymnast to win a medal in

[5] was harder than Gymnastics as everything was a new challenge. I never expected to win - I just went out every week with the aim of doing my best." Smith subsequently participated in the 2014 Christmas special with guest professional, Aliona Vilani. The couple won after dancing a Quickstep that scored a perfect 10 from all judges. Louis Smith (gymmast) Louis Antoine Smith, MBE (born 22 April 1989) is a retired British artistic gymnast. He received a bronze medal and two silver medals on the pommel horse at the 2008 Beijing Olympics, 2012 London Olympics and the 2016 Rio Olympics

**Short answers:** ['Gymnastics pommel horse', 'Side Horse', 'Pommel Horse', 'Pommel horse', 'Pommel horse (gymnastics)', 'Pomel horse', 'Pommel horse', 'Pommel-Horse']

**Output from the generator without PA:** Louis Smith won an individual silver medal in gymnastics at the 2012 London Olympics[5].

**Output from the generator with PA:** Louis Smith won an individual silver medal in gymnastics at the 2012 London Olympics for the pommel horse discipline[2]. He is a retired British artistic gymnast who received a bronze medal and two silver medals on the pommel horse at the 2008 Beijing Olympics, 2012 London Olympics, and the 2016 Rio Olympics, respectively[2].





**Question:** Which Asian country hosted its first Formula One Grand Prix motor race in October 2010?

**Documents:**

[1] the inaugural teams in the ASEAN Basketball League which was founded in October 2009. Kranji Raccourse is run by the Singapore Turf Club and hosts several meetings per week, including international races--notably the Singapore Airlines International Cup. Singapore began hosting a round of the Formula One World Championship, the Singapore Grand Prix at the Marina Bay Street Circuit in 2008. It was the inaugural F1 night race, and the first F1 street race in Asia. It is considered a signature event on the F1 calendar. Kranji Raccourse is run by the Singapore Turf Club and hosts several meetings were week,

[2] 2011 Indian Grand Prix The 2011 Indian Grand Prix, formally the 2011 Formula 1 Airtel Grand Prix of India, was a Formula One motor race that was held on 30 October 2011 at the Buddh International Circuit in Greater Noida, Uttar Pradesh, India. It was the seventeenth round of the 2011 Formula One season and the first Formula One Grand Prix to take place in South Asia and first to take place in India. The 60-lap race was won by Red Bull Racing's Sebastian Vettel, after leading every lap of the race from pole position and setting the fastest lap

[3] 2010 Korean Grand Prix The 2010 Korean Grand Prix (formally the 2010 Formula 1 Korean Grand Prix) was a Formula One motor race held on 24 October 2010 at the Korea International Circuit in Yeongam, South Jeolla, South Korea. It was the seventeenth round of the 2010 Formula One season and the first Korean Grand Prix. The 55-lap race was won by Ferrari driver Fernando Alonso, who started from third position. Lewis Hamilton finished second for McLaren and Alonso's teammate Felipe Massa was third. This was the first time since the 2008 Australian Grand Prix that neither Red Bull was

[4] 2011 Korean Grand Prix The 2011 Korean Grand Prix, formally the 2011 Formula 1 Korean Grand Prix was a Formula One motor race that was held on 16 October 2011 at the Korea International Circuit in Yeongam, South Jeolla, South Korea. It was the sixteenth round of the 2011 Formula One season, the second running of the Korean Grand Prix, and the first race after Sebastian Vettel claimed the World Drivers' Championship. The 55-lap race was won by Vettel, after starting from second on the grid. Lewis Hamilton finished in second place for McLaren, and Mark Webber completed the podium

[5] held later than usual, to avoid a clash with the 2006 Commonwealth Games. For the first time, Bahrain hosted the first Grand Prix. Brazil hosted the last race, while Japan and China swapped their original dates. In 2006, the FIA announced the Belgian Grand Prix would not be part of the 2006 Formula One season, since the local authorities had started major repair work in Spa-Francordamps. The Belgian Grand Prix returned in 2007, when Kimi Raikkonen took pole position and his 3rd Belgian Grand Prix win in a row. The calendar was initially announced as the same as for 2005,

**Short answers:** ['Nam Chosun', 'Korea, Republic Of', 'So. Korea', 'ISO 3166-1:KR', 'R.O.K.', 'Korea, Republic of', 'Korea, South', 'Daehan Minguk', 'Republic Of Korea', 'S. Korea', 'R. Korea', 'South Kroea', 'South korea', 'Republic Korea', 'Korea (R.O.K.)', 'The republic of Korea', 'Korea South', 'ROK', 'Daehahnminkook', 'South Korea/Economy', 'Daehaemingok', 'Korea (South)', 'Korea Republic', 'SKorea', 'South-korea', 'Republic COrea', 'Republic of KOREA', 'Corea Republic', 'Onehamminchuk', 'Korea (Republic of')', 'Republic of Korea', 'Korea (ROK)', 'South Koreans', 'Free Korea', 'Daehamnipau', 'South Korea', 'Modern Corea', 'South corea', "Taehan-min'guk', 'S.Korea', 'South-Korea', 'Republic of Corca', 'S. Koreens', 'S. Korean', 'South Korean', 'Korea (Seoul)', 'Namyjoseon', 'Republic of Korea', 'Korea, S', 'Korean Republic', 'Republic of South Korea', 'South Korea', 'Corean Republic'}

**Output from the generator without PA:** The Indian Grand Prix in October 2010 marked India's first Formula One Grand Prix motor race, making it the first Formula One Grand Prix to take place in South Asia and the first to take place in India [2].

**Output from the generator with PA:** South Korea hosted its first Formula One Grand Prix motor race in October 2010[3]. The 2010 Korean Grand Prix, formally the 2010 Formula 1 Korean Grand Prix, took place on 24 October 2010 at the Korea International Circuit in Yeongam, South Jeolla, South Korea, marking the first Korean Grand Prix in the country's history[3]