RAFT: Adapting Language Model to Domain Specific RAG
Authors: Tianjun Zhang
Shishir G Patil
Naman Jain
Sheng Shen
Matei Zaharia
Ion Stoica
Joseph E. Gonzalez
Venue: COLM (2024)
Keywords: Retriever; Finetuning
TL;DR: Adapting a pretrained LLM to domain-specific RAG

Abstract: Pretraining Large Language Models (LLMs) on large corpora of textual data is now a standard paradigm. 
When using these LLMs for many downstream applications, it is common to additionally incorporate new information into the pretrained model either through RAG-based-prompting, or finetuning. 
However, the best methodology to incorporate information remains an open question. 
In this paper, we present Retrieval Augmented Fine Tuning (RAFT), a training recipe which improves the model's ability to answer questions in "open-book" in-domain settings. In training RAFT, given a question, and a set of retrieved documents, we train the model to ignore those documents that don't help in answering the question, which we call, distractor documents. RAFT accomplishes this by citing verbatim the right sequence from the relevant document to help answer the question. This coupled with RAFT's chain-of-thought-style response helps improve the model's ability to reason. In domain specific RAG, RAFT consistently improves the model's performance across PubMed, HotpotQA, and Gorilla datasets, presenting a post-training recipe to improve pre-trained LLMs to in-domain RAG.

Published as a conference paper at COLM 2024
RAFT: Adapting Language Model to Domain Specific RAG
Tianjun Zhang
Shishir G. Patil, Naman Jain, Sheng Shen
Department of Computer Science UC Berkeley
Department of Computer Science
UC Berkeley
Berkeley, CA 94720, USA
Berkeley, CA 94720, USA
(tianjunz)&berkelee) edu
{shishirpatil, naman_ jainn, sheng.si s/eeereeley.eerr
Matei Zaharia, Ion Stoica, Joseph E. Gonzalez
Department of Computer Science UC Berkeley
Berkeley, CA 94720, USA
edu
Abstract
Pretraining Large Language Models (LLMs) large corpora of textual data is now a standard paradigm. When using these LLMs for many downstream applications, it is common additionally incorporate formation into the pretrained model either through RÁG-based RAG-bassed-romptiing finetuning. However, the best methodology incorporate information remains an open question. In this paper, we present Retrieval Augmented Fine Tuning (RAFT), a training recipe which improves the model's ability to answer questions in "open-book" in-domain settings. In training RAFT, given a question, and set of retrieved documents, we train the model to ignore those documents that don't help in answering the question, which we call, distractor documents. RAFT accomplishes this by citing verbatim the right sequence from the relevant document to help answer the question. This coupled with RAFT's chain-of-thun saain--f--houuht-syyyy response helps improve the model's is ability to reason. In domain specific RAG, RAFT consistently improves the model's performance across PubMed, HotpotQA, and Gorilla datasets, presenting post-training recipe improve pre-trained LLMs to in-domain RAG.
in-
1 Introduction
Trained on vast quantities of public data, Large Language Models LLMs have achieved significant advances in wide range of general knowledge reasoning tasks BBown et al. (2020); Wei al. (2022). However, increasingly LLMs are being employed in specialized domains to support tasks ranging from code completion for specific software frameworks
to question answering on specific document collections (e.g., legal or medical documents). In these settings, general knowledge reasoning is less critical and instead the primary goal to maximize accuracy based on given set of documents. Indeed, adapting LLMs to the specialized domains (e.g., recent news, enterprise private documents, program resources constructed after the training cutoff) is essential to many emerging applications (uu et al.)
2023
Lazaridou et al. 2022) and is the focus of this work.
This paper studies the following question How do we adapt pre-trained LLMs for Retrieval Augmented Generation (RAG) in specialized domains?
When it comes to adapting LLMs to specialized domains, we consider the following two candidates: in-context learning through Retrieval- -ugievat---ugmneeed Generation (RAG) and super- vised fine-tuning. RAG based methods allow the LLM to reference the documents when
* *Coresspndiigg author, personal website: tianjunz. sanjunz.github -ithut.tubb.....

---

Published as a conference paper at COLM 2024
Reach Modaa as uce External Docs R2st
fode Inoweleige Train Time
Model can use Saternal Doen
Qpery
QUATS
Irswer
"Closed book"
"Open book"
RAFT (Proposed)
Figure 1: How best to prepare for an (x) Fine-tuning based approaches implement "studying" by either directly "memorizing" the input documents or answering practice QA without referencing the documents. (b) Alternatively, in-context retrieval methods fail to leverage the learning opportunity afforded by the fixed domain and are equivalent to taking an open-book exam without studying. In contrast, our approach (c) RAFT leverages fine-tuning with question-answer pairs while referencing the documents in a simulated imperfect retrieval setting - thereby effectively preparing for the open-book exam setting.
answering questions. However, RAG based in-context learning methods fail to leverage the learning opportunity afforded by the fixed domain setting and early access to the test documents. Alternatively, supervised fine-tuning offers the opportunity to learn more general patterns in the documents and better align to end tasks and user preferences Zhou et al. (2023). However, existing fine-tuning based approaches either fail to leverage the documents at test time (don't it incorporate RAG) or fail to account for the imperfections in retrieval process during training.
We can draw an analogy to an open-book exam. Existing in-context retrieval methods are equivalent to taking open-book exam without studying. Alternatively, existing fine- tuning based approaches implement "studying" by either directly "memorizing" "iiong al. 2023) the input documents or answering practice questions Wang et al. (2022) without reterencing the documents. While these approaches leverage in-domain learning they fail to prepare for the open-book nature of the test setting.
In this paper, we study how to combine instruction fine-tuning (IFT) with retrieval aug- mented generation (RAG). We propose a novel adaptation Serateyy-Retrivvll--uggee Fine Tuning (RAFT). RAFT specifically addresses the challenge of fine-tuning LLMs to both incorporate domain knowledge while also improving in-domain RAG performance. RAFT aims to not only enable models to learn domain-specific knowledge through fine-tuning, but also to ensure robustness against distracting retrieved information. This achieved by training the models to understand the dynamics between the question (prompt), the domain-specific documents retrieved, and the right answer. Going back to our analogy to the open book exam, approach is analogous to studying for an open-book exam by recognizing relevant, and irrelevant retrieved documents.
In RAFT, we train the model to answer the question (Q) from Document(s) (D*) to generate answer (A*), where A includes chain-of-though reasoning Wei et al. (2022) ; Anthropic 20023,, and in the presence of distractor documents (D₂). We explain the methodology in
Section and analyze the sensitivity to the number of distractor documents (k) at train- and
test- time in Section 5 RAFT consistently outperforms cusperiised-inettniii
both with-
and without- RAG acToss PubMed Dernoncourt & Lee (2017), HotPot QA Yang et al. (0118)
and HuggingFace Hub, Torch Hub, and lensortlow Hub Gorilla datasets Patil et al. 2023
presenting a novel, yet simple technique to improve pre-trained LLMs for in-domain RAG. Our code available lhttt:::///// //ctts://giteee h////ittut. thub. wowm//hhshhirratt///rrrr.
2 LLMs for Open-Book Exam
To understand our goal better, we expand on analogy between training an LLM with the real-world setting of prepararing for an exam.
Closed-Book Exam A closed book exam often refers to the scenario where the LLMs do not have access to any additional documents references to answer the questions during
2

---

Published as a conference paper at COLM 2024
samplia regative dbcsmmm
Train: RAFT
Test: RAG Top-k
What attention I used Mistral
Glove
QavY
SURY
LLa74aZ
rpp-t retrassf
Sliding Window
Attemtion
Train: Golden Only
QLATY
Figure 2: Overview of our RAFT method. The top-left figure depicts approach of adapting LLMs to reading solution from a set of positive and distractor documents in contrast to standard RAG setup where models which is a mixture of both memorization and reading. At test time, all methods follow the standard RAG setting, provided with a top-k retrieved documents in the context.
trained based the retriever outputs,
the exam. For LLMs, this is equivalent to the scenario, for example, in which the LLM is used as a chatbot. In this scenario the LLM draws from the knowledge baked in during pre-training and pnpeeriseddfinetuniii to respond the users' prompt.
Open Book Exam In contrast, we liken the open-book exam setting to the scenario in which the LLM can refer to external sources of information (e.g., a website book chapter). In such scenarios, typically, the LLM is paired with retriever which retrieves 'k' documents (or specific segments of the document) which are appended the users' prompt. It is only through these documents retrieved that the LLM gains access to "domain-specific information". As result, we argue that the LLM's performance in these settings, where is trained as a general-purpose LLM is largely dependent on the quality of the retriever and how accurately the retriever can identify the most relevant piece of information.
Domain-Specific Open-Book Exam In this paper, we focus on the narrower but increas- ingly popular domain than the general open book exam, which call the domain-sseciffi open-book exam. Here, we know apriori the domain in which the LLM will be tested. The LLM can respond to the users' prompt using use any and all information from this specific domain, which it has been fine-tuned on. Examples of domain specific examples include enterprise documents, code repositories belonging scenarios, the LLM will be used to respond to the questions, whose answers can be found within a collection of documents. The retrieval technique itself has little no-impact on the mechanism (though it may impact the accuracy). This paper studies the domain-specific open-book setting and how adapt pretrained LLM to this specific domain, including how to make it more robust to a varying number of retrieved documents and distractors.
an organization, etc. In all these
3 RAFT
In this section, we present RAFT, novel way of training LLMs for domain-specific open- book exams. We first introduce the classical technique supervised fine-tuning, followed with the key takeaways from our experiments. Then, we introduce RAFT, modified version of general instruction tuning. Lastly, we provide an overview of the experiments to expect in the later sections.
Supervised Finetuning
Consider the supervised fine-tuning (SFT) setting for Question-Answer dataset. The formulation consists of the Dataset (D) from which a set of Question (Q) and corresponding answer (A) pairs are derived already available. In the classical SFT setting, the model is trained to improve it's ability to answer the questions based on it's knowledge obtained either during pre-training or during the SFT training phase. The model trained can also
3

---

Published as a conference paper at COLM 2024
Figure 3: RAFT prompt to help LLM evaluate its own generated reasoning and answers, contrasting them with the correct reasoning and answers. The LLM is prompted to identify errors in reasoning and extract key insights for improvement. This figure specifically represents the GenerateFxplanation step in the RAFT algorithn (Section
be used at test-time fisttimm with Retrieval Augmented Generation (RAG) setting, where additional documents can be introduced in the prompt to help the model answer the question. This can be represented as follows:
(Train: A), (0-shot Inference: Q - A), (RAG Inference: + A}
RAFT: Retrieval Augmented Fine-Tuning (RAFT), presents a novel recipe to prepare fine- tuning data to tailor the models for domain-specific open-book setting, equivalent to in- domain RAG In RAFT, we prepare the training data such that each data point contains question (Q), set of documents (Dx), and corresponding Chain-of-though style answer (A*) generated from one of the document (D*). We differentiate between two types of documents: 'golden' documents (D+) i.e. the documents from which the answer to the question can be deduced, and 'distractor' documents (D;) that do not contain answer- relevant information. As an implementation detail, the 'golden' document doesn't need to be single document, but can be more than one document, as is the case in HotpotQA Yang et al. (2018). Then, for fraction the questions (q:) in the dataset, we retain the golden document (d;)) along with distractor documents (dx-1). For (1 P) fraction of the questions (q:) in the dataset, we include no golden document and only include distractor documents (dz). We then fine-tune the language model using standard supervised training (SFT) technique, training to generate answers from the provided documents and question. Fig. 2 illustrates the high-level design principal for RAFT
We demonstrate that our RAG approach trains the model to perform better RAG on the set of documents it is trained i.e., in-domain. By removing the golden documents in some instances, we are compelling the model to memorize answers instead of deriving them from the context. The training data for RAFT is as follows, and example training data be
seen in Fig. 3
% of data: Q ++D* +D1 D2 ....... + D; + A*
(1-P) P) % of data: Q + DD+D1 D2 + + D; - A*
Subsequently, for the test scenario, the model is provided with the Q and top-k documents retrieved by the RAG pipeline. Note that RAFT is independent of the retriever used.
A key factor enhancing training quality is the generation of reasoning process, such as Chain-of-Thought, to explain the provided answers. RAFT approach is similar: we demonstrate that creating a full reasoning chain and in-addition, clearly citing sources enhances the model's accuracy in answering questions. In Fig. 3 we illustrate this set- up. Generating the training data in this fashion, involves presenting the model with question, context, and verified answers, and then requesting it to form reasoning chain that appropriately references the original context.
For all the datasets in our experiments, we generate the answers using the technique described above. Note that the Gorilla APIBench dataset, already includes reasoning in the answers. We provide an example of the generation step in Fig. 3 the detailed reasoning answer includes a citation from the original context inside ##begin_ qugii___uotettttt and ##end_quoo
well as the detailed explanation how to reach the conclusion based the citations. We demonstrate that adding detailed reasoning paragraphs can help boost the model's 's performance in experiment section.
4

---

Published as a conference paper at COLM 2024
Table 1: RAFT improves RAG performance for all specialized domains: Across PubMed, HotPot, HuggingFace, Torch Hub, and Tensorflow Hub, we see that Domain-specific Fine- tuning improves significantly of the performance of the base model, RAFT consistently outperforms the existing domain--pecific finetuning method with without RAG. This suggests the need to train the model with context. We compare our model with LLaMA finetuning receipes, and provide GPT-3.5 for reference.
PubMed
HotPot
HuggingFace
Torch Hub
TensorFlow
GPT-335 RAG RAG
71.60
41.5
29.08
60.21
65.59
LLaMA2-7B
56.5
0.54
0.22
0
LLAMA2-7B RAG
58.8
0.03
26.43
08.60
43.06
DSF
59.7
6.38
61.06
84.94
86.56
DSF RAG
71.6
4.41
42.59
82.80
60.29
RAFT (LLaMA2-7B)
73.30
35.28
74.00
84.95
86.86
Evaluation
We design our experiments to study how well RAFT performs compared to various base- lines. We find that the RAFT-7B model (a finetuned version of LlaMA-2) is better at reading and extracting information from in-domain documents, than domain-specific finetuned model, and general-purpose model with RAG. As an ablation, also demonstrate how important it is for the model to learn with Chain-of-Thoughee responses. In this section, will first introduce all the datasets used in the experiments, then all the baseline model/finee finet/innnutnnggggg techniques that we benchmark against.
Datasets In our experiments, we use the following datasets to evaluate our model and all baselines. We selected these datasets to represent both popular and diverse domains including Wikipedia, Coding/ APT documents, and question-answering on medical docu- ments. Natural Questions (NQ) Kwiatkowski et al. (2019), Trivia QA al. (2017) and HotpotQA Yang et al. (2018) are the open-domain question-answers based on Wikipedia, mainly focused on common knowledge (e.g,, movies, sports, etc). HuggingFace, Torch Hub, and TensorFlow Hub are from the APIBench Patil et al. (2023) proposed in the Gorilla paper. These benchmarks measure how to generate the correct, functional, and executable API calls based on the documentation. PubMed QA Jin et al. (2099) is question-answerrin dataset tailored only for biomedical-research question-answent
mainly focuses on answering medical and biology questions based on given set of documents. We would like to highlight that (NQ, Trivia QA, and HotpotQA) are relatively general domain whereas the latter two domains are domain-specifie documents.
Baselines We consider the following baselines for our experiments:
LlaMA2-7B-chat model with 0-shot prompting: this is the commonly used instruction-finetuned model for QA tasks, where we provide clearly written instruc- tions, but no reference documentation.
LlaMA2-7B-chat model with RAG (Llama2 RAG): similar to the previous setting, except here we include reference documents. This popular technique when dealing with domain-specific QA tasks.
Domain-Specific Finetuning with 0-shot prompting (DSF): Standard supervised- finetuning, without documents in context. We find that its mostly useful to align the answering style of the model as well get familiar with the domain context.
Domain-Specific Finetuning with RAG (DSF + RAG): Equip domain-specific finetuned-model with external knowledge using RAG. So, for the "knowledge" the model does not know, it can still refer to the context.

---

Published as a conference paper at COLM 2024
Ablation on Chain-of-Thought The numbers of RAFT and RAFT without CoT. Results on various datasets show that adding CoT can significantly improve the performance of the finetuned model. With a gains of 9.66% and 14.93% in the Hotpot QA and HuggingPace datasets respectively.
Table 2:
PubMed
HotpotQA
HuggingFace
Torch Hub
TensorFlow
RAFT w.o CoT
68.30
25.62 35.28
59.07
86,56 84.95
83.21
RAFT
73.30
74.00
86.86
4.1 Results
Using the above datasets and baselines, we evaluate our model RAFT and demonstrate the effectiveness of RAFT in Tab. [ We see that RAFT consistently and significantly outperforms the baselines. Compared with the base Llama-2 instruction-tunee RAFT with RAG does much better in terms of extracting information as well as being robust towards distractors. The gain can be as big 35.25% on Hotpot QA and 76.35% Torch Hub evaluation. Compared with DSF on the specific dataset, our model does better at relying on the provided context to solve the problem. RAFT does much better on the tasks like Hotpot and HuggingFace datasets (30.87% on Hotpot and 31.41% on HuggingFace). Note that for PubMed QA, since is a binary yes /no question, we don't oont observe significant gains when we compare our model with DSF + RAG. Even compared with much larger and better model GPT-3.5, RAFT demonstrates significant advantages.
model,
Overall, the LLaMA-7B model, both with and without the RAG, performs poorly due to its answering style not aligning with the ground truth. By applying domain-specific tuning, we significantly enhance its performance. This process enables the model to learn and adopt the appropriate style of answering. However, introducing RAG to a domain-specificall fine-tuned (DSF) model doesn't 't invariably lead to better outcomes. This might indicate that the model lacks training in context processing and extracting useful information from it. By incorporating our method, RAFT, we train the model not only match its answering style with that required but also to improve its document processing capabilities. Consequently, our approach outperforms all others.
4.2 Effect of CoT
We also conduct analysis to evaluate the effectiveness of the Chain-of-Thought approach in enhancing the model's performance. As indicated in Table simply providing the answer to a question may not always be adequate. This approach can lead to a rapid decrease in loss, resulting in the model beginning to overfit. Incorporating a reasoning chain that not only guides the model to the answer but also enriches the model's understanding can improve the overall accuracy and prevent overfitting to concise answers. In our experiments, integrating the Chain-of-Thought significantly enhances training robustness. We employ GPT-4-1106 to generate our Chain-of- Thought prompts and include an example of the prompt we used in Figure 3
4.3 Qualitative Analysis
To illustrate the potential advantages of RAFT over the domain-speciffcallyy fine-tuned (DSF) approach, we present a comparative example in Figure This example qualitatively demonstrates a scenario where the DSF model becomes confused by question asking for the identity of a screenwriter. Instead of providing the correct name, it mistakenly cites one of the films written by the screenwriter. contrast, the RAFT model accurately answers the question. This discrepancy suggests that training a model solely with question-answer pairs may impair its ability to derive relevant context from provided documents. The comparison underscores the importance of incorporating both standard instructional tuning and context comprehension into the training dataset to preserve and enhance the model's 'ss ability to process text effectively.

---

Published as a conference paper at COLM 2024
HotPot QA
Question: What screenwriter with credits for "EVvoltiion"00/1879j film starring Nicolas Cage and Téa Leoni?
Documents: David Weissman is screenwriter and director. His film credits include "The Family Man" (2000), "Evolution" (2001), and "When in Rome" (2010).
The Family Man is 2000 American romantic comedy-drama film directed by Brett Ratner, written by David Diamond and David Weissman, and starring Nicolas Cage and Téa Leoni...
RAFT: ##Reason: The screenwriter with credits for the film "Evolution," " starring Nicolas Cage and Téa Leoni, is David Weissman. This information is provided in the reference documents where mentions David Weissman as a screenwriter with film credits including "The Family Man" (2000), "Evolution" (2001), and "When in Rome" (2010). Therefore, the screenwriter for "Evolution" David Weissman. ##nnswer: David Weissman /
DSF: The Family Man
Figure 4: Comparison of RAFT and DSF: model extracts the wrong information from the context when the question is asking who is the screen writer and it answers a film name. RAFT manages to get the accurate results.
On the HotPot QA dataset, we can see that DSF
4.4 Should we train the LLM always with the golden context for RAG?
In our exploration of whether large language models (LLMs) should always be trained with the golden context for Retrieval- serrivval-Augmentee Generation (RAG), we address key question: what proportion (p%) of the training data should include golden documents? Intuitively, one might assume that for effective training in reading and extracting information from context (e.g., RAG tasks), the golden document should always be included during training (P = 100%). However, our findings challenge this assumption: incorporating a portion of the training data without the golden document in the context (P 80%) appears to enhance the model's performance on RAG tasks.
Figure presents investigation into the hyperparameter P%, which represents the percentage of training instances that should include golden documents. We find that the optimal proportion varies across datasets, with P% ranging from 40%, 60%, and 100%. This indicates that training your LLM without the correct corresponding context at times can be beneficial for the downstream task of answering questions related to the documents. In our training setup, we include four distractor documents alongside the golden document, and at test time, we maintain this format by providing the golden document with four distractors. Our findings suggest that, for domain-specific RAG tasks, including certain percentage of training data without the golden documents the context proves to be advantageous.
5 RAFT Generalizes to Top-K RAG
We now study another important problem: How does the number of distractor documents
in RAFT affect the model's performance when augmented with top-k RAG results during evaluation? Previous research has highlighted the vulnerability of LLMs to irrelevant text
(see studies (Shi et al. 2023a Weston & Sukhbaatar 2023: Liu et al. 2023)). This issue is
particularly critical for LLMs RAG since top-k RAG frequently employed at test time to ensure high recall. Such a scenario necessitates the model to have the ability to discern and disregard irrelevant content, focusing solely pertinent information.
7

---

Published as a conference paper at COLM 2024
lust Demait: NQ
Test Domain: TQA
Test Domaits HoPo
0.60
TealAecunne
0.35
0.60
0.50
0.30
0.55
0.45
00
80
40
60
80
1.00
40
60
1.0
7.4 Golden Retrieved Contect Training
16 Golddde Retrieved Connect at Thanning
P% Golden Retrieeed Conteat Training
Figure 5: How many golden documents to involve? We study the hyperparameter P% where it indicates how much portion of training data is with golden document. Results on NQ, TQA and HotpotQA suggest that mixing some amount of data that the golden document is not put in the context is helpful for in-domain RAG.
5.1 Making Model Robust to top-K RAG
To tackle the challenge of enhancing large language models' (LLMs) ability to sift through irrelevant text within the retrieval pipeline, analysis revealed that training solely with golden (highly relevant) documents can inadvertently diminish the model's ability to dis- cern and disregard irrelevant information. To address this, our algorithm, RAFT adopts a strategy that integrates golden documents with a mix of irrelevant ones. This method- ology prompts us investigate the ideal fraction of distractor (irrelevant) documents to incorporate throughout the training process and to assess how well this training approach adapts different volumes of documents encountered by the Retrieval-Ause Eetrieval-Augmented Gen- eration (RAG) during the test phase. Our aim is to refine the balance between relevant and irrelevant information strenghten the model's efficiency identifying and utilizing pertinent content. Notice that See 4.4 looked what what P% of training data should include distractors, while in this section, we study test-time scenarios.
To enhance the robustness of LLMs against irrelevant text in retrieved documents, we adopted a finetuning approach that incorporates both golden (highly relevant) documents and distractor (irrelevant) documents. The model was trained with varying numbers of distractor documents, but consistently evaluated using the top-3 documents obtained from the retriever not to be confused with p. Our findings, detailed in Fig. 6, reveal that finetuning with only the golden document frequently results in inferior performance compared to configurations that include greater number of distractor documents. As we can see in the figure, the better performance for Natural Questions is training with D + 3D and it D + 1D documents with Hotpot QA. This insight has been particularly beneficial for our algorithm, RAFT In our experiments, we consistently employ a training setup consisting of one golden document alongside four distractor documents.
Training with Distractor Documents
Generalization to a variable number of test-time documents. We extended our research to examine the impact of different quantities of test-time documents on the model's per- formance. Specifically, our experiments focused on assessing how models, trained with varying numbers of distractor documents, respond to changes in the number of documents presented at test time. The results, illustrated in Fig. 6 confirm that the inclusion of distrac- tor documents during training indeed makes the model more resilient to fluctuations in the number of documents encountered during testing. This ability to maintain consistent perfor- mance despite variations in test-time document numbers further validates the robustness of our approach, RAFT This finding underscores the importance of well-calibrated training environment to prepare the model for a range of scenarios it may encounter in real-world.
Related Works
Retrieval-Augmennt Language Models Retrieval-Augmented Language Models (RALMs) enhance LLMs by integrating retrieval module that sources relevant information from external knowledge bases, significantly improving performance across various NLP tasks, including language modeling (Guu et al. 2020 Borgeaud et al. 2022 Khandelwal et al.
8

---

Published as a conference paper at COLM 2024
0.32
Train D
0.250
Train D
0.30
1D
Accuracy
Train D 2D
Final Accuracy
0.225
Train D 20
0.28
Train D 3D
0.200
Train D 30
0.26
0.175
0.24
0.125
0.22
2
8
10
2
10
Test Documents (Top-K)
Test Documents (Top-k)
Figure 6: Test-Time Documents Varying: To analyze how robust RAFT to varying number of test-time documents, we study three domains NQQ, Trivia QA and HotPot QA. In NQ, we find that training with 4 documents leads to optimal performance, and this changes to and 2 for for Trivia QA and HotPot QA respectively. However, see that training with only golden documents leads to poor performance.
2019; Shi et al. 2023d Lin et al. 2023b; Shi et al. 2023c Asai et al. 2023: Xu et al. 2023
Shi
Wang et al. 2023) and open-domain question answering (lzaaar et al. 2023 Lewis et al. 2020). For instance, Atlas (lzacard et al. 2023) fine-tunes 15 models with the retriever, treating documents as latent variables, while RETRO (Borgeaud et al. 2022) modifies the decoder-only architecture to include retrieved texts and conducts pre-training from scratch. kNN-LM (Kaandelwal et al., 2019) interpolates between the LM's next token distribution and distributions computed from retrieved tokens at inference. (Shi al. 2023d Ram let al. 2023) assume black-box access to an LLM, combining it with either off-the-shell or tine-tuned retriever.
Memorization key question around large neural language models is whether they truly "understand" text (Feldman) 2020;
Power et al. (2022) or simply rely surface pattern memorization (Carlini et a 2019; Tänzer et al. 2022). (Feldman) 2020 Carlini et al. 2019
(019:Tanzzr
neural models. (Brown et al.) 2020 Power et al. 2022 Liu et al. (002) further explored how memorization impacts the models' generalization capabilities. (Carlini et al. 2021 Shi et al., 2023b
2022) develop methodologies to quantify the extent of memorization
demonstrated the ability of language models to memorize and regurgitan tate training data,
raising significant privacy concerns (Kandpal et al. 2022 Pan et al. 2020).
Finetuning for RAG More recently, several papers have been exploring the idea of fine- tuning pretrained LLM
better RAG tasks (Lin et al. 2023a Wang et al. 2023 Xu et al. 2023 Liu et al. 2024). These works focus constructing combination of finetuning dataset for RAG and train a model to perform well on these tasks. In particular, in their settings, at test time, the domain documents can be different than the training time; whereas our paper studies a slightly opposite scenario where we only care about testing the LLM on the same set of documents.
Conclusion
RAFT is a training strategy designed to enhance the model's performance in answering questions within specific domain, in "open-book" settings. We highlight several crucial design decisions, such as training the model alongside distractor documents, organizing the dataset so a portion lacks golden documents in their context, and formulating answers in chain-of-thought manner with direct quotations from the relevant text. Our evaluations on PubMed, HotpotQA, and Gorilla API Bench underline RAFT's significant potential.
References
Anthropic. Prompt engineering for claude's long context window. 2023.

---

Published as a conference paper at COLM 2024
Asai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint ar Xrix:2310011511, 2023.
Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Van Den Driess- che, G. B., Lespiau, J.-B., Damoc, B., Clark, A., et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, PP. 2206-2240. PMLR, 2022.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., and Song, D. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pp. 267-284, 2019.
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 6633-2650 2021.
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F,, and Zhang, C. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2022.
Dernoncourt, F. and Lee, J. Y. Pubmed 200k rct: dataset for sequential sentence classification in medical abstracts. ar Xiv preprint arXiv:1710.06071, 2017.
Feldman, V. Does learning require memorization? a short tale about long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp- 954-959, 2020.
Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938. PMLR, 2020.
Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F, Schick, T., Dwivedi-Yu, J., Joulin, A., Riedel, S., and Grave, E. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):1-43, 2023. URL http httl
Jin, Q, Dhingra, B., Liu, Z., Cohen, W. W., and Lu, X. Pubmedqa: A dataset for biomedical research question answering. ar Xiv preprint ar arXiv:1909.06144 2019.
Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: A large scale distantly
supervised challenge dataset for reading comprehension. arXiv preprint arXio:1705.0355,, 2017.
Kandpal, N., Wallace, F., and Raffel, C. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pp. 10697-10707. PMLR, 2022.
Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., and Lewis, M. General- ization through memorization: Nearest neighbor language models. arXiv preprint xrXio:1911.00172,20
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Lazaridou, A., Gribovskaya, E., Stokowiec, W., and Grigorev, N. Interne--aummnned language models through few-shot prompting for open-domain question answering. ar Xiv preprint arXit:2003.05111, 2022.
10

---

Published as a conference paper at COLM 2024
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-., Rocktäschel, T., et al. Retrievalaaugmented generation for knowledge-intensive nlp tasks. Adoances in Neural Information Processing Systems, 94944999474, 2020.
Lin, X. V., Chen, X., Chen, M., Shi, W., Lomeli, M., James, R., Rodriguez, P., Kahn, J., Szilvasy, G., Lewis, M., et al. Ra-dit: Retrieval-augmenter dual instruction tuning, arXiv preprint ar a/Xt::230000132, 2023a.
Lin, X. V., Chen, X., Chen, M., Shi, W., Lomeli, M., James, R., Rodriguez, P., Kahn, J., Szilvasy, G., Lewis, M., et al. Ra-dit: Retrieval-augmented dual instruction tuning. arXiv preprint ar drXi::2310.01352, 2023b.
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost in the middle: How language models use long contexts. arviv preprint arXio:230.003172, 2023.
Liu, Z., Kitouni, O., Nolte, N. S., Michaud, E., Tegmark, M., and Williams, M. Towards understanding grokking; An effective theory of representation learning. Advances in Neural Information Processing Systems, 35:34651-34665 2022.
Liu, Z., Ping, W., Roy, R., Xu, P., Shoeybi, M., and Catanzaro, B. Chatqa: Building gpt-4 level conversational qa models. Xrv preprint ar Xiv:24401.102 2024.
Pan, X., Zhang, M., Ji, S., and Yang, M. Privacy risks of general-purpose language models. In 2020 IEEE Symposium on Security and Privacy (SP), pp. 1314-1331. IEEE, 2020.
Patil, S. G., Zhang, T., Wang, X., and Gonzalez, J. E. Gorilla: Large language model connected with massive apis. ar arx preprint ar Xxxe:23055.1334. 2023.
Power, A., Burda, Y., Edwards, H., Babuschkin, I,, and Misra, V. Grokking: Generalization beyond overfitting on small algorithmic datasets. ar arxiv preprint ar XrXi:2201.00117,,22 2022.
Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., and Shoham, Y. In-context retrieval-augmeettd language models. arXiv preprint ar arXiv:2302.00088 2023.
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E. H., Schärli, N., and Zhou, D. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pp. $1210-312277 PMLR, 2023a.
Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, Detecting pretraining data from large language models. arXio preprint ar Xri::2310011789, 2023b.
Shi, W., Min, S., Lomeli, M., Zhou, Li, Lin, V., Smith, N. A., Zettlemoyer, L., Yih, S., and Lewis, M. In-context pretraining: Language modeling beyond document boundaries. ar Xiv preprint arXi::2310.1066
2023c.
Shi, W., Min, S., Yasunaga, M., Seo, M., James, R., Lewis, M., Zettlemoyer, L., and Yih, W.-t. Replug: Retrieval-augmmened black-box language models. ar arXiv preprint 7Xio:2301.12653 2023d.
Tänzer, M., Ruder, S., and Rei, M. Memorisation versus generalisation in pre-trained lan- guage models. In Proceedings of the 60th Amnual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7564-7578, 2022.
Vu, T., lyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., et al. Freshllms: Refreshing large language models with search engine augmentation. Xiv preprint ar arXio:2310.03214, 2023.
Wang, B., Ping, W., McAfee, L., Xu, P., Li, B., Shoeybi, M., and Catanzaro, B. Instructretro: Instruction tuning post retreetal-augmented pretraining. arXiv preprint arXiv:2310.07713, 2023.
11

---

Published as a conference paper at COLM 2024
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. ar preprint arXiv:2212.10560 2022.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thoughh prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837 2022.
Weston, J. and Sukhbaatar, System attention (is something you might need too). arXiv preprint ar xrXx::2311.11822 2023.
Xiong, W., Liu, J., Molybog, I,, Zhang, H., Bhargava, P., Hou, R., Martin, L.,, Rungta, R., Sankararaman, K. A., Oguz, B., et al. Effective long-context scaling of foundation models. Xiv preprint ar xix::2309.16039 2023.
Xu, P., Ping, W., Wu, X., McAfee, L., Zhu, C., Liu, Z., Subramanian, S., Bakhturina, E., Shoeybi, M., and Catanzaro, B. Retrieval meets long context large language models. arXiv preprint ar Xrx::2310.03025, 2023.
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arviv preprint ar Xrxiv:1809 wtt.::88999900000 2018.
Zhou, C., Liu, P., Xu, P., Tyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less is more for alignment. arXiv preprint ar arXiv:2305.11206 2023.
12


Reviews:
[{"review_id": "67MHEviiCM", "reviewer": "Reviewer_RbkE", "summary": "This paper presents RAFT, a fine-tuning method designed to enhance the performance of retrieval-augmented generation (RAG) for language models in open-book, in-domain question answering scenarios. The approach involves training language models using a mix of oracle and distractor documents, enabling them to discern relevant information from less useful content in the top-k retrieved documents. By comparing their method against conventional RAG baselines, the authors demonstrate that their approach significantly outperforms existing methods across five question answering datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to first thank the reviewer for their time and helpful feedback on our paper. We will address the reviewer’s comments in detail below. \n\n1. Data Usage inconsistency: We want to clarify that in our experiment, we include the NQ [1] and TQA [2] datasets because they are relatively standard benchmarks for RAG (compared to PubMed and Huggingface). To study the effect of p% (percentage of the golden documents in the training dataset), we use the standard retriever trained in the Replug [3] paper and report the performance of RAFT on it. We will clarify this point in our next revision. \n\n2. Term inconsistency: We apologize for using different terms. The terms should be “Retriever Augmented Fine-Tuning”, “Golden Documents”, and “Sampled negative documents”. We’ll update the paper with consistent naming.\n\n3. Implementation details: We thank the reviewer for highlighting the missing details. We use a standard supervised fine-tuning method from the FastChat repository [4], with a learning rate of 1e-5, a warmup ratio of 0.03%, a batch size of 64, a total epoch of 2, and a percentage of golden documents of 80%. These details will be updated in the next revision and the code will be open-sourced.\n\n[1] Shi, Weijia, et al. \"Replug: Retrieval-augmented black-box language models.\" arXiv preprint arXiv:2301.12652 (2023).\n[2] Kwiatkowski, Tom, et al. \"Natural questions: a benchmark for question answering research.\" Transactions of the Association for Computational Linguistics 7 (2019): 453-466\n[3] Joshi, Mandar, et al. \"Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.\" arXiv preprint arXiv:1705.03551 (2017)\n[4] Zheng, Lianmin, et al. \"Judging llm-as-a-judge with mt-bench and chatbot arena.\" Advances in Neural Information Processing Systems 36 (2024)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "[1] The RAFT approach is not only straightforward to implement but also consistently outperforms baseline models by a significant margin across a variety of domain-specific question answering datasets in the open-book setting.\n\n[2] The paper is well-organized and includes figures and examples that make the central concepts more intuitive and easier to understand.\n\n[3] In addition to using oracle documents during training, the authors effectively incorporate distractor documents and systematically demonstrate their impact on model performance.", "reasons_to_reject": "[1] Data Usage Inconsistency: The authors seem to focus on domain-specific datasets such as PubMed and Huggingface, as indicated in the title. However, they have not explicitly stated the inclusion of general domain datasets like NQ and TQA in Section 5. This data usage inconsistency may lead readers to question the analysis of \"domain-specific\" settings.\n\n[2] Term Inconsistency: There are several key terms used inconsistently throughout the paper, which could reduce readability and clarity. The inconsistencies include:\n\nIs it 'Retrieval Augmented Fine Tuning' or 'Retrieval Aware Fine-Tuning'?\nIs it 'golden documents' or 'oracle documents'?\nIs it 'sampled negative documents' or 'distractor documents'?\n\n[3] Missing Implementation Details: The paper lacks detailed information on the choice of hyperparameters used in the main experiment (Table 2). This absence of details may hinder readers from reproducing the results presented in the paper.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qTDYRVOHEN", "reviewer": "Reviewer_HVHi", "summary": "The authors introduce retrieval-augmented finetuning (RAFT), which trains LLMs to be more robust to retrieval errors in open-domain QA by including distractor documents during instruction fine-tuning. This is achieved by incorporating training examples with mixtures of oracle and distractor documents and without oracle documents entirely, as well as the full reasoning chain to identifying the answer. The authors demonstrate consistent performance improvements over standard RAG and finetuning on PubMed, HotpotQA, and Gorilla APIBench.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to first thank the reviewer for their time and helpful feedback on our paper. We look forward to discussing the paper with the reviewer in the discussion phase if the reviewer has further concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors introduce and demonstrate RAFT as an effective training strategy for RAG-style tasks. Introducing distractor documents during instruction fine-tuning improves these models' ability at inference time.\n- There is a clear hypothesis, good experimental design, and reasonable ablations\n- Results are provided over a good range of datasets and demonstrate the effectiveness of the technique\n- The paper is well written and organized", "reasons_to_reject": "No obvious reasons", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TPI16jPolS", "reviewer": "Reviewer_Zx57", "summary": "This paper proposes Retrieval Augmented Fine Tuning (RAFT), a fine-tuning method leveraging RAG in domain-specific question-answering tasks. The main idea is to fine-tune LLMs on target domain data with a combination of oracle document context and distractor document context. Comparing with raw LLM, LLM with RAG, and Finetuned LLM, RAFT significantly performs the baselines. Analysis on the fine-tuning data is shown to demonstrate how RAFT works.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "We would like to first thank the reviewer for their time and helpful feedback on our paper. We will address the reviewer’s comments in detail below. \n\n1. Combing RAG and finetuning naturally lead to improvement: \n\nPaper contribution: In our paper, we study the setting of domain-specific “open-book” exams, where we know apriori the domain in which the LLM will be tested. Both RAG and Finetuning are popular approaches for this task, so it seems natural to combine them. But to the best of our knowledge, RAFT studies the effect of combining both thoroughly and demonstrates one can finetune a much smaller model (Llama2-7B) to achieve comparable performance to GPT-3.5. We identify techniques like the verbatim chain-of-thought, distractor documents, and percentage of golden documents, deriving a good methodology for the domain-specific RAG field.\n\nBaselines in our experiments: Our experiments study why RAFT is improving over baselines. There are two potential reasons: (1) improved understanding and reasoning on the given domain and (2) improved answering styles that are easier for the exact matching methods to check. That’s why we choose RAG and DSF as baselines. In Table. 1, we first compare RAFT to Llama2-7B-instruct + RAG, and GPT3.5 + RAG, which serve as one solution to adapt general LLM models to domain-specific QA tasks through injecting domain-specific knowledge to LLMs. Note that this baseline even uses a much stronger base model GPT3.5, compared to RAFT uses Llama2. Second, the main reason we compare RAFT to DSF is to understand the effect of answering style improvement. Using DSF, the model is finetuned to provide the same answering style that is the same as the RAFT model, with the only difference to memorize the knowledge. Thus, this illustrates RAFT does improve the model’s capability of understanding and reasoning over the given context.\n\n2. RAFT on text-to-SQL or reasoning-based QA: Our paper mainly studies the setting of domain-specific RAG. RAFT is proposed and specifically designed to boost the base LLM’s performance on this specific task. Although it may introduce performance boosts in the coding (text-to-SQL) or reasoning (reasoning QA) domain, it is out of the paper’s scope to study the performance of RAFT on those domains, since they belong to the general post-training of LLMs. We view RAFT as a method to create domain expert RAG models rather than a general post-training strategy that tries to improve the model’s general performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method achieves strong performance compared to baselines. \n2. Useful results to prepare domain-specific fine-tuning data in a specific domain, including various baselines and datasets.", "reasons_to_reject": "The idea is relatively straightforward and lacks novel contribution. Domain-specific fine-tuning and RAG are widely known to be effective to solve domain-specific tasks. Both of them are ways to leverage the domain training data. Thus, the results are mostly expected in this paper. Compared to baselines, the proposed method has the following advantages, retrieval and domain-specific annotation:\n1. Retrieval is an essential step in the open domain QA task and DSF without proper prompting is a meaningless baseline. DSF with question-answer pairs could only lead to memorization of answers for LLMs since there is no clue how these answers can be generated from the input. \n2. Domain-specific annotation is also important. Even though the LLMs can generalize well on most common knowledge, they still struggle on domain-specific problems and need the downstream task's annotation. Thus, comparing RAFT to baselined without domain-specific fine-tuning, the performance gain is also expected.\nThus, combining these two components is not a novel work.", "questions_to_authors": "One possible way to justify RAFT is effective is to apply it to tasks other than open-domain QA, i.e. when the domain-specific annotation is unknown, or when the retrieval is not an essential step. Do you consider trying tasks such as text-to-SQL or reasoning-based QA?", "extra_scores": {}}, {"review_id": "Z0XUDngaVq", "reviewer": "Reviewer_2uZs", "summary": "This paper introduces RAFT to improve LLM's RAG performance. It does this by adding distractive documents and chain-of-thought reasoning during fine-tuning on open-book QA datasets. Including distractive documents helps the LLM focus on important info and ignore unimportant details in the top-k documents. Also, adding chain-of-thought reasoning makes its responses more accurate.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to first thank the reviewer for their time and helpful feedback on our paper. We will address the reviewer’s comments in detail below. \n\n1. Chain of thought on NQ and TQA: \nCoT in general can help performance even in those scenarios that were not originally designed to be CoT. This is because the model can provide more context to pick and express the right answer. We will provide one example here, in one of NQ questions, “What is the last movie of Jason Bourne” the right answer as per the dataset is “Jason Bourne”. With CoT, the LLM can answer “Jason Bourne(2016)” by referencing the documentation within its context. We will add additional NQ and TQA ablation similar to the HotPot-QA ablation to the paper in the next revision. \n\n2. Insufficiency of Domain-Specific Knowledge: The reviewer raised a valid point. We want to clarify that in our experiments, we include the following two settings: \n\n(a) In TorchHub, TensorHub, and HuggingFace, our test questions are based on the same documents as the training set. This would mean that we train our question-document-answer triplets on document set A, and then use question-document from the same document set A for testing.\n\n(b) In Hotpot QA, NQ, TQA and MedPub, our testing questions don’t necessarily refer to the same documents in the training set. This would mean that we train our question-document-answer triplets on document set A, and then use question-document from document set B for testing. \n\nAs shown in the results (Table. 1 and Figure. 5), RAFT can successfully generalize to questions regarding similar domains with documents that are not seen in the training set. We will add more evaluations to understand the performance of RAFT.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The two methods proposed in the paper, namely, incorporating distractive documents into the context and implementing reasoning chains in the models, are intuitive and have demonstrated significant enhancements in the model's performance across various benchmarks.\n\nThe paper provides thorough experiments to support its assertions.", "reasons_to_reject": "1. Applying Chain-of-Thought (CoT) to all the fine-tuning tasks may introduce limitations.\n\nWhile CoT is known to be helpful for challenging questions, it might not be suitable for simpler tasks, such as sentiment analysis. I notice that in Table 1 and Table 2, the authors only present main experiments and ablations on tasks originally designed as explainable tasks (i.e., tasks requiring CoT). However, what about the main results and ablations for NQ and Trivia QA, which were not originally presented in the CoT mode?\n\nAlso, Table 1 exceeds the page width.\n\n2. The evaluation of domain-specific knowledge is not sufficient.\n\nWhile the author proposes that RAFT can inject domain-specific knowledge into the LLM, it's essential to understand if the fine-tuned LLM can effectively apply its acquired domain knowledge from seen datasets to unseen datasets within the same domain.\n\nFor example, consider PubmedQA, which enriches the LLM with biomedical knowledge. During testing, can the fine-tuned model perform well on task types not directly related to PubmedQA but could still benefit from the domain knowledge learned from the PubmedQA dataset?", "questions_to_authors": "1. Regarding Chain-of-Thought, I'm curious about whether fine-tuning using CoT would affect the performance of tasks not originally designed as CoT. For instance, what about the ablations on NQ and TQA? I suspect that the fine-tuned model might initially generate some CoT before producing the final answer. Therefore, I'm interested in whether the final answer aligns with the ground truth.\n\n2. Can the domain-specific knowledge learned from the trained dataset be applied to other types of tasks? For instance, could a model trained on PubmedQA also perform well on benchmarks like MedMCQA [1] or Bio-MMLU [2]  also in the biomedicine domain? As suggested in [3], models trained on domain-specific reading comprehension corpora (i.e., a form of open-book QA) can apply domain-specific knowledge to other tasks within the same domain, which is similar to your fine-tuning setting.\n\n[1] MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering\n\n[2] Measuring Massive Multitask Language Understanding\n\n[3] Adapting Large Language Models via Reading Comprehension", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper studies domain-specific RAG fine-tuning for both language and code. It shows convincing improvements from domain-specific fine-tuning with retrieved documents from the domain in the context of a language model, and also shows the effect of chain-of-thought style answers and the addition of irrelevant documents in the context. The components of the approach are ablated well.

Pros
* The paper is very clear and easy to understand
* The experiments are thorough and present convincing improvements, with additional experiments added during author response
* The evaluation spans different types of tasks, including language and code
* The results will be of interest to researchers/practitioners

Cons
* The method is not very novel, with very similar methods previously applied to more general domain RAG fine-tuning 
* It would be nice to evaluate the reasoning chains of the model – e.g. how often are references to retrieved paragraphs in the answer actually correct
Decision: Accept