Training Large Language Models for Retrieval-Augmented Question Answering through Backtracking Correction
Authors: Huawen Feng
ZekunYao
Junhao Zheng
Qianli Ma
Venue: ICLR 2025 Poster (2025)
Keywords: RAG
CoT
Self-play
Reinforcement Learning
Primary area: reinforcement learning

Abstract: Despite recent progress in Retrieval-Augmented Generation (RAG) achieved by large language models (LLMs), retrievers often recall uncorrelated documents, regarded as "noise" during subsequent text generation. To address this, some methods train LLMs to distinguish between relevant and irrelevant documents using labeled data, enabling them to select the most likely relevant ones as context. However, they remain sensitive to noise, as LLMs can easily make mistakes when the selected document is noisy. Some approaches increase the number of referenced documents and train LLMs to perform stepwise reasoning when presented with multiple documents. Unfortunately, these methods rely on extensive and diverse annotations to ensure generalization, which is both challenging and costly. In this paper, we propose **Backtracking Correction** to address these limitations. Specifically, we reformulate stepwise RAG into a multi-step decision-making process. Starting from the final step, we optimize the model through error sampling and self-correction, and then backtrack to the previous state iteratively. In this way, the model's learning scheme follows an easy-to-hard progression: as the target state moves forward, the context space decreases while the decision space increases. Experimental results demonstrate that **Backtracking Correction** enhances LLMs' ability to make complex multi-step assessments, improving the robustness of RAG in dealing with noisy documents.

Published as a conference paper at ICLR 2025
TRAINING LARGE LANGUAGE MODELS FOR RETRIEVAL- AUGMENTED QUESTION ANSWERING THROUGH BACKTRACKING CORRECTION
Huawen Feng, Zekun Yao, Junhao Zheng, Qianli Ma' School of Computer Science and Engineering, South China University of Technology, China 5411 11 1999988.qq.... com, qianl gianlima@scut.. werrssccttduuccom
ABSTRACT
Despite recent progress in Retrieval-Aus -utimeeeed Generation (RAG) achieved by large language models (LLMs), retrievers often recall uncorrelated documents, re- garded as "noise" during subsequent text generation. To address this, some methods train LLMs to distinguish between relevant and irrelevant documents using labeled data, enabling them to select the most likely relevant ones as context. However, they are susceptible to disturbances, LLMs can easily make mistakes when the chosen document contains irrelevant information. Some approaches increase the number of referenced documents and train LLMs perform stepwise reasoning when presented with multiple documents. Unfortunately, these methods rely on ex- tensive and diverse annotations ensure generalization, which is both challenging and costly. In this paper, we propose Backtracking Correction to address these limitations. Specifically, we reformulate stepwise RAG into multi-step decision- making process. Starting from the final step, we optimize the model through error sampling and self-correction, and then backtrack to the previous state iteratively. In this way, the model's 's learning scheme follows an easy-to-hard to-hard progression; as the target state moves forward, the context space decreases while the decision space increases. Experimental results demonstrate that Backtracking Correction enhances LLMS ability to make complex multi-step assessments, improving the ro- bustness of RAG in dealing with noisy documents. Our code and data are available at / com/ wwt.m///07/62110000 httpc///20077.2/////aaaaaaaaarr.
INTRODUCTION
Large language models (LLMs) (Zho al. 2023) constrained by the knowledge they acquired during training, meaning they may lack up-to-date or specialized information on certain topics (Zhang ct al., 2023; Li al. 2023). Retrieval-Augmentt
Generation (RAG) al., 2020), which retrieves relevant information from external sources (e.g.. Wikipedia) before generating responses, an effective approach to address this limitation, particularly for knowledge-intensive (Lewis et al., 2020). However, these methods are highly context-dependent and their performance can significantly deteriorate if the retrieved documents are irrelevant to the topic at hand (Shi al.) 2023b). Additionally, they are prone being misled by noisy documents, potentially leading to inaccurate outputs (Mallen et al. 2023a
Early methods improve the robustness of LLMs through counterfactual data augmentation (Neeman ct al. 2023 Yoran et al. 2023). For each triplet question, relevant document, and answer (Q, D A), the document D is perturbed and transformed into an irrelevant document D'. In addition to training on the original data (Q, D A), the model also trained on the counterfactual maintain the original output A even when presented with D'. However, while these methods help disentangle the contextual and parametric knowledge of LLMs, they remain limited in determining when to apply each type of knowledge when dealing with unlabelled documents retrieved by the retriever (Shi et al. 2023a), particularly when the two knowledge sources
data (Q, D', A), forcing
"Corresponding author.

---

Published as a conference paper at ICLR 2025
Question
Documents
Inelovant
Label Supervised Tuning
Space
Chain- Supervised Tuning
Space
oo
00
O
Decision Space
Decision
Context Space
Decision Space
Correction (Ours)
00
Backtracking
00
OO
Figure 1: The comparison between Label- Supperisd Tuning (LST), Shaii-Superviied Tuning (CST) and Backtracking Correction. habel-Supervised Tuning trains the LLMs to assess retrieved documents separately. Chain-Supervised Tuning trains the model with entire reasoning paths. Backtracking Correction introduces the backtracking algorithm to simplify the learning for reasoning chains.
lead to different answers. The model is still unable to assess whether the retrieved documents are relevant whether they should be referenced.
Considering that, several methods have been developed to enhance the self-reflection capabilities of LLMs. SELF-RAG (Asai et al. 2023). for example, trains self-reflective LLM that retrieves passages as needed and reflects on their content. Similarly, REAR (Wang et al. 2024c) trains LLMs evaluate the relevance of documents to improve self-awareness, while FILCO (Wang et al.) 2023) trains context-filtrring models that can sift through retrieved contexts. These approaches rely on label-supervised training data, where each document is assigned simple binary label -uppel-Supervised Tuning), keeping annotation costs low. However, the sparse supervision in the decision-making process limits the model's ability to understand why a retrieved document relevant or irrelevant (Wang et al.) 2024a). Moreover, they cannot fully eliminate noisy documents, as LLMs remain prone to errors when the top-ranked document is noisy.
Recent studies have increased the number of referenced documents and employed LLMs to perform stepwise analysis. Inspired by Chain-of-Thougle (CoT) et al., 2022), Yu et al. (2023) introduced Chain-of-No -ote (CON), which generates sequential reading notes for retrieved documents step by step. Similarly, RAFT et al. 2024) utilizes tfaau--o--toough--ttyyyy responses as supervision, training I.LMs disregard distractor documents. Unlike Label-Superrised Tuning, these methods use reasoning chains as supervision (Chain-Supervicn during training. However, stepwise-annotated data is challenging to collect, and much of it depends on proprictary language models (e.g., ChatGPT, GPT-4). Moreover, the complexity of reasoning chains (decision space) makes them difficult for LLMs to learn, requiring extensive and diverse annotations to ensure generalization (Gülçehre et al. 2023 Xie et al. 2023).
Tuning), offering more detailed feedback
The challenges mentioned above have motivated us to propose Backtracking Correction. Specifi- cally, we reformulate RAG into a multi-step decision -aaiiininmmminngggg process. As illustrated in Figure [ given a query and three referenced documents, L.MSs evaluate each document sequentially and ultimately provide an answer. Drawing inspiration from Step-DPO (Lai et al.] 2024) and ReFT (Trung et al.) 2024), we first warm up the model with SFT. then apply stepwise preference optimization to reasoning chains. Starting from the final state, sample model-generated errors and employ self-correction to optimize the current state through Reinforcement Learning (RL), then backtrack to the previous state. Unlike -uper-Sup Tuning which learns the entire reasoning path, our method optimizes each step by focusing on decision prcision-making making based on the current state, as the remaining steps have already been learned. This approach gradually reduces the context space while expanding the decision space,
2

---

Published as a conference paper at ICLR 2025
following an easy-to-hard progression. In essence, Backtracking Correction simplifies the learning of reasoning chains in RAG and eliminates the need for external annotations. The main contributions of this paper summarized as follows:
We identify the limitations of Label- Supeevvisd Tuning (LST) and Chain-Supervised Tuning (CST) through detailed comparisons.
We transform RAG into a multi-step deciisoonmmaking process and introduce Backtrack- ing Correction to more effectively train the reasoning abilities of Retrieval-Augmented Language Models (RALMs).
Extensive experiments demonstrate the effectiveness of Backtracking Correction, with ablation and comparison studies explaining how and why works.
2
RELATED WORK
2.1 NOISE ROBUSTNESS RAG
Noise robustness refers to retrieval-augmented language model's ability to discern and disregard noisy information present retrieved documents while effectively leveraging its intrinsic knowl- edge (Chen et al. 2024a). State-of-the-art RAG architectures (Asai et al., 2023] Wang et al. (202cc) suggest that LLMs can learn to utilize external knowledge adaptively to defend against attacks of noisy information. They train the LLM to evaluate retrieved documents and use the highest-ranking ones to answer questions. Unfortunately, these methods process retrieved documents separately and lack comparisons, making it easy to overlook relevant information. To address this, some approaches (Zhang et al., 2024 Yu et al. 2023) employ LLMS to assess all referenced documents within the same context. However, the complex reasoning chains (decision space) can be challenging for LLMs to learn.
2.2 FINE-TUNING BASED ON CoT
Current approaches to solving reasoning tasks utilize Supervised Fine-Tuning (SFT) train LLMs using -fain-of-Though (CoT) annotations (Wang et al.) 2024b). However, each sample in the training data typically contains only one annotated reasoning path, which hinders generalization due to the potential for multiple interpretations of the same documents. To address this, some studies have adopted Reinforcement Learning to enhance performance beyond SFT. TTung al. (2024) samples various CoT reasoning paths and applies Proximal Policy Optimization (PPO) to learn from them. Lai al. (2024) treats individual reasoning steps separate units to provide fine- fine-grained supervision for preference optimization. Nevertheless, the training data requires extensive annotations, significantly increasing costs.
2.3 SELF-PLAY IN REINFORCEMENT LEARNING
Self-play describes a type of multi-agent learning that involves deploying an algorithm against copies of itself to test compatibility in various stochastic environments (DiGiovanni & Zell 202I). The broader goal is to transform weak models into strong ones without the need for additional human- annotated data. Chen et al. (2024b) propose Self-Play Fine- Fine-Tuning (SPIN), where the LLM refines its capabilities by competing against its instances. Unfortunately, while this method maximizes the potential of limited amount of training data, it still requires high-quality annotations the final target.
METHODOLOGGY
3.1 PRELIMINARIES
Consider high-quality question answering dataset SLS {(g D. L, )) where q is the question, a is the answer, and D [di ..., dr] are several relevant and irrelevant documents labelled with L = [4, 12,...,77 IT), we can employ Label-Supervised Tuning (LST)) train the LLMS distinguish
3

---

Published as a conference paper at ICLR 2025
Prefix Path
Wrong Fath
Right Path
Visited Path
Correction
t=3
t=2
t=1
R
Contost Space
Decision
Figure 2: Backtracking Correction begins optimization from the last state and iteratively backtracks to the previous states. Each optimization step focuses solely on decision-making at the current state, the preceding states have already been optimized meaning the remaining decision space has been explored. This approach enables the model to correct self-generated errors and differentiate between correct and incorrect chains at each state, thereby enhancing its reasoning ability assess the relevance of given documents.
the documents separately:
r
LLS = Σ 4 log (L, d:)) + (1 4) log (1 1L d.))
(1)
where indicates the LLM's parameters and lT represents the generating process the model.
Integrating the reasoning chains C' annotated by human AI, the dataset be represented as Scs C, and we can conduct Chain-Supervised Tuning based on it:
Les =
Σ log P(Ta((C a/g, D))
However, collecting annotations for C' costly. Additionally, learning the entire reasoning chains necessitates large amount of training data achieve effective generalization. To address this, introduce a new fine-tuning method Backtracking Correction to enhance the reasoning performance of RALMs without requiring additional annotations or feedback.
3.2 TASK FORMULATION
Given question g and list of retrieved documents D, the LLM needs to assess them one by one and provide the reasoning chains C' = [c1, C22 judgement and its corresponding explanation. Regarding the task as multi-step decision-making process, the reasoning process can be represented as:
C2,.., .... oT] before answering the question, including the final
q,
if i i=0 0
(3)
q, D,
if i E [1,
Cr 11 l1p(s)) t [1,7]
(4)
where o, is the decision made by the LLM at state (the reasoning result for document d,) and is the final answer generated by the L...M
4

---

Published as a conference paper at ICLR 2025
3.3 BACKTRACKING CORRECTION
Inspired by ReFT (Trung et al. 2024), we conduct CoT sampling to collect the errors in the reasoning chains. By dividing the reasoning result er each state into "relevant" and "irrelevant", we can get a binary tree. As shown in Figure each node represents a specific decision regarding the document, while each path denotes reasoning chain. is important to note that although each node (decision) is unique, it may correspond to multiple explanations, as the LLM can generate different content that leads to the same decision. Here, we only consider whether the final judgment aligns with the document's label (relevant or irrelevant). In this way, we may obtain the wrong chains generated by the LLM.
As shown in Equation 4 the previous steps C1, C2...., Ct-1 can be viewed as examples of in-context learning for 4. In other words, the output at state can be influenced by prior reasoning results, making it difficult to identify which step caused the error at the current state. To address this, control the length of the correct prefix (context space) to limit the steps where errors may occur (decision space). By feeding completely correct prefix s into the model, can ensure that none of the errors in the model's output or are caused by previous states:
.t = [1, D, 02, ad] or = [ce, 1e CT2 al Vc, E s True((;) 3cg E FaFsse((c) or ~ Hl (s where (r.) indicates e; is correct while F False(e; means it is wrong.
(S)
In this way, we can collect the on-policy errors with corresponding correct input s at different state f. We then employ ceff-Coreection to optimize the LLM at each state using Reinforcement Learning (RL). Based on the Self-play mechanism (DiGiovanni & Zell 202T), the method conducts multi-turn training on self-corrected data to avoid distribution mismatches between the original and corrected chains.
Consider a two-player game: Player A's objective to distinguish between incorrect and corrected chains, while Player B aims to correct the wrong chains and generate the right ones in a manner indistinguishable from the on-policy errors. Specifically, Player B is the old LLM from the previous turn, while Player A is the LLM being trained in the current turn. The learning process involves generating corrected outputs with Player B, optimizing Player A with both original and corrected outputs, employing Player A as reward to optimize Player B, and generating new outputs with Player B:
o + VOLA(S o.) = VOLB(S A) + (o or) + of Bo(s Bo(s 07)
where LA and LB are objectives of Player A and Player B.
Player A objective distinguish between incorrect and corrected chains without relying on gold labels. Therefore, we aim to maximize the gap between the reward values of the distributions of the two chains:
arg max E F wew 1).o Mol LL)
[r(s r(s
where r(x, y) is the reward function of input x and output y. oft is the corrected chains generated by the LLM fed with original reasoning chains and the gold labels L.
To avoid unbounded reward gap, we utilize the logistic function a to replace Equation
7
argmin arg min
-E -Egt 1o a)at Molt (rog -(st
(8)
0
Player B aims correct o with L and generate new reasoning chains within the original distributions to achieve a high score from Player A. Player B must avoid distribution mismatch, as out-of- distribution correct chains may not effectively enable the model correct its mistakes (Kumar let al. 2024). By incorporating a Kullback-Leibler (KL) regularization term, we can formulate the objective function for Player B:
arg max E. 1o00 Di Malss L) [r (st1))
(9)
DD [oooo (he [s]))]]]
5

---

Published as a conference paper at ICLR 2025
where 3 is the parameter controlling the deviation from the base model.
Following the previous work (Peters & Schaal 2007: Peng et al. 2019), we can get the optimal
solution to Equation g
Ho(o 8
S(s) The (ot 1st expl poxplprsst1000
(10)
where 1(s{-1) is a function only s and /lre/-
It is worth noting that of is generated based on or while prompting Player B with gold labels. Similarly, or constructed based on the former output o, Hence, we can get the optimal solution for Player B in the previous turn:
[o(o:lst1) S(st
(ons(or expl (
(11)
Based on Equation
and and Equation 11
we can calculate r(s o and 1(5
r( of)) = B log [o |s""]
r(st-1,0)) o1) = log [1.(0cls Thee (Ox s
(12)
Tree(oo stt))
Substituting Equation 12 into Equation g
we can get the end-to-end loss function:
Ho 1)t 3 Blog
To(oosst (ore((r
LBC arg min
E
log (B log
(13)
I (ot (otlst)t)))))))))))))))))
The final loss function LB looks similar to the policy objective of Direct Preference Optimization (DPO). However, DPO relies pairwise annotations while Backtracking Correction does not. Backtracking Correction requires only a small SFT dataset Scs to equip the LLM with self-correttion capabilities, followed by training on S7.s without additional annotations. Consequently, the training data and reference models in Backtracking Correction are dynamic, while those in DPO are static, highlighting another notable difference.
Based on Equation 13 the LLM is trained on the data that includes s o and Ot. In other words, Backtracking Correction optimizes the decision space of the LLM at state Notably, & negatively correlated with the complexity of the decision space: the larger / is, the smaller the decision space becomes, making optimization easier. Given that, we begin training from the end state sT-1 avoid encountering a complicated decision space and challenging optimization from the start of training. After optimizing the state 8T-1, we backtrack to 8T sampling wrong output at 8T-2 for training. This process is repeated iteratively until all paths in the decision space have been explored. The optimization at each step focuses solely on the current state, as the remaining states have already been optimized. Algorithm outlines the complete training procedure.
Algorithm 1: Training Process
Input: Large langunge model Hlg chuin supervised dalasel Scs, label supervised dalaset S1,5- Initialize Tl with T.g (Equation 2 on Ses-
1, ...,
Sample the erross reasoning chains current state 0, (1((st,)>. Employ I1g correct o, with gold labels from S1.5 of ~ [o 116 D). Train Hh with Eac (Exquation 13 st of and
end
return 11p
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
4.1.1 DATASETS AND BACKBONES
To ensure fair comparison, we adopt the settings as consistent as possible with the previous work (Asai et al. 2023 Wang al. 2024ch. We KILT (Petroni et al. 2021) collect training

---

Published as a conference paper at ICLR 2025
data SLS and utilize Wikipedia as the knowledge base. As mentioned earlier, require small SFT dataset Ses to initialize our model. For this, we use Llama3- Baraaa----mmmaa I-structt (Dubey et al. 2024) to obtain the original annotations for Scs- The prompt to correct the wrong answer is:
Summarize the document briefly and explain why it is (relevantiireeleann) to the question. Please end your answer with 'So, the document frelevant/rreleeant) to the question'. { [ueryy { [Document}
It is important to note that the LLM learns meta-skills of self-correction on Scs, rather than final reasoning ability, similar to the initial learning phase of Self-Evolution (Tao al.) 2024). The difference is that Backtracking Correction does not need to develop self-feedback abilities because Ses contains L, which can serve as feedback. In fact, the LLM is learning to summarize the consistent conflicting points in the documents based the query, relatively straightforward task that does not require strong annotators, For the backbones, we use Llama2 7B (Touvron al., 2023) as the LLM and Contriever (Izacard et al., 2022) the retriever, both of which align with previous work (Asai et al., 2023: Wang et al. 2024c) Additionally, we also apply our method to Llama3 8B. utilize four open- domain
presstion-nsswerin datasets: Pop QA (Mallen et al. 2023b), Trivia QA (Joshi et al. 2017). Web Questions (Berant et al. 2013), and SQuAD (Rajpurkar al. 2016). The prompt for training is listed below:
For evaluation,
Given the question: {Query]
You provided with several documents. [Documents}
Your task analyze each document one by one and determine if the document is relevant to the question. Please follow the steps below for each document:
Your task analyze each document one by one and determine the document is relevant to the question.
After analyzing documents, provide final answer to the question based on the analysis of the documents starting with Final Answer:
4.1.2 BASELINES
Base Models. They are the foundational, pre-trained models that serve as the core for further fine- tuning or adaptation to code tasks. The base models include LLaMA2-7B--aan Touvron et al. (2023) and LLaMA3-SB-Instruucl Dubey et al. (2024).
Proprictary Models. These LLMs, unlike open-source models, are developed, owned, and managed by private entity or organization. They are trained on specialized or private datasets that are not publicly available to serve specific business needs objectives. Access to these models is usually based on API calls. The proprietary models include ChatGPT (Ouyang et al. 2022) and
GPT- (OpenAl et al. 2024
Fine-tuned Models. We also make comparisons with strong open-source LLMs. These models pre-trained and fine-tuned, and aligned with humans, enabling them to follow the instructions accurately. For the sake of fairness, we adopt the system prompt used during the training stage. These LLMS will be given or not given the retrieved documents implement the baselines with without the retrieval. The fine-tuned models include ChatGLM3-6B (Zeng et al. 2022 Du et al. 2022). Toolformer-6B (Schick al. (023). Mistral-7B (Jiang et al. (023). PaiChuan2-77 2-Bnnhh (Yang et al. 2023). LLaMA2-7B-on Alpaca-7B (Taori et al. 2023). and some strong RAG models like RobustL Belf-RAG-7B. and REAR.
4.1.3 TRAINING AND INFERENCE SETTINGS
The training is conducted on Alignment Handbook. We use DeepSpeed zero stage 3 (Rajbhandari et al. 2020) to conduct distributed training on 8 A800 GPUs with 80GB memory. FlashAttention (Dao et al. 2022) is also applied to improve the training efficiency. We set the learning rate to 2e-5 and le-6 and the epochs to and for stage and stage 2, respectively. The warm-up ratio is configured 0.1, the global batch size 128 and T' is set to 5.

---

Published as a conference paper at ICLR 2025
Pop QA
Trivia QA
Web Questions
SQuAD
ChatGPT
24.7
78.2
57.0
28.5
Proprictary
GPT-4
38.7
83.4
61.6
35.1
LLaMA2-7B-bass
4.5
10.9
6.1
3.7
Base
LLaMA3-BB-basl
5.7
10.8
4.4
3.8
ChatGLM3-6B
9.5
19.6
14.9
4.3
Mistral-7B
25.2
66.1
56.8
25.9
Fine-tuned
BaiChuan2-7B-chat
25.7
40.7
38.6
13.1
w/o Retrieval
LLaMA2-7B-chat
25.1
58.7
48.6
19.1
Alpaca-7B
25.6
49.4
39.6
14.1
ChatGLM3-6B
42.6
35.7
24.3
19.5
ToolFormer-6B
48.8
26.3
33.8
Mistral-7B
59.1
71.2
57.6
42.0
BuiChuan2-7B-chat
56.2
63.3
50.0
38.7
Fine-tuned
LLaMA2-7B-chat
54.0
63.1
45.3
37.2
w/ Retrieval
Alpaca-7B
54.2
58.6
47.3
32.6
Self-RAG-7B
53.8
62.6
32.4
26.5
REAR
52.9
71.6
38.4
43.8
RobustLM
49.1
62.0
27.3
27.4
Backtracking
+ LLLaMA2-BB-ban
58.4
68.1
47.6
44.5
Correction
+LLaMAA-8BB-aas
59.3
70.1
49.3
44.9
Table 1: Exact match of all the baselines on four open-domain question answering datasets.
During inference, we use vllm Wwon et al. 2023) to speed up inference. For all baselines. the temperature to 0.8, and the cumulative probability of the top tokens is 0.95. We evaluate LLMs' final performance based on whether their generation contains gold answers Asai et al. 2023).
4.2
MAIN RESULTS
Table showws the overall results on four open-domane QA datasets. To make fair comparison, for all the ranking methods, the number of retrieved documents is set to 5 for all ranking methods. In general, fine-tuned L.MSs with retrieval perform better than those without, particularly on tasks involving their knowledge blind spots. Nearly all LLMs struggle to directly answer questions in Pop QA, with accuracy rarely exceeding 26%, including the strong proprietary LLM, ChaIGPT. However, their performance significantly improved when provided with retrieved documents. Almost all achieving an accuracy above 50%, underscoring the importance of RAG. Conversely, RAG may not be beneficial when LLMs possess the relevant knowledge. For example, the accuracy of LLaMA2-13B-
without external documents on Web Questions is 52%, but drops to 47.1% % when augmented retrieval. This phenomenon suggests that retrieval negatively impact LLMs in areas where they are already proficient, as it may introduce additional noise.
Compared strong fine-tuned LLMs and expert RAG models, Backtracking Correction outperforms most of them across the four datasets. Notably, chat models are instruction-tune on massive external data, and RAG models require annotations from proprietary LLMSs. In contrast, Backtracking Correction does not rely on extra data or annotations. Instead, it trains the model in self-generated and self-improved manner, which demonstrates the advantages of our approach. These results indicate that it is possible to achieve competitive performance by further leveraging the original dataset, rather than relying on additional data and annotations.
4.3 BACKTRACKING CORRECTION V.S. OTHER TRAINING METHODS
To enable a fairer comparison with other training methods, we employ Label-Supervised Tuning (LST), Multi-document Sabel-Supervised Tuning (MLST), Chain-Supervised Tuning (CST), and Backtracking Correction (BC) to train the same LLM (Llama2 7B) on the same training data (St. and Scs used by BC), eliminating the impact of additional data and annotations on performance. In particular, MLST trains the LLM SLS similarly to LST, but its input consists of multiple documents rather than a single document. Table Saoleepper presents a case where LST, MLST, CST, and BC produce different answers based the same documents. LST assigns the highest score the
8

---

Published as a conference paper at ICLR 2025
Who is the author of Fruits?
Natsuki Takaya
Her manga series "Fruits Basket", which debuted in 1998, became of the best selling manga in
North America. (Relevance Score: 2.34)
Charles Mason Hovey
Charles Mason Hovey was an American nurseryman, seed merchant, journalist and author of horti- cultural books best known for his two-volume large quarto, The Fruits of America published between 1848 and 1856. (Relevance Score: 2.41)
Fruits (book)
Fruits: A Caribbean Counting Poem (ISBN: 0805051716) is children's picture book written by Valerie Bloom and illustrated by David Axtell. In 1997 it won the Nestlé Smarties Book Prize Bronze Award. (Relevance Score: 2.37)
LST: Charles Mason Hovey.
MLST: Natsuki Takaya,
CST: Document 1: The document states that Natsuki Takaya is the author of the manga series "Fruits Basket," which first appeared in 1998 and has gained significant popularity in North America. This information directly answers the query about the author of "Fruits." So, the document is relevant to the question.
The document states that Charles Mason Hovey is the author "The Fruits of America," a notable work published between 1848 and 1856. This directly answers the query regarding the author of "Fruits," " confirming Hovey's 's authorship of significant book on horticulture. So, the document is relevant to the question.
Document 2:
Document 3: The document identifies Valerie Bloom the author of the children's picture book "Fruits: A Caribbean Counting Poem," which won the Nestlé Smarties Book Prize Bronze Award 1997. This directly answers the query about the author of "Fruits," confirming Bloom's 's authorship. So, the document is relevant to the question.
Final Answer: Based on the analysis of the documents, the answer is Natsuki Takaya.
BC: Document 1: The document provides the name of the author, Natsuki Takaya, and mentions her work "Fruits Basket," which is successful manga series. However, it does not clarify "Fruits" refers to "Fruits Basket" or another work, leaving the specific question unanswered. So, the document is irrelevant the question.
Document 2: The document identifies Charles Mason Hovey as an American nurseryman and the author of "The Fruits of America," a horticultural book published in the mid-19th century. However, it does not specify whether "Fruits" in the query refers to Hovey's 's work or another title, leaving the question of authorship unclear. So, the document is irrelevant to the question.
Document 3: The document states that "Fruits: A Caribbean Counting Poem" is children's picture book written by Valerie Bloom and illustrated by David Axtell. It highlights that the book won the Nestlé Smarties Book Prize Bronze Award in 1997. This directly answers the query regarding the author of "Fruits," confirming that Valerie Bloom the author. So, the document is relevant to the question.
Final Answer: Based the analysis of the documents, the answer is Valerie Bloom.
Table 2: Different models' behaviors for the same context. orange, green, red, and blue text are query, relevant document, irrelevant document and final output.
Pop QA
Trivia QA
Web Questions
SQuAD
LST
2634
17120
$114
25910
BC
776
5024
1360
7392
Table 3: The time efficiency (in seconds) of different methods.
second document, while MI.ST is misled by the first document, leading errors in the final results in both documents. CST incorrectly judges the first document, which negatively impacts the criteria for later decisions. In contrast, Backtracking Correction effectively learns how to handle similar situations during training and ultimately answers the question correctly. More cases can be found in
Appendix B
Figure 3 illustrates the final performance of BC compared with others. Both CST and BC outperform LST and MLST across all benchmarks, demonstrating that learning reasoning chains is beneficial for enhancing the LLM'' ability to assess documents. Chain-of-Thought C' can be viewed as an
9

---

Published as a conference paper at ICLR 2025
Parfomaancc 1.0
Figure 3: The performance of Backtracking Correction compared with other training methods.
Sy:Sttociing
2222
2202
3202
3302
Figure 4: The training process of different methods on Pop QA.
additional supervision signal providing more information for learning alongside the label L. MLST does not consistently outperform LST. However, the increased number of documents may improve the retrieval of relevant ones, which also introduce additional noise. Furthermore, BC achieves higher accuracy than CST, probably due to its more effective utilization of limited data and annotations. The details about the accuracy of classification results are listed in Appendix A
4.4 EFFICIENCY ANALYSIS
reveasJ the time efficiency for inference across four datasets, based on an A800 80GB system. The inference stage our method is similar to CST, where all retrieved documents for a given query are analyzed within the same context. In contrast, LST processes each retrieved document separately. This results in significant increase in the number items to be processed, which limits the ability to fully leverage frameworks like vllm for speed optimization, despite having shorter average context length.
4.5 ABLATION STUDIES
Figure illustrates the training fluctuations of different settings Pop QA. SFT exhibits better stability but a lower ceiling, as good generalization requires extensive annotated data. While improve- ments substantial during the early steps, they become insignificant later on. In contrast, RL-based methods show greater fluctuations but achieve better overall performance. The backtracking algorithm simplifies the RL process by allowing optimization at each step to focus solely on the errors of the current state, as the remaining parts of the reasoning chains have already been learned. This approach enables BC achieve more stable training process.
5 CONCLUSION
This paper highlights the limitations of Lupel-Supervised Tuning (LST) and Chain-Supervised Tuning (CST) in retrieval-augmenter generation (RAG). To address these issues, we reframe the task as a multi-step decision-man maaiiinnnmaiingggg process and propose Backtracking Correction to train retrieval- augmented language models. Compared to existing strong baselines, our method achieves competitive performance without relying on additional data or annotations. Moreover, our approach is not limited to enhancing LLMs' reasoning abilities in RAG. We anticipate that it can be applied to other reasoning tasks like math and code in the future. However, training LLMs' correction abilities may prove to be more challenging outside the RAG context.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDDEMENTT
The work described in this paper was partially funded by the National Natural Science Foundation of China (Grant No. 62272173), the Natural Science Foundation of Guangdong Province (Grant Nos. 2024A1515010089, 2022A15150101791 and the Science and Technology Planning Project of Guangdong Province (Grant No. 2023A0505050106).
REFERENCES
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. CoRR, abs/2310.11511, 2023. doi: 10.48500ARKXI.2310.11111 URL htLpssss:://do orgg 10.g//0.440 48550/arXiv. 2310 11511
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang, Semantic parsing on Freebase from ceestion-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533- 1544, Seattle, Washington, USA, October 2013. Association for org/
Computational Linguistics. URL https: //ww:///wwwwwwo http://wwww.rr.//.oooo//////////
160
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval- augmented generation. In Michael Wooldridge, Jennifer Dy, and Sriraam Natarajan (eds.), Thihty-Lighth AAAI Conference on Artificial Intelligence. AAAII 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024. Vancouver, Canada, pp. 17754-17762 17762. AAAI Press, 2024a. doi: 10. 10.1609/AAAL V38116.2993 URL
https://doi... org/rogg//// 1609 aaai. www.16........ 29722
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024b.
URL httpttttts///oeeetiiiiii
xW9BS
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashatten- tion: Fast and memory-efficient exact attention with io-awareness. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA November 28 December 9, 2022, 2022. URL http:::/pppers..ttttte/// mips. ccppppp /illes/pppe/2222//aaaa haw.aa/ceeeeeeeeeeoooaaatrreeeee html
Anthony DiGiovanni and Ethan C. Zell. Survey of self-play in reinforcement learning. CoRR, abs/2107.02850, 2021. URL htttps://axxi...tt. /ttp///rwwvv.rrrasss////.//
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320-335, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak. Chloe Bi, Chris Marra, Chris McComnel,, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieaa Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie
11

---

Published as a conference paper at ICLR 2025
Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe. Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone. and et al. The llama 3 herd of models. CoRR, abs/2407.2178.. 2024. doi: 10.48550ARXI..440222111
1r0/ 48550/arXiv. 2407 2407.
Çaglar Gülşehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. Reinforced self self-training (rest) for language modeling. CoRR, https2338.00999 2023. doi: 10.48550/ARXIV.2388... 089998. URL. https:t//oo....... org/0 48550/arXiv. 2
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wes Chang. Retrieval aug- mented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 18 July 2020, Virtual Event, volume 119 of Proceedings of PMLR, 2020. URL http::t/tp/:/// http::/prroceedings. mlr
Machine Learning Research, pp. press/v119/guu222 htmll
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski. Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Trans. Mach. Learn. Res., 2022, 2022. URI. httttttts////eeereveeee wwtpeeewwwwwwweeeeeeeeeeeee wwwpteeeteiidnaeeeeee patpprmmmiiddKKKK1XXX1X7100
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Amn Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.0688 2023. doi: 10. 1445500/ARXI..33100066888
URL htthttpp:///oo org.oio..orgg//// //0 138550/aXiiv..o 2310 06825
Mandar Joshi, Funsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Mceting of the Association for Computational Linguisties, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pp. 1601-1611. Association for Computational Linguistics, 2017. doi: 22118833/1/1111111 //doi.org/10 18653/v1/P17-1144)
URL https
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq lqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust. Training language models to self-correet via reinforcement learning, 2024. URL https //arxiv.org/aale hwt.ii/.orgabbs//e/e 2409. 12917
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Jason Flinn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and Jonathan Mace (eds.), Proceedings of the 29th Symposium on Operating Systems Principles. SOSP 2023, Koblenz, Germany, October 23-26, 2023, pp. 61 126.. ACM, 2023. doi: 10. 3600006.3613166 /RL/htttps///doi.rrg// 11 1145/ 36000060000000 3613165
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng. and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning of llms. CoRR, abx/2406. 18629, 2024. doi: 18550/arXiv... 2406 18629
10.44555/ARXXV.2440 110224466.16999 18629. URL http:hhtt::::ooorrpppeeee
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augment tasks. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Floring Bal- can, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL 6b493230285f788e 1bc24455f264
generation for wowleede-intensive NLP
Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian- Yun Nie, and Ji-Rong Wen. Halueval: A large-scale halluci- nation evaluation benchmark for large language models. In Houda Bouamor, Juan Pino, and Kalika
12

---

Published as a conference paper at ICLR 2025
Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023. pp. 6449 6464. Association for Computational Lingusstis,, 2023. URL htLLttss/acaaaatoololoo org/ emnlp-main 397
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Ha- jishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACI. 2023, Toronto, Canada, July 9-14, 2023, pp. 9802-9822. Asso- ciation for Computational Linguistics, 2023a. doi: 10. 10.18653/V1/20233 ACL-LLNG.. URL https://doi.org
18653/v1/2023, acl-long..00 546
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Ha- jishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the st Annual Meeting of the Association for Computational Linguistics (Volume 1: Lonng Papers). ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 9802-9822. Asso- ciation for Computational Linguistics, 2023b. doi: 111163//V112223ALLL-LLL https://doi. hrg////oooorg///// 10. 18653/v1/20223 aol- 546
URL
Ella Neeman, Roce Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. Disentqa: Disentangling parametric and contextual knowledge with counterfactual question an- swering. In Anna Rogers, Jordan L. Boyd Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACI 2023, Toronto, Canada, July 9-14, 2023, pp. (0056-10070) Association for Computational Linguistics, 2023. doi: N011866611/20233ALL-L00 //doi.org/18,
URL https
18653/v1/2023 www.:o///.....63/112222 acl acl-long. 559
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine. Gabriel Bernadett- Shapiro. Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage. Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carcy, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang. Angela Jiang, Roger Jiang. Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez. Ryan Lowe, Patricia Lue, Anna Makanju. Kim Malfacini. Sam Manning, Todor Markov, Yaniv Markovski. Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reicchrr Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokomny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power,
13

---

Published as a conference paper at ICLR 2025
Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond. Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder. Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang. Alvin Wang. Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng. Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Chris- tiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 Decembee 9, 2022. 2022. URL http::/ nips. wttp:ccpeapaeeeeeeeeeeeeeeeseeee hrt.://6662333444000000007711660
html
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off- oolly reinforcement learning. CoRR, abs/1910.00177, 2019. URL
80177
Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Zoubin Ghahramani (ed.), Machine Learning. Proceedings of the Twenty-Fourd International Conference (ICML 2007), Corvallis, Oregon. USA, June 20-24, 2007, volume 227 of ACM International Conference Proceeding Series, pp. 145-750. ACM, 2007. doi: 10.1145/
1273496. URL |RL httphhhtt/ts...... org/ 10. 11 1273197..99 1273590
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick S. H. Lewis. Majid Yazdani, Nicola De Cao. James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 25444. Association for Computational Linguistics, 2021. doi: 11653/v1/22221 nww.naannnaaaaa maal--mainn... 200
11533//1/2221.NAACL.
URL httpss/////iiiio http:///oo01orgg111111163331///2
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimizations toward training trillion parameter models. In Christine Cuicchi, Irene Qualters, and William Kramer (eds.), Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, pp. 20. IEEE/ACM, 2020. doi: org/ 10. 1109/SC41405 2020 00024
URL https://doi
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang, Squad: 100, 000+ questions for machine comprehension of text. In Jian Su, Xavier Carreras, and Kevin Duh (eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4. 2016, pp. 2383 2392. The Association for Computational Lin- guistics, 2016. doi: 10.18653/V1/D1666 1264 URI. wttttt:p///ooo org/10. 0t0.111..166668
1264.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves use tools. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz
14

---

Published as a conference paper at ICLR 2025
Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA. USA, December 10 16. 2023, 2023. URL http: nips. cc/ pappap files/paper/ 2023/hash/dB4242554bb 7tt.5/44bbb77b00035555aaoadoof 658599 -wwtrrbtttttttterraaaaaa html
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning. ICMI, 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 31210-31227. PMLR, 2023a. URL htt.hhtssppprcceeiinggssss mlr wtt.:...prrees/22////2222222
html
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 31210-31227. PMLR, 2023b. URL httptttpps///r/rreeeeinss mlr. pwwps//222/shi2222222 2ttpp//eeeeaaeao.oo.r.e///a/.aa. html
Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou. survey on self-evolution of large language models. CoRR, abs/2404. 14387, 2024. doi: 10.48550/ARXIV.24044 143877 URL httph:tttp:/oii http:://dooooogg///// htgg/0..48550 arXiv 0rX4v.2404.1 14387
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cetrucctionfollovweee llama model.
https://githur com/ htt.:/ccc/tttuuutte waw.ssttsassssaatttaeeeaa
alpaca 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie. Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang. Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, 0e4/2307.09288 2023. doi: 10448550/RXXI.2307..000 092888. Utths:ts::///// org/10../110 00.1104444855 MiXiv.2307.0 092288
Luong Quoc Trung Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning. In Lun-Wei Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 7601-7614. Association for Computational Linguistics. 2024. URL httpps://aclanthollggy org/2024
acl-long. 410
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, Songyang Gao. Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang. Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang. Zuxuan Wu, and Yu-Gang Jiang. Secrets of RLHF in large language models part II: reward modeling. CoRR, abs/2401 006080, 2024a. doi: 10.48550/ARXI..24100000 https://doi /rg//0 48550/ wttpv..iivvvvvee.e 2401 06080
URL
Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in Ilms for enhanced mathematical reasoning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b. URL
15

---

Published as a conference paper at ICLR 2025
Yuhao Wang, Ruiyang Ren, Junyi Li, Wayne Xin Zhao, Jing Liu, and Ji-Rong Wen. REAR: A relevance-aware vetrieval-augmentee framework for open-domain question answering. CoRR. abs/2402.17497, 2024c. doi: 10.48550AARXIV2402..222 2402.
https://::///o1.. org/ o0g/10. (1.4455501
Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md. Rizwan Parvez, and Graham Neubig. Learning to filter context for retrieval- augmented generation. CoRR, abs/231 1.08377, 2023. doi: 10.48550/
ARXIV.2311.0837. URL http/hht:///:oiorrr 00w.1144455000001..222 23111.8377
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian lchter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 Deecember 9, 2022, 2022. URL
nips wwtp:/sisssssccccpppppeeepepppee cwtpcpcppppppeeeeecceeeeee..eeee httppppcccpppeeileeee//eeeeeeeee wwtes//apper/222///aaeee html
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min- Yen Kan, Junxian He, and Michael Qizhe Xie. Self-evaluation guided beam search for reasoning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December cctppppppprriipp.cppppppppppp cc/paper saapperrffllssee files/paper Setaaper/2223//ee 22/23//a22////////
10 16. 2023, 2023. URL 0b- 10--:///00000000--bbbaattt--o-r-
81 81
html
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang. Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie. Juntao Dai, Kun Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang, Mickel Liu, MingAn Lin, Nuolan Nie. Peidong Guo. Ruiyang Sun, Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xiangrong Zeng, Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu, Xuehai Pan, Yanjun Shen, Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao. Yupeng Zhang, Zenan Zhou, and Zhiying Wu. Baichuan 2: Open large-scale language models. CoRR, wwwpaa/s./...//.22 2023. doi: 10.48550/ARXXVV2309.000
//tpss:://dooooo org/ 10. 48550/ 2309. 10305
Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval- language models robust to irrelevant context. CoRR. abs/2310.01558. 2023. doi: 10.48550/ARXIV.23110 01558. URL httpp:://oi..rrr
411..44855///riii 2310.01558.
Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma. Hongwei Wang, and Dong Yu. Chain-of- note: Enhancing robustness in
language models. CoRR, abs/2311.09211 2023. Itth:tt::://dol org/ 10, 485504 2311. 09210
doi: 10.45550/AKXXV2311..0000
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. arXiv preprint
arXiv:2210.02414, 2022.
Tianjun Zhang. Shishir G. Patil, Naman Jain, Sheng Shen, Matei Zaharia, lon Stoica, and Joseph E. Gonzalez. RAFT: adapting language model to domain specific RAG. CoRR, abs/2403. 2024. doi: 10.48550/ARXIV.2440 101311 UtL/httss://dbi.ot /rg/////....r///// 48550/arXiv 2403. 10131
Zihan Zhang, Meng Fang, Ling Chen, Mohammad-Reze Namazi-Rad, and Jun Wang. How do large language models capture the ever-changing world knowledge? A review of recent advances. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore. December 6-10, 2023, pp. 8289-8311. Association for Computational Linguistics, 2023. URL https 202..emm.......... main. 516
//aclanthology.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang. Yupeng Hou, Yingqian Min, Beichen Zhang. Junjic Zhang. Zican Dong, Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Jong Wen. A survey of large language models. CoRR, a 2023. doi: 10.48550/ 2303. 18223
ARXIV.2303.1D 18223. htLpst / //ooi. org/10 10. 48550/arXiv 230.......33.....
16

---

Published as a conference paper at ICLR 2025
T
Pop QA
Trivia QA
Web Questions
SQuAD
2
56.4
60.9
42.9
36.8
3
57.0
63.1
43.2
37.5
LST
57.0
65.3
44.5
38.3
5
56.9
64.5
45.2
38.3
2
58.1
66.8
46.2
44.0
3
58.8
67.3
46.6
44.8
CST
59.2
67.3
47.1
45.4
5
58.6
67.0
47.7
45.2
2
60.9
70.4
50.6
46.8
3
61.7
71.2
51.4
47.3
BC
62.0
71.5
51.5
47.6
5 61.7
71.6
51.5
47.3
Table 4: The hit rate of different methods under different T.
A THE ANALYSIS OF INTERMEDIATE RESULT
LST methods select the document with the highest score as the final context, making hard to use accuracy to assess their classification abilities. For example, there may be more than one documents relevant to the given query. Only the one with the highest score determined as 'relevant' and the others are all determined as 'irrelevant' However, the accuracy can not reflect models' classification abilities. Considering that, we adopt the hit rate the documents determined as 'relevant', The performance under different T is listed in Table 4 The results show that BC achieves higher classification accuracy when judging the relevance of retrieved documents.
B CASE STUDY
We evaluate the performance of Backtracking Correction in extreme scenarios. There are no instances where all the retrieved documents "relevant", but there are cases where all the retrieved documents are judged as "irrelevant". Table shows an example. The model refuses to provide any answers, resulting an accuracy of O. Although there are few similar samples in the training data, the model does not typically follow this pattern. We suspect this is due to the small proportion of such samples the training set.
Besides, we also found some documents that have ambiguous relevance to the query. As shown in Table 6 multiple documents (Document 1, Document 2, Document 3, and Document 5) mention Henry Feilden (due to multiple namesakes). The gold answer in the dataset is "politician," indicating that Document is relevant, while the others are not. However, it can be ambiguous to classify Documents 2, 3, and 5 as irrelevant, since they also mention the name "Henry Feilden." Such ambiguities are difficult to avoid during both training and inference, as the original human annotators may not have reviewed all documents with potentially ambiguous relevance to the query. As with previous work, we consider only those documents containing the gold answers as relevant.
17

---

Published as a conference paper at ICLR 2025
What sport does Best play? Best Player
Best Player is 2011 I television film that aired on Nickelodeon March 12, 2011. The movie stars
Jerry Trainor and Jennette McCurdy, from the show iCarly. Filming started on October 24, 2009 in Victoria, British Columbia, Canada and wrapped up production on November 18, 2009.
George Best
Best played for three clubs in the United States: Los Angeles Aztecs, Fort Lauderdale Strikers and later San Jose Earthquakes; he also played for the Detroit Express European tour. Best was a success on the field, scoring 15 goals in 24 games in his first season with the Aztecs and named as the NASL's best midfielder in his second. He and manager Ken Adam opened "Bestie's 's Beach Club" (now called "The Underground" after the London subway system) in Hermosa Beach, California in the 1970s, and continued to operate it until the 1990s. Ultimate in Japan
Ultimate, originally called ultimate frisbee, is non-contact team field sport played with a flying disc, invented in New Jersey, USA, in 1968. Japanese players and teams rose to prominence in the 1990s, and today are among the strongest competitors in the sport globally. Football
Thus, the best association football and American football players are among the highest paid athletes in the world. Australian rules football has the highest spectator attendance of all sports in Australia, Similarly, Gaelic football is the most popular sport in Ireland in terms of match attendance, and the All-Ireland Football Final is the most watched event of that nation's 's sporting year. Rugby union is the most popular sport in New Zealand, Samoa, Tonga, and Fiji. It is also the fastest growing sport in the U.S. with college rugby being the fastest growing college sport in that country. List of rural sports and games
Aunt Sally - An Oxfordshire game, it is the under arm throwing of the dolly (a truncheon shaped stick) at a suspended larget. Each player in the team has 6 throws. The hest score out of 24 wins. Bat and trap - An English bat-and-ball game, played in pub gardens mostly in Kent. ; Bog snorkelling - Competitors must complete two consecutive lengths of a water filled trench cut through peat bog, in the shortest time possible. ; Coconut shy- - Each player has 6 balls to throw at targets of coconuts balanced on raised stands.
BC: Document I: The document is about a television film called "Best Player," which stars Jerry Trainor and Jennette McCurdy, but it doesn't mention a sport. It appears to be a film about a character named Best, possibly nickname or character's 's name, but the document does not specify what sport the character is associated with. So, the document not relevant to the question.
Document 2: The document is about George Best, former professional soccer player, and his career in the United States and his business ventures, specifically his nightclub, "Bestie's 's Beach Club." It mentions his playing career in the North American Soccer League (NASL) and his achievements, but does not mention any information about the sport he currently plays. The document is irrelevant to the question "What sport does Best play" because it does not provide any information about Best's s current sport or activity, but rather focuses on his past career and personal life.
Document 3: The document is brief overview of the sport of Ultimate, specifically its origins, its rise in popularity in Japan, and its global competitiveness. is irrelevant to the question "What sport does Best play" because the document does not mention person named Best. It is actually talking about the sport Ultimate, which is team sport, not person's 's name. So, the document is not relevant to the question.
Document 4: The document discusses the popularity and revenue of various types of football, including association football, American football, Australian rules football, Gaelic football, and Rugby union. It mentions that these sports have high revenue, high spectator attendance, and are popular in different countries. This document is irrelevant to the question "What sport does Best play" because it does not mention any specific individual or team named "Best" and does not provide information about their sport of choice. The document is a general overview of the popularity of different sports, but it does not contain any information that would help answer the question. So, the document is irrelevant to the question.
Document 5: The document appears be list of various traditional rural sports and games, including Aunt Sally, Bat and trap, Bog snorkelling, Coconut shy, and Cooper's s Throw. This document is irrelevant to the question "What sport does Best play" because the document does not mention the name "Best" or any information about a person named Best participating in any of the listed sports, The question seems to be asking about an individual's 's sport, whereas the document lists various traditional sports and games in general. So, the document is irrelevant to the question. Final Answer: The sport that Best plays is not uniquely determined by the provided documents.
Table 5: The case that Backtracking Correction handle an extreme case where all retrieved documents are entirely irrelevant.
18

---

Published as a conference paper at ICLR 2025
What is Henry Feilden's 's occupation?
Henry Feilden (Conservative politician)
Henry Master Feilden (21 February 1818 2013 5 September 1875) was an English Conservative Party politician.
Feilden
Bernard Feilden (191920132008) British conservation architect; ; Bob Feilden (191720132004) British mechanical engineer; ; Gerry Feilden (190420131981), British general and horse racing identity Henry Feilden (disambiguation) Joseph Feilden (18242013 1895), British politician Richard
Feilden (195020132005), British architect; ; William Feilden (177220131850) British politician.
Feilden baronets
Henry Wemyss Feilden, second son of the second Baronet, was an Arctic explorer.
Bernard Feilden
Feilden was born in Hampstead, London. He was educated at Bedford School and The Bartlett School of Architecture, University College, London, completing his training at the Architectural Association after the second world war. His love of architecture was inherited from his grandfather, Brightwen Binyon (1846-1905), an Ipswich architect and former pupil of Alfred Waterhouse. He joined the practice of Edward Boardman and Son in Norwich, where he designed the Trinity United Reformed Church. He set up architectural practice, Feilden+Mawson, with David Mawson in 1956, to which offices in Norwich, London and Cambridge were later added.
Henry Wemyss Feilden
Colonel Henry Wemyss Feilden, CB (6 October 1838 2013 8 June 1921) was a British Army officer, Arctic explorer and naturalist. BC: Document I:
The document a brief biography of Henry Feilden, stating that he was an English Conservative Party politician. It is relevant to the question about Henry Feilden's 's occupation because it explicitly mentions that he was politician, specifically a Conservative politician, which directly answers the question. So, the document is relevant to the question.
Document 2: The document lists various individuals with the surname Feilden, including their birth and death years, along with a brief description of their profession or notable achievements. Henry Feilden is mentioned as disambiguation, but no specific information is provided about his occupation. However, upon examining the list, there is information about a notable Henry Feilden who is a well-known figure, suggesting that the document is likely referring to the disambiguation page, which is placeholder for a person with that name. So, the document is relevant to the question, but the answer cannot be definitively determined from the provided information.
Document 3: The document briefly describes Henry Feilden as the second son of the second Baronet and mentions that he was Arctic explorer. This document is relevant to the question because it provides information about Henry Feilden's occupation, specifically that he was an Arctic explorer. Document 4: The document briefly describes the life and career of Bernard Feilden, an architect, including his education, family background, and professional accomplishments. mentions his love for architecture, his apprenticeship with Edward Boardman and Son, and his establishment of his own architectural practice, Beilddn++Mwsson This document is irrelevant to the question "What is Henry Feilden's 's occupation" because the document docs not mention a person named Henry Feilden, but rather Bernard Feilden, and there is no information about Henry's 's occupation.
Document 5: The document is a brief biographical description of Henry Feilden, stating that he was a British Army officer, Aretic explorer, and naturalist. This information is relevant to the question "What is Henry Feilden's occupation?" because it directly answers the question by listing his occupations, which include being a British Army officer, an Arctic explorer, and naturalist. So, the document is relevant to the question.
Final Answer: Based the analysis of the documents, the answer is Arctic explorer.
Table 6: The case that Backtracking Correction handle an extreme case where all retrieved documents entirely irrelevant.
19


Reviews:
[{"review_id": "nuRkqOXDbq", "reviewer": "Reviewer_gD7f", "summary": "This paper introduces Backtracking Correction (BC), a novel approach for training large language models (LLMs) to detect noisy retrieved documents.\n\nBacktracking Correction reformulates the task as a multi-step decision-making process, optimizing the model by iteratively correcting errors from the final reasoning steps back to the initial steps, thereby improving the model's ability to assess the relevance of retrieved documents and enhance the robustness of RAG against noisy inputs. \n\nExperimental evaluations on multiple open-domain question-answering datasets demonstrate that Backtracking Correction outperforms multiple baselines without relying on additional data or extensive annotations.", "questions": "+ Does BC implied a conditional depedency between the relevancy of the retrieved documents?\n+ Could you provide the exact performance numbers for figure 3?\n+ Could you include an efficiency analysis comparing the BC method to the baselines?\n+ In section 4.2, the author claim that BC does not use extra data or annotations. Can you elaborate. Is BS's backbone trained from scratch? Does the training process of BC still require label of the reasoning chain as training data?\n+ Can you elaborate on the learned ability of \"summarizing the consistent or conflicting points\" from BC?\n+ Can you elaborate on the following claim that \"the training data and reference models in Backtracking Correction are dynamic, while those in DPO are static\"?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+  The proposed Backtracking Correction method offers a novel approach to training LLMs for RAG tasks without relying on extensive external annotations, which could potentially reduce the cost and complexity of developing such systems.", "weaknesses": "+ The paper needs additional ablation studies and analysis for the proposed algorithm, especially for the efficiency claim and the scaling problem in term of chain length.\n+ The experimental results are not consistently significant, with three out of four benchmarks (excluding SQuAD) showing either comparable performance or underperformance relative to baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7FJcAm5YQD", "reviewer": "Reviewer_ofNF", "summary": "In the RAG setting, it is sometimes required to distinguish whether each document is relevant or irrelevant, and stepwise reasoning may be utilized for this purpose. However, training an LLM for this task poses challenges such as the need for extensive annotation. This paper proposes Backtracking Correction, which reformulates the stepwise reasoning of RAG as a decision process for each document. It separates the decision process of the multiple documents into context space (given the correct decision) and decision space (model output) starting from the final state, enabling an easy-to-hard progression.", "questions": "- Please refer to the mentioned weaknesses.\n- Also, as the number of documents increases, the binary decision tree is likely to grow very rapidly. Is there a way to sample better paths without considering all paths in the decision space during training?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- It is very well-written, and the figures are well-illustrated, making it easy to follow.\n- A well-defined problem with a reasonable proposed method.", "weaknesses": "There are some concerns with the experimental setup.\n- Fair comparison with baselines: Why weren’t the base models of Backtracking Correction (LL2-7B-base, LL3-8B-base) used as baselines? Non-chat base models could also have been fine-tuned. Including the base models in the baseline or incorporating chat models into Backtracking would ensure a fair comparison.\n- \"Without relying on additional data or annotations.\" Lines 411-416 mention that competitive performance can be achieved without additional training data (e.g., chat models). However, the model used to obtain the original annotation itself is a chat model. Is there a reason why the model used for SCS original annotation (line 326) and the backbone model were set differently?\n- (Very minor) There is a typo \"err\" in the abstract, and it does not match the abstract on OpenReview.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2K2BIz7dBG", "reviewer": "Reviewer_hZRq", "summary": "This paper proposes an interesting Backtracking Correction method to enhance LLMs' capability on RAG QA. Specifically, they employ Self-play algorithm to correct LLM's own CoT errors in a backward manner, and leverage the reward difference between the corrected and original outputs to train LLMs.  The authors present positive results by comparing their proposed algorithm comparing with Proprietary LLMs, SFT without retrieval, and previous SOTA SFT retrieval methods. An example and some analysis presented to help readers understand how their method work.", "questions": "- The naming of backtracking is a bit unintuitive. A typical backtracking algorithm starts with T=0, explore all path, and then backtrack. Here you don't have the initial explorations. Have you considered a less confusing name?\n- Have you tried to shuffle the order of the documents and try your method there? How would that change the results?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The idea of the paper is quite interesting in that it leverages the self-play algorithm to produce corrected CoT, which mitigates the lack of annotations problems.\n2. The paper is well structured so it is relatively easy to follow despite the complexity (though it can be made clearer - see my comments below).\n3. Results are in general in favor of the proposed method and some further analysis presented to help readers understand the improvements.", "weaknesses": "1. The overall structure of the paper is clear. However, some important details are missing.\n- I think some notations are not clearly defined. For example, in Section 3.3, what exactly are $\\Pi_{ref}$ and $o_t^-$?\n- In Eq. 6, there appears to be inputs and outputs after the chain of equations, what are those? If this is an iterative process, I suggest using an actual example to illustrate better.\n- Some math derivations are not immediately clear. For example, Eq. 10 and Eq. 12. I suggest the authors to add more details here or use the appendix to elaborate more.\n\n2. Experiments can be more thorough.\n- In Table 1 and Figure 3, we could conclude that this methods works for Llama-2-7b models, but does it work for other models? Adding experiments here will make the paper stronger. \n- Similarly, most QA datasets are short-form factoid QA, would this method generalizes to other more complicated RAG-QA tasks, for example: https://arxiv.org/abs/2407.13998?\n\n3. The authors show 1 example in Table 2, which is helpful, but I encourage them to dive deep by analyzing more examples and showing readers exactly how exactly correction of CoT works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DKi65kh6IQ", "reviewer": "Reviewer_ED7n", "summary": "This paper focuses on issues with RAG models having difficulties when encountering irrelevant documents and in particular, in sequential decision making processes where documents are classified as relevant or irrelevant before a final answer is produced. The authors explain issues with existing methods for chain and label based supervised fine tuning and instead propose a backtracking correction method. Overall the paper is well written and, for the most part well presented. \n\nWhile the proposed method shows some modest gains, Its hard to determine the full benefits as there is only one comparison of the same model being trained with backtracking vs baselines. It would be informative to see more comparisons of models in the `Fine-tuned w/retrieval `section of table 1 also evaluated using backtracking.  Additionally, it would be useful to see a bit more detail on the experimental setup and some analysis or detail on how much more expensive the backtracking method is to use.", "questions": "- what is the wallclock train time of the different methods?\n- does the method improve over baseline training setups for other models? Only llama2 7B is shown in fig 3 but llama3-8b was also trained with backtracking as could many of those found in fine-tuned w/ retrieval section of table 1\n- what is the accuracy of relevance /irrelevance classification with different methods? afaict only end-to-end answering performance is shown", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Overall, the paper is well written, organized, and presented with informative figures\n- the presented method demonstrates some gains over baseline methods", "weaknesses": "- only a single model is shown comparing baseline training methods against the proposed method with modest gains\n- evaluation datasets are all fairly old and most certainly appear in the training data of many of the investigated models\n- the presentation of sec 3.3 is overly complex and may benefit from some editing, subsections, or the addition of a simple figure", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JAA0wrdhvG", "reviewer": "Reviewer_c4sg", "summary": "The paper introduces Backtracking Correction, a novel method to enhance RAG by transforming it into a multi-step decision-making process. BC optimizes models through error sampling and self-correction, starting from the final step and backtracking to previous states, simplifying the learning of reasoning chains and improving robustness against noisy documents.", "questions": "1.Could the authors share additional insights into the error analysis of the model, particularly in the context of multi-step reasoning questions? A breakdown of errors by reasoning step or a comparison of error types between BC and baseline methods would be meaningful.   \n2.What impact does an increase in the number of documents have on BC's performance? It would be beneficial to see experiments demonstrating how performance metrics change as the number of retrieved documents increases.   \n3.Can Backtracking Correction effectively handle extreme cases where all retrieved documents are either highly relevant or entirely irrelevant? Evaluating BC's performance in these scenarios could provide valuable insights.  \n4.How does Backtracking Correction manage documents that have ambiguous relevance to the query? Furthermore, what effect does this ambiguity have on the model's final output? A detailed analysis of these cases would enhance the understanding of BC's robustness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.BC's backtracking algorithm provides a fresh perspective on training LLMs for RAG, simplifying complex reasoning into manageable steps, which is innovative in the context of handling noisy information in document retrieval.  \n2.By eliminating the need for extensive annotations, BC lowers the barrier for training complex LLMs, making it more accessible and cost-effective compared to methods relying on detailed reasoning chain annotations.  \n3.The paper demonstrates BC's effectiveness through extensive experiments, showing improved performance over strong baselines, including fine-tuned LLMs and proprietary models, highlighting the practical value of the proposed method.", "weaknesses": "1.The paper lacks an adequate exploration of BC's limitations in processing larger datasets and executing more complex reasoning tasks, like multi-hop reasoning. Including a thorough discussion on these aspects could significantly strengthen the analysis.  \n2.While BC is proposed as a simplified approach, the backtracking process might still be computationally expensive, especially for environments with limited resources, which could hinder its adoption.  \n* The numbering of the equations contains errors, which has resulted in confusion in their citations throughout the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)