LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data
Authors: Cehao Yang, Xueyuan Lin, Chengjin Xu, Xuhui Jiang, Shengjie Ma, Aofan Liu, Hui Xiong, Jian Guo
Accepted at: acl25

LONGFAITH: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data

Cehao Yang1,2*, Xueyuan Lin1,2,4*, Chengjin Xu1,3*, Xuhui Jiang1,3,
Shengjie Ma1, Aofan Liu1, Hui Xiong2†, Jian Guo1†
1IDEA Research, International Digital Economy Academy
2Hong Kong University of Science and Technology (Guangzhou)
3DataArc Tech Ltd.
4Hithink RoyalFlush Information Network Co., Ltd
{yangcehao,linxueyuan,xuchengjin,jiangxuhui,mashengjie,liuaofan,guojian}@idea.edu.cn
{cyang289,xlin058}@connect.hkust-gz.edu.cn, xionghui@ust.hk

Abstract

Despite the growing development of long-context large language models (LLMs), data-centric approaches relying on synthetic data have been hindered by issues related to faithfulness, which limit their effectiveness in enhancing model performance on tasks such as long-context reasoning and question answering (QA). These challenges are often exacerbated by misinformation caused by lack of verification, reasoning without attribution, and potential knowledge conflicts. We propose LONGFAITH, a novel pipeline for synthesizing faithful long-context reasoning instruction datasets. By integrating ground truth and citation-based reasoning prompts, we eliminate distractions and improve the accuracy of reasoning chains, thus mitigating the need for costly verification processes. We open-source two synthesized datasets—LONGFAITH-SFT and LONGFAITH-PO—which systematically address multiple dimensions of faithfulness, including verified reasoning, attribution, and contextual grounding. Extensive experiments on multi-hop reasoning datasets and LongBench demonstrate that models fine-tuned on these datasets significantly improve performance. Our ablation studies highlight the scalability and adaptability of the LONGFAITH pipeline, showcasing its broad applicability in developing long-context LLMs.

1 Introduction

Long-context processing ability has emerged as a significant challenge for large language models (LLMs) (Shi et al., 2023; Liu et al., 2024; Wu et al., 2024; Levy et al., 2024), especially arises when models process extensive textual information, making it hard to recognize relevant evidence and address downstream tasks such as question answering (QA), summarization, and complex reasoning (Bai et al., 2023, 2024b; Zhang et al., 2024d; Hsieh et al., 2024; Yen et al., 2024). A variety of model-centric methods have been proposed to extend the length of context windows in LLMs (Chen et al., 2023a,b; Peng et al., 2023; Han et al., 2024; Ding et al., 2024). Additionally, many data-centric methods, such as synthesizing long-context understanding instructions for fine-tuning, have gained attention for enhancing models’ ability to handle and utilize extended contexts (Xiong et al., 2023; An et al., 2024; Fu et al., 2024; Bai et al., 2024a; Li et al., 2024b; Gao et al., 2024; Chen et al., 2024; Zhang et al., 2024c; Li et al., 2024a; Jiang et al., 2025).

Despite the improvements in downstream QA performance enabled by synthetic long-context reasoning instructions, concerns remain regarding the faithfulness of such generated data. Specifically: (1) Misinformation due to lack of verification: existing methods often generate QA pairs without rigorous rule-based verification. For instance, (Chen et al., 2024; Zhang et al., 2024c; Li et al., 2024a) directly synthesize QA pairs using LLMs while bypassing verification, whereas (Zhang et al., 2024c) relies on AI-generated feedback in soft dimensions rather than human annotation. (2) Reasoning without attribution: prompting LLMs to generate responses with citation, such as using Chain-of-Citation (CoC) prompting (Li et al.,

![A brief introduction of LONGFAITH. Synthesized long-context reasoning instruction sets and preference datasets are fed into the post-training stage of downstream LLMs.](page_1016_370_388_246.png)

Figure 1: A brief introduction of LONGFAITH. Synthesized long-context reasoning instruction sets and preference datasets are fed into the post-training stage of downstream LLMs.
The All Saints Church is located in Lockerbie, Scotland, which is part of the United Kingdom.
……
The formal British colonization of Panama occurred later, after the separation of Panama from Colombia in 1903, when the UK recognized Panama’s independence.
The answer is 1903.

Misinformation  Reason w/o Attribution  Knowledge Conflicts

Generate Reasoning Chain Directly
When was the country containing Alanje colonized by the country housing the All Saints Church in the town known for the bombing by Gaddafi’s Libya?
[1] Muammar Gaddafi:
From childhood, Gaddafi was aware of the involvement of European colonialists in Libya…..
[2] Alanje: Alanje is a corregimiento in Alanje District, ChiriquiProvince, Panama. It is the seat of Alanje District

Generate Reasoning Chain with Ground Truth Guidance and Chain-of-Citation
Alanje is a corregimiento in Panama, specifically in Chiriqui Province as stated in the document [10].
……
According to document [14], in 1695, the Scottish Parliament granted a charter to the Company of Scotland, which established a settlement on the isthmus of Panama in 1698.
The answer is 1698.

Verified Reasoning w/ Correct Answer  Reasoning w/ Attribution

Previous Studies
Instructions with low-faithful reasoning chains for training

Reasoning Chains with Misinformation
Reasoning Chains w/o Attribution
Reasoning Chains with Knowledge Conflict
Reasoning Chains with High Faithfulness!

SFT
Preference Optimization
LongFaith

Figure 2: Overview of LongFaith pipeline for synthesizing faithful long-context reasoning instruction and preference datasets. Comparing generated reasoning chains with misinformation, lack of attribution, and knowledge conflicts, LongFaith generates ground truth guidance prompting by chain-of-citation to build LongFaith-SFT. Fine-grained faithfulness is modeled by optimization on our preference datasets LongFaith-PO.

2023, 2024b; Fierro et al., 2024; Huang et al., 2024; Berchansky et al., 2024; Gao et al., 2023) can enhance the credibility and interpretability of model outputs under long-context QA tasks (Gao et al., 2023; Zhang et al., 2024b), yet most prior works ignore to incorporate this technique during their synthesis of training instruction pairs. (3) Potential knowledge conflicts: some approaches (Bai et al., 2024a; Zhang et al., 2024c; Chen et al., 2024) over-rely on the Self-Instruct technique (Wang et al., 2022) to generate QA pairs, encouraging models to rely on parametric knowledge rather than grounding reasoning in explicit contextual evidence (Xu et al., 2024). Additionally, (Zhang et al., 2024c) feeds the query and response to a short-context reward model ignoring the long context to score, purely relying on the parametric knowledge inside LLM. These limitations underscore the necessity for a more robust pipeline that ensures the faithfulness of long-context instructions synthesis.

We propose LongFaith, a novel pipeline for synthesizing faithful long-context reasoning instruction datasets. We incorporate ground truth directly into the prompt for synthesizing long-context reasoning chains, which comprise supporting facts and the correct answer, and prompt LLMs to reason with attributions. This method ensures the faithfulness of synthesized reasoning chains without requiring costly verification by a curated rule-based evaluator, LLM-as-a-judge (Gu et al., 2024) or human annotator. We open-source LongFaith-SFT, synthesized under the guidance of ground truth and CoC prompting. We leverage the faithful long-context reasoning chains with attributions for training, leading to performance improvements after fine-tuning Llama-3.1-8B-Instruct. Additionally, we synthesize preference datasets by sampling preference pairs around fine-grained faithfulness: (1) encouraging model to reason with attributions; (2) encouraging model to learn on verified reasoning chains; and (3) encouraging model to reason with contextual documents grounded. We open-source LongFaith-PO, synthesized by various LLMs in different sizes, which integrates all three faithfulness dimensions for preference optimization. We leverage these faithful preferred instruction pairs for training Llama-3.1-8B-Instruct, achieving performance improvements on the multi-hop reason-
ing dataset and LongBench (Bai et al., 2023).

Our main contributions are as follows: (1) We introduce LONGFAITH, a novel pipeline for synthesizing faithful long-context reasoning instruction data. (2) We open-source LONGFAITH-SFT and LONGFAITH-PO, two synthesized datasets that are systematically designed considering multiple dimensions of faithfulness. (3) We conduct extensive experiments on two types of datasets (comprising eight sub-tasks) to show that models trained on LONGFAITH datasets can improve in long-context reasoning and QA tasks. (4) Our ablation studies illustrate the scaling potential and adaptability of LONGFAITH pipeline, underscoring its broad applicability in the development of long-context LLMs.

2 Related Work

Long-Context Utilization. Amounts of studies focus on enhancing LLMs to better utilize long-context information. Model-centric approaches, for instance, optimizations on attention mechanism aim to capture specific sequential features (Beltagy et al., 2020; Ding et al., 2023; Chen et al., 2023b; Han et al., 2024), while positional interpolation techniques are utilized to scale positional encoding while ensuring valid index ranges (Zhu et al., 2023; Chen et al., 2023a; Ding et al., 2024; Peng et al., 2023; Beltagy et al., 2020). In addition, data-driven approaches also gain popularity, emphasizing high-quality data synthesis for fine-tuning to improve LLMs’ long-context processing capabilities. For example, (Xiong et al., 2023; Gao et al., 2024) employ long-sequence continuous pre-training on foundation models, while (Fu et al., 2024) explores the impact of pre-training data composition and balance. Additionally, works on SFT with synthetic instructions (An et al., 2024; Bai et al., 2024a; Li et al., 2024b; Chen et al., 2024) not only consider long-context understanding but also strengthen multi-hop reasoning capabilities. Lastly, preference optimization approaches (Zhang et al., 2024c; Li et al., 2024a) generate fine-grained pairwise preference instruction sets and incorporate training techniques (Rafailov et al., 2024; Hong et al., 2024). From the perspective of improving the faithfulness of synthetic data, our work effectively addresses the shortcomings of prior studies in this area.

Faithful Reasoning. Hallucination in LLMs presents a major challenge in knowledge-intensive tasks (Zhang et al., 2023; Huang et al., 2023). Recent work has focused on enhancing faithful reasoning, where the goal is to trace the LLM’s generated content back to reliable sources and ensure its factual grounding. (Berchansky et al., 2024; Li et al., 2023, 2024b) aim to improve the identification and verification of attributions by focusing on generating reasoning outputs that link claims to specific sources. Benchmarks such as (Gao et al., 2023; Yue et al., 2023) evaluate the quality of citations and highlight the limitations of current systems in providing citation support to ensure more reliable output. Additionally, integrating external knowledge sources has gained attention, which use retrieval-augmented generation (RAG) methods to facilitate deep and faithful reasoning (Sun et al., 2023; Ma et al., 2024). Our LONGFAITH is motivated by previous work, towards faithful reasoning in long-context reasoning tasks.

3 LongFaith

In this section, we present an exposition of LONGFAITH pipeline. Specifically, we explain how it synthesizes LONGFAITH-SFT for supervised fine-tuning and LONGFAITH-PO for preference optimization from the perspective of faithfulness.

Synthesize Reasoning Chains with High Faithfulness. Previous studies (Bai et al., 2024a; Chen et al., 2023b, 2024; Zhang et al., 2024c; Li et al., 2024a) tend to directly distill synthesized long-context QA and reasoning instructions for training without filtering out incorrect information. These low-faithfulness synthesized data limit the performance improvements of the trained models. In response to this challenge, LONGFAITH integrates ground truth into the synthesized reasoning chains. For a sample \( S \) from the training set \( S = (Q, D, A, F) \), where \( Q \) is the reasoning question, \( D \) is the full document used for querying, \( A \) is the correct answer, and \( F \) represents the supporting facts where \( F \in D \). We use the LLM \( M_{syn} \) to synthesize the reasoning chain as follows:

\[
O_c = M_{syn}(P_{coc}, Q, F, A)
\]

Here, \( O_c \) represents the output of \( M_{syn} \), which is a step-by-step reasoning chain. The prompt \( P_{coc} \) utilizes a chain-of-citation (Li et al., 2024b) prompting approach, requiring the model to reason with attribution (e.g., "Let’s reason step by step while citing the document using [1], [2], etc."). The prompt template is shown in Figure 7.
LONGFAITH-SFT Dataset. Towards training a downstream LLM to reason with high faithfulness for a long-context QA task, we construct the dataset for supervised fine-tuning, where each instruction pair is built as follow:

\[
I_{sft} = \{\text{input} = (P_{\text{coc}}, Q, D), \text{output} = O_c\}
\]

Synthesize Reasoning Chain with Questionable Faithfulness. To model fine-grained preferences, we address three challenges that affect the faithfulness of synthesized instructions: (1) misinformation due to lack of verification, (2) reasoning without attribution, and (3) potential knowledge conflicts. We synthesize reasoning chains with questionable faithfulness, including reasoning chains with misinformation as follows:

\[
O_m = M_{\text{syn}}(P_{\text{coc}}, Q, D)
\]

Since there is no ground truth to guide the synthesis, the output \( O_m \) may contain errors in reasoning, as illustrated in Figure 8, where the model generates an incorrect answer of "1903" instead of the correct answer "1698". This hallucination is common in synthesized data from previous works unless rules or human experts are involved in filtering (Li et al., 2024b). Next, we synthesize reasoning chains without attribution:

\[
O_{\text{cot}} = M_{\text{syn}}(P_{\text{cot}}, Q, F, A)
\]

Here, the CoT (Wei et al., 2022) prompting only requires the model to provide step-level reasoning, but as shown in Figure 9, reasoning without attribution not only loses interpretability and credibility (Gao et al., 2023; Li et al., 2023), but our results in Tab. 4 (Sec. 4) also demonstrate that CoT prompting performs worse than CoC. Finally, we synthesize reasoning chains with potential knowledge conflicts:

\[
O_{\text{kc}} = M_{\text{syn}}(P_{\text{cot}}, Q, A)
\]

Since no context is provided, the model relies solely on its parametric knowledge for reasoning, as shown in Figure 10, where the model states, "Panama was not colonized by the United Kingdom; Panama was colonized by Spain," based on internal parametric knowledge rather than the contextual documents. Previous studies (Zhang et al., 2024c) using short-context reward models observes performance degradation by ignoring long-context materials, highlighting the limitation of knowledge conflicts in affecting LLM’s performance in long-context QA and reasoning tasks.

<table>
  <tr>
    <th colspan="5">Synthesis of Reasoning Chains</th>
  </tr>
  <tr>
    <th>Models</th>
    <th>Prompt</th>
    <th>w/ GT</th>
    <th>w/ Doc</th>
    <th>Output</th>
    <th>Size</th>
  </tr>
  <tr>
    <td>Q-7B</td>
    <td>CoC</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>1</td>
    <td>1-8K</td>
  </tr>
  <tr>
    <td>Q-7B</td>
    <td>CoT</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>2</td>
    <td>1-8K</td>
  </tr>
  <tr>
    <td>Q-7B</td>
    <td>CoC</td>
    <td>✗</td>
    <td>✓</td>
    <td>✓</td>
    <td>3</td>
    <td>1-8K</td>
  </tr>
  <tr>
    <td>Q-7B</td>
    <td>CoT</td>
    <td>✓</td>
    <td>✗</td>
    <td>✓</td>
    <td>4</td>
    <td>1-8K</td>
  </tr>
  <tr>
    <td>L8,L70,G</td>
    <td>CoC</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>5</td>
    <td>2K</td>
  </tr>
  <tr>
    <td>L8,L70,G</td>
    <td>CoT</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>6</td>
    <td>2K</td>
  </tr>
  <tr>
    <td>L8,L70,G</td>
    <td>CoC</td>
    <td>✗</td>
    <td>✓</td>
    <td>✓</td>
    <td>7</td>
    <td>2K</td>
  </tr>
  <tr>
    <td>L8,L70,G</td>
    <td>CoT</td>
    <td>✓</td>
    <td>✗</td>
    <td>✓</td>
    <td>8</td>
    <td>2K</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="4">Datasets for Supervised Fine-tuning</th>
  </tr>
  <tr>
    <th>Name</th>
    <th>Models</th>
    <th>Instruction</th>
    <th>Output</th>
    <th>Size</th>
  </tr>
  <tr>
    <td>LF-SFT</td>
    <td>Q-7B</td>
    <td>CoC</td>
    <td>1</td>
    <td>1-8K</td>
  </tr>
  <tr>
    <td>w/o CoC</td>
    <td>Q-7B</td>
    <td>CoT</td>
    <td>2</td>
    <td>1-8K</td>
  </tr>
  <tr>
    <td>w/o GT</td>
    <td>Q-7B</td>
    <td>CoC</td>
    <td>3</td>
    <td>1-8K</td>
  </tr>
  <tr>
    <td>w/o Doc</td>
    <td>Q-7B</td>
    <td>CoC</td>
    <td>4</td>
    <td>1-8K</td>
  </tr>
  <tr>
    <td>LF-SFT</td>
    <td>L8,L70,G</td>
    <td>CoC</td>
    <td>5</td>
    <td>2K</td>
  </tr>
  <tr>
    <td>w/o CoC</td>
    <td>L8,L70,G</td>
    <td>CoT</td>
    <td>6</td>
    <td>2K</td>
  </tr>
  <tr>
    <td>w/o GT</td>
    <td>L8,L70,G</td>
    <td>CoC</td>
    <td>7</td>
    <td>2K</td>
  </tr>
  <tr>
    <td>w/o Doc</td>
    <td>L8,L70,G</td>
    <td>CoC</td>
    <td>8</td>
    <td>2K</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="5">Datasets for Preference Optimization</th>
  </tr>
  <tr>
    <th>Name</th>
    <th>Models</th>
    <th>Instruction</th>
    <th>Chosen</th>
    <th>Rejected</th>
    <th>Size</th>
  </tr>
  <tr>
    <td>w/ CoC</td>
    <td>Q-7B</td>
    <td>CoC</td>
    <td>1</td>
    <td>2</td>
    <td>1-8K</td>
  </tr>
  <tr>
    <td>w/ GT</td>
    <td>Q-7B</td>
    <td>CoC</td>
    <td>1</td>
    <td>3</td>
    <td>1-8K</td>
  </tr>
  <tr>
    <td>w/ Doc</td>
    <td>Q-7B</td>
    <td>CoC</td>
    <td>1</td>
    <td>4</td>
    <td>1-8K</td>
  </tr>
  <tr>
    <td>LF-PO</td>
    <td>Q-7B</td>
    <td>CoC</td>
    <td>1</td>
    <td>2,3,4</td>
    <td>1-8K</td>
  </tr>
  <tr>
    <td>w/ CoC</td>
    <td>L8,L70,G</td>
    <td>CoC</td>
    <td>5</td>
    <td>6</td>
    <td>2K</td>
  </tr>
  <tr>
    <td>w/ GT</td>
    <td>L8,L70,G</td>
    <td>CoC</td>
    <td>5</td>
    <td>7</td>
    <td>2K</td>
  </tr>
  <tr>
    <td>w/ Doc</td>
    <td>L8,L70,G</td>
    <td>CoC</td>
    <td>5</td>
    <td>8</td>
    <td>2K</td>
  </tr>
  <tr>
    <td>LF-PO</td>
    <td>L8,L70,G</td>
    <td>CoC</td>
    <td>5</td>
    <td>6,7,8</td>
    <td>2K</td>
  </tr>
</table>

Table 1: Statistics of synthesized datasets for SFT and PO. We first synthesize large-scale reasoning chains and then refactor them to datasets, where the second stage does not require llm inference. Q-7B means Qwen2.5-7B-Instruct, L8 means Llama-3.1-8B-Instruct, L70 means Llama-3.1-70B-Instruct and G means GPT-4o mini. GT means Ground Truth, CoC means chain-of-citation, Doc means contextual documents, and LF means LONGFAITH. 1-8K includes {1K, 2K, 4K, 8K}.

LONGFAITH-PO Dataset. Towards training a downstream LLM to address three challenges above in long-context reasoning, we force the LLM to learn reasoning with high faithfulness while rejecting outputs of questionable faithfulness:

\[
I_{po} = \{\text{input} = (P_{\text{coc}}, Q, D), \\
\text{chosen} = O_c, \text{rejected} = O_r\}
\]

where \( O_r \) is a combination of (\( O_m, O_{\text{cot}}, O_{\text{kc}} \)), or a subset of them.
<table>
  <tr>
    <th rowspan="2">LLAMA-3.1-8B-INSTRUCT</th>
    <th colspan="4">MuSiQue</th>
    <th colspan="4">2Wiki</th>
    <th colspan="4">HotpotQA</th>
    <th colspan="4">Qasper(S)</th>
    <th colspan="4">MFQA-En(S)</th>
    <th colspan="4">MuSiQue(M)</th>
    <th colspan="4">2Wiki(M)</th>
    <th colspan="4">HotpotQA(M)</th>
  </tr>
  <tr>
    <th>Fl</th><th>SubEM</th><th>Fl</th><th>SubEM</th>
    <th>Fl</th><th>SubEM</th><th>Fl</th><th>SubEM</th>
    <th>Fl</th><th>SubEM</th><th>Fl</th><th>SubEM</th>
    <th>Fl</th><th>SubEM</th><th>Fl</th><th>SubEM</th>
    <th>Fl</th><th>SubEM</th><th>Fl</th><th>SubEM</th>
    <th>Fl</th><th>SubEM</th><th>Fl</th><th>SubEM</th>
  </tr>
  <tr>
    <td colspan="26"><b>Zero-Shot Prompting</b></td>
  </tr>
  <tr>
    <td>+ CoT</td><td>15.9</td><td>56.8</td><td>34.0</td><td>83.8</td><td>20.8</td><td>78.6</td><td>3.2</td><td>22.0</td><td>5.7</td><td>29.3</td><td>14.1</td><td>43.5</td><td>30.1</td><td>77.0</td><td>13.4</td><td>60.5</td>
  </tr>
  <tr>
    <td>+ CoC</td><td>25.8</td><td>64.2</td><td>43.6</td><td>86.2</td><td>32.7</td><td>76.6</td><td>4.6</td><td>26.0</td><td>7.0</td><td>32.7</td><td>11.8</td><td>41.0</td><td>28.1</td><td>79.5</td><td>19.9</td><td>58.0</td>
  </tr>
  <tr>
    <td colspan="26"><b>Supervised Fine-tuning</b></td>
  </tr>
  <tr>
    <td>+ LongAlpaca</td><td>21.6</td><td>50.2</td><td>47.8</td><td>80.4</td><td>32.7</td><td>76.6</td><td>5.7</td><td>25.0</td><td>5.8</td><td>30.7</td><td>8.5</td><td>48.5</td><td>25.4</td><td>77.0</td><td>12.5</td><td>61.0</td>
  </tr>
  <tr>
    <td>+ LongAlign</td><td>24.8</td><td>48.4</td><td>55.6</td><td>84.2</td><td>51.0</td><td>79.2</td><td>6.5</td><td>24.0</td><td>10.7</td><td>38.7</td><td>15.0</td><td>40.0</td><td>33.4</td><td>76.5</td><td>35.8</td><td>61.0</td>
  </tr>
  <tr>
    <td>+ MuSiQue-Attribute</td><td>13.9</td><td>19.2</td><td>23.9</td><td>49.6</td><td>20.2</td><td>37.2</td><td>10.0</td><td>11.5</td><td>8.3</td><td>12.0</td><td>15.2</td><td>26.5</td><td>21.2</td><td>55.0</td><td>25.6</td><td>41.0</td>
  </tr>
  <tr>
    <td>+ LongMIT</td><td>4.9</td><td>33.0</td><td>3.3</td><td>58.0</td><td>10.1</td><td>63.6</td><td>9.5</td><td>18.5</td><td>5.6</td><td>30.0</td><td>7.5</td><td>29.0</td><td>3.6</td><td>55.5</td><td>23.7</td><td>50.0</td>
  </tr>
  <tr>
    <td>+ LongReward-SFT</td><td>6.2</td><td>48.4</td><td>23.3</td><td>80.0</td><td>15.6</td><td>74.2</td><td>2.6</td><td>22.5</td><td>0.5</td><td>34.0</td><td>1.1</td><td>43.0</td><td>6.6</td><td>71.5</td><td>8.9</td><td>54.0</td>
  </tr>
  <tr>
    <td>+ SEALONG-SFT</td><td>31.3</td><td>64.6</td><td>55.8</td><td>89.2</td><td>59.4</td><td>83.0</td><td>14.5</td><td>26.0</td><td>18.6</td><td>31.3</td><td>24.1</td><td>59.5</td><td>34.1</td><td>84.5</td><td>37.3</td><td>69.0</td>
  </tr>
  <tr>
    <td>+ LONGFAITH-SFT</td><td>56.8</td><td>62.8</td><td>73.8</td><td>85.6</td><td>70.5</td><td>80.8</td><td>36.9</td><td>29.5</td><td>47.0</td><td>52.0</td><td>50.1</td><td>56.5</td><td>63.9</td><td>82.0</td><td>53.1</td><td>68.0</td>
  </tr>
  <tr>
    <td colspan="26"><b>Preference Optimization</b></td>
  </tr>
  <tr>
    <td>+ LONGREWARD-PO</td><td>3.3</td><td>46.0</td><td>14.3</td><td>76.6</td><td>8.9</td><td>71.2</td><td>1.6</td><td>21.0</td><td>0.1</td><td>32.7</td><td>0.0</td><td>37.5</td><td>4.4</td><td>67.0</td><td>3.3</td><td>53.0</td>
  </tr>
  <tr>
    <td>+ SEALONG-PO</td><td>30.2</td><td>60.4</td><td>50.1</td><td>89.4</td><td>58.3</td><td>83.4</td><td>17.1</td><td>28.0</td><td>20.1</td><td>32.0</td><td>18.1</td><td>53.3</td><td>34.0</td><td>86.0</td><td>40.2</td><td>69.5</td>
  </tr>
  <tr>
    <td>+ LONGFAITH-PO</td><td><b>60.5</b></td><td><b>66.4</b></td><td><b>68.0</b></td><td><b>85.0</b></td><td><b>65.4</b></td><td><b>81.2</b></td><td><b>38.1</b></td><td><b>30.5</b></td><td><b>46.7</b></td><td><b>32.0</b></td><td><b>50.2</b></td><td><b>52.0</b></td><td><b>73.7</b></td><td><b>83.5</b></td><td><b>55.6</b></td><td><b>67.5</b></td>
  </tr>
</table>

Table 2: Main experiment results on three multi-hop reasoning test sets and five long-context QA test sets from LongBench. The best results are in bold and second-best are underlined. (S) means single-doc QA task and (M) means multiple-docs QA task in LongBench. LONGFAITH-SFT and LONGFAITH-PO are synthesized by GPT-4o mini both in 2K size. To ensure fairness, we sample first 2K examples from baseline datasets.

4 Experiments

4.1 Implementation Details

Following previous studies, we leverage the training set of MuSiQue (Trivedi et al., 2022b), which is build on Wikipedia documents with supporting documents and correct answers. The officially retrieved 20 documents are provided and read only once in the input context in distractor setting. The statistics of training set is given in Tab. 6, covering 1K, 2K, 4K and 8K, where the balance of questions with different hops are considered. Following the pipeline we describe in Sec. 3, reasoning chains are samples to build LONGFAITH-SFT and LONGFAITH-PO. The statistics are presented in Tab. 1.

We conduct our experiments on a Linux server equipped with 4 A100-SXM4-40GB GPUs. For data synthesis of long-context reasoning instructions, we take Llama-3.1-8B-Instruct (Dubey et al., 2024), Qwen2.5-7B-Instruct (Yang et al., 2024), Llama-3.1-70B-Instruct and GPT-4o mini (Hurst et al., 2024) as generators and prompt LLMs to synthesize reasoning chains with vLLM (Kwon et al., 2023). We adopt the LoRA technique (Hu et al., 2021) for fine-tuning and ORPO technique (Hong et al., 2024) for preference optimization using the LLaMA-Factory framework (Zheng et al., 2024) to train Llama-3.1-8B-Instruct. Hyperparameters of post-training are given in App. F.

4.2 Evaluation Setup

Following prior work (Li et al., 2024b), we utilize three multi-hop reasoning datasets, including MuSiQue (Trivedi et al., 2022b), 2WikiMultiHopQA (Ho et al., 2020), and HotpotQA (Yang et al., 2018), evaluating in distractor-setting, where the officially retrieved 10 or 20 documents are provided and read only once in the input context. We adopt the test sets sampled by (Trivedi et al., 2022a), with 500 examples in each set. Furthermore, in line with previous studies (Chen et al., 2024; Zhang et al., 2024c; Li et al., 2024a), we as-

![Performance of Llama-3.1-8B-Instruct trained on different size of instructions synthesized by Qwen2.5-7B-Instruct from 1K to 8K, evaluated by EM and F1 metrics on multi-hop reasoning sets and LongBench.](page_1012_1042_420_420.png)

Figure 3: Performance of Llama-3.1-8B-Instruct trained on different size of instructions synthesized by Qwen2.5-7B-Instruct from 1K to 8K, evaluated by EM and F1 metrics on multi-hop reasoning sets and LongBench.
<table>
  <tr>
    <th rowspan="2">LLAMA-3.1-8B-INSTRUCT</th>
    <th colspan="4">MuSiQue</th>
    <th colspan="4">2WikiMultiHopQA</th>
    <th colspan="3">HotpotQA</th>
  </tr>
  <tr>
    <th>Overall</th>
    <th>2-Hop</th>
    <th>3-Hop</th>
    <th>4-Hop</th>
    <th>Overall</th>
    <th>2-Hop</th>
    <th>4-Hop</th>
    <th>Overall</th>
    <th>Bridge</th>
    <th>Comparison</th>
  </tr>
  <tr>
    <th colspan="11">Zero-Shot Prompting</th>
  </tr>
  <tr>
    <td>+ CoT</td>
    <td>11.0</td>
    <td>7.5</td>
    <td>16.2</td>
    <td>12.0</td>
    <td>29.0</td>
    <td>22.3</td>
    <td>54.3</td>
    <td>17.4</td>
    <td>17.5</td>
    <td>17.0</td>
  </tr>
  <tr>
    <td>+ CoC</td>
    <td>19.0</td>
    <td>16.1</td>
    <td>22.7</td>
    <td>20.7</td>
    <td>39.2</td>
    <td>31.4</td>
    <td>68.6</td>
    <td>30.4</td>
    <td>28.6</td>
    <td>38.6</td>
  </tr>
  <tr>
    <th colspan="11">Supervised Fine-tuning</th>
  </tr>
  <tr>
    <td>+ LONGFAITH-SFT</td>
    <td>40.6</td>
    <td>44.1</td>
    <td>37.7</td>
    <td>35.9</td>
    <td>55.4</td>
    <td>51.1</td>
    <td>71.4</td>
    <td>53.6</td>
    <td>57.0</td>
    <td>37.5</td>
  </tr>
  <tr>
    <td>w/o CoC</td>
    <td>40.2</td>
    <td>41.7</td>
    <td>39.6</td>
    <td>37.0</td>
    <td>51.8</td>
    <td>48.9</td>
    <td>62.9</td>
    <td>52.0</td>
    <td>56.6</td>
    <td>30.7</td>
  </tr>
  <tr>
    <td>w/o GT</td>
    <td>30.4</td>
    <td>31.9</td>
    <td>28.6</td>
    <td>29.3</td>
    <td>55.8</td>
    <td>49.6</td>
    <td>79.0</td>
    <td>56.6</td>
    <td>54.9</td>
    <td><b>64.8</b></td>
  </tr>
  <tr>
    <td>w/o Doc</td>
    <td>20.0</td>
    <td>23.6</td>
    <td>18.2</td>
    <td>13.0</td>
    <td>55.8</td>
    <td>47.1</td>
    <td><b>88.6</b></td>
    <td>47.4</td>
    <td>45.9</td>
    <td>54.5</td>
  </tr>
  <tr>
    <th colspan="11">Preference Optimization</th>
  </tr>
  <tr>
    <td>w/ GT-PO</td>
    <td>44.0</td>
    <td>45.7</td>
    <td>42.9</td>
    <td>41.3</td>
    <td>56.0</td>
    <td>50.4</td>
    <td>77.1</td>
    <td>54.4</td>
    <td>58.3</td>
    <td>36.4</td>
  </tr>
  <tr>
    <td>w/ CoC-PO</td>
    <td>43.6</td>
    <td>44.5</td>
    <td>44.8</td>
    <td>39.1</td>
    <td>53.2</td>
    <td>48.6</td>
    <td>70.5</td>
    <td>56.2</td>
    <td>59.2</td>
    <td>42.0</td>
  </tr>
  <tr>
    <td>w/ Doc-PO</td>
    <td>41.4</td>
    <td>42.5</td>
    <td>40.3</td>
    <td>40.2</td>
    <td>56.0</td>
    <td>52.7</td>
    <td>68.6</td>
    <td>56.4</td>
    <td>59.5</td>
    <td>42.0</td>
  </tr>
  <tr>
    <td>+ LONGFAITH-PO</td>
    <td><b>46.6</b></td>
    <td><b>47.2</b></td>
    <td><b>48.1</b></td>
    <td><b>42.4</b></td>
    <td><b>59.0</b></td>
    <td><b>55.9</b></td>
    <td><b>70.5</b></td>
    <td><b>58.6</b></td>
    <td><b>59.7</b></td>
    <td>53.4</td>
  </tr>
</table>

Table 3: Main experiment results on three long-context multi-hop reasoning datasets using the Exact-Match(EM) metric. The best results are in bold. The training set has 2K samples for both SFT and PO, synthesized by Qwen2.5-7B-Instruct. Detail statistics of synthetics datasets are presented in Tab. 1.

sess the performance on LongBench (Bai et al., 2023), which includes two test sets for single-doc QA including Qasper (Dasigi et al., 2021) and MultiFieldQA-EN (Bai et al., 2023), as well as three test sets for multi-docs QA tasks including HotpotQA, 2WikiMultiHopQA, and MuSiQue. Notably, although there is an overlap in multi-hop reasoning tasks, the LongBench version further extends the lengths of document text. To apply CoC prompting, single document is split into 20 even paragraphs with order. The statistics of datasets are listed in Tab. 7.

To ensure fairness, we use Substring Exact-Match (SubEM) (Yen et al., 2024; Li et al., 2024a) metric in main experiments, in case that models trained on baseline datasets are not good at instructions following to summarize the answer with "The answer is", and SubEM goes through the whole response to check whether the answer is in. Furthermore, following previous work (Choi et al., 2018; Zhang et al., 2024a; Li et al., 2024b), we use EM metric and F1 scores for the trimmed part after "The answer is" for evaluation in main experiments and ablation studies. Comparing with LLM-as-a-Judge (Bai et al., 2024a; Chen et al., 2024; Zhang et al., 2024c) using strong API models like GPT-4o, the rule-based metrics cost much lower.

4.3 Baselines

We compete LONGFAITH-SFT and LONGFAITH-PO with datasets proposed in previous studies, including LONGALPACA (Chen et al., 2023b), LONGALIGN (Bai et al., 2024a), MuSiQUE-ATTRIBUTE (Li et al., 2024b), LONGMIT (Chen et al., 2024), LONGREWARD-SFT (Zhang et al., 2024c), SEALONG-SFT (Li et al., 2024a) for supervised fine-tuning, and LONGREWARD-PO and SEALONG-PO for preference optimization. All of them aim at enhancing the performance of LLMs on long-context understanding, reasoning, and QA tasks. To ensure fairness, we keep the training setting consistent with App. F and truncate the size of training samples to a maximum of 2K.

4.4 Main Results

LONGFAITH Outperforms Previous Datasets. Following previous work and to ensure a fair comparison, we evaluate the performance of LONGFAITH on multi-hop reasoning test sets (Trivedi et al., 2022b; Ho et al., 2020; Yang et al., 2018) and LongBench (Bai et al., 2023), comparing it against baseline methods, including zero-shot prompting with Llama-3.1-8B-Instructand models trained on synthetic datasets from previous works. As shown in Tab. 2, LONGFAITH outperforms baseline datasets on most test sets. The performance of the model trained on LONGFAITH-PO surpasses that trained on LONGFAITH-SFT. This aligns with our expectations: compared to directly using positive samples for supervised fine-tuning, incorporating rejected samples to provide more fine-grained faithfulness preferences for optimization leads to better improvements in long-context reasoning and QA capabilities. We observe that some synthetic
Figure 4: Scatter plot with a linear regression line fitting the relationship between QA - EM and Attribution - F1 metrics on three long-context multi-hop reasoning test sets. A point refers to the performance of a model trained with a specific size between 1K to 8K by SFT or PO.

instruction sets degrade performance compared to native Llama-3.1-8B-Instruct. This proves that datasets with questionable faithfulness are even harmful to long-context reasoning ability of LLMs.

LONGFAITH Arrives at the Correct Answer without Redundant Exploration. We find that on 2WikiMultiHopQA, HotpotQA, and part of tasks in LongBench, SEALONG achieves a slight advantage in the SubEM metric against LONG-FAITH, but fails in F1 scores. We investigate the length of response and present in Tab. 9. It turns out that the LLM trained on SEALONG conducts redundant exploration in response, producing more noisy content related to the answer, but actually arrives at a wrong answer, which means SubEM metric is easily to be "hacked". In contradiction, F1 scores requires to truncate the part after "The answer is", which demonstrates that a model trained on LONGFAITH datasets can arrive at the correct answer without redundant exploration and achieve a high score in a more strict metric. A case study is shown in Fig. 11 in Appendix.

Generalization. Based on statistics from Tab. 8, the main experiment demonstrates that LONG-FAITH uses instructions with shorter context as input compared to baseline methods, reducing training costs while generalizing to LongBench tasks that require processing an average of 24K-70K tokens as input. This further highlights the generalization ability of our pipeline.

4.5 Analysis

Exploration on Different Perspective of Faithfulness. To validate the specific impact of different dimensions of faithfulness, we fine-tune models using negative samples as output and optimize using preference datasets that reject only a subset of negative samples. The statistics of the constructed datasets are shown in Tab. 1. Since each task in LongBench contains no more than 200 questions, performance evaluations can be prone to errors, so we chose to test on multi-hop reasoning datasets. Experimental results are shown in 3. The models trained with LONGFAITH-SFT and LONGFAITH-PO achieved high performance respectively in SFT and PO especially in F1 scores, as expected.

However, we note that in 4-Hop part of 2Wiki and comparison part of HotpotQA, LONGFAITH-SFT w/o CoC and w/o GT demonstrated better performance. Analysis reveals that for the question "Do both films, Cuban Colony and Prathartha, have directors from the same country?", as the training set MuSiQue used specific entities as answers, the model responds "Both directors are from the same country, which is India. The answer is India.". Actually, the correct answer is "yes." Model trained on LONGFAITH w/o GT and LONGFAITH w/o Doc performed better with more exploration, but also lost overall performance due to hallucinations. Models trained on all PO datasets outperformed those trained using only positive samples for SFT, demonstrating the performance improvement brought by each fine-grained, credible preference. Finally, models trained on LONGFAITH-PO, which integrates three dimensions of faithfulness, achieved the best overall performance.

Figure 5: Visualization of F1 scores in Tab. 2.
<table>
  <tr>
    <th rowspan="2">LLAMA-3.1-8B-INSTRUCT + LONGFAITH</th>
    <th colspan="2">MuSiQue</th>
    <th colspan="2">2WikiMHQA</th>
    <th colspan="2">HotpotQA</th>
    <th colspan="2">LongBench(S)</th>
    <th colspan="2">LongBench(M)</th>
  </tr>
  <tr>
    <th>EM</th><th>F1</th>
    <th>EM</th><th>F1</th>
    <th>EM</th><th>F1</th>
    <th>EM</th><th>F1</th>
    <th>EM</th><th>F1</th>
  </tr>
  <tr>
    <th colspan="11">Supervised Fine-tuning</th>
  </tr>
  <tr>
    <td>w/ Llama-3.1-8B-Instruct</td><td>35.4</td><td>48.4</td><td>59.4</td><td>69.5</td><td>54.6</td><td>67.7</td><td>9.9</td><td>29.0</td><td>42.2</td><td>48.4</td>
  </tr>
  <tr>
    <td>w/ Qwen2.5-7B-Instruct</td><td>40.6</td><td>54.0</td><td>55.4</td><td>65.0</td><td>53.6</td><td>69.5</td><td>14.1</td><td>38.4</td><td>43.0</td><td>53.8</td>
  </tr>
  <tr>
    <td>w/ Llama-3.1-70B-Instruct</td><td><b>44.8</b></td><td><b>58.1</b></td><td>54.0</td><td>64.4</td><td>54.4</td><td>69.5</td><td>16.0</td><td>41.0</td><td>44.8</td><td><b>56.7</b></td>
  </tr>
  <tr>
    <td>w/ GPT-4o mini</td><td>41.6</td><td>56.8</td><td><b>64.6</b></td><td><b>73.8</b></td><td><b>55.4</b></td><td><b>70.5</b></td><td><b>16.9</b></td><td><b>42.0</b></td><td><b>47.2</b></td><td>55.7</td>
  </tr>
  <tr>
    <th colspan="11">Preference Optimization</th>
  </tr>
  <tr>
    <td>w/ Llama-3.1-8B-Instruct</td><td>41.2</td><td>53.2</td><td>57.4</td><td>67.0</td><td>55.6</td><td>68.5</td><td>14.7</td><td>36.9</td><td>44.0</td><td>55.3</td>
  </tr>
  <tr>
    <td>w/ Qwen2.5-7B-Instruct</td><td>46.6</td><td>59.2</td><td>59.0</td><td>67.4</td><td><b>58.6</b></td><td><b>72.8</b></td><td>15.4</td><td>35.3</td><td>44.8</td><td>55.6</td>
  </tr>
  <tr>
    <td>w/ Llama-3.1-70B-Instruct</td><td><b>50.4</b></td><td><b>63.2</b></td><td>52.8</td><td>62.7</td><td>57.2</td><td>71.0</td><td><b>16.4</b></td><td>40.0</td><td><b>48.3</b></td><td>59.4</td>
  </tr>
  <tr>
    <td>w/ GPT-4o mini</td><td>48.4</td><td>60.5</td><td><b>59.8</b></td><td><b>68.0</b></td><td>49.8</td><td>65.4</td><td>15.9</td><td><b>42.4</b></td><td>45.0</td><td><b>59.8</b></td>
  </tr>
</table>

Table 4: Ablation study on various LLMs for synthesizing LONGFAITH-SFT and LONGFAITH-PO in the size of 2K. The base model for training and testing is Llama-3.1-8B-Instruct. (S) and (M) refer to Single-doc QA and Multi-docs QA in LongBench.

Scalability and Performance Gains. We explore the scaling-up potential of LONGFAITH on multi-hop reasoning test sets and LongBench. As presented in Tab. 7, we train Llama-3.1-8B-Instruct using LONGFAITH-SFT and LONGFAITH-PO synthesized by Qwen2.5-7B-Instruct across four dataset sizes, ranging from 1K to 8K. According to the performance trend in Fig. 3, LONGFAITH generally exhibits scaling-up potential, indicating that expanding the training dataset can further enhance performance. Moreover, LONGFAITH-PO, which incorporates fine-grained preference optimization, demonstrates a more stable upward trend compared to LONGFAITH-SFT, particularly in LongBench tasks. This result validates the robustness of the LONGFAITH pipeline.

Attribution-Based Reasoning Leads to Higher Performance. Utilizing CoC prompting for reasoning with attributions not only outperforms CoT in performance, as it presents in Tab. 3, but also provides greater interpretability and faithfulness as shown in Fig. 2. We use Attribution F1 as a metric to quantify the model’s attribution capability using annotated supporting facts. Under CoC prompting, we analyze the references within reasoning chains, matching them against supporting facts like [1], [2], etc., and compute F1 scores based on recall and precision. We evaluate the attribution capability and overall performance of Llama-3.1-8B-Instruct trained on LONGFAITH-SFT and LONGFAITH-PO across four sizes and visualize the results in a scatter plot. The findings in 4 demonstrate a strong positive correlation between attribution capability and model performance, further validating the effectiveness of the LONGFAITH pipeline.

<table>
  <tr>
    <th rowspan="2">LLAMA-3.1-8B</th>
    <th colspan="2">LongBench(S)</th>
    <th colspan="2">LongBench(M)</th>
  </tr>
  <tr>
    <th>EM</th><th>F1</th>
    <th>EM</th><th>F1</th>
  </tr>
  <tr>
    <td>w/ 10 docs</td><td>11.2</td><td>31.7</td><td>39.7</td><td>52.2</td>
  </tr>
  <tr>
    <td>w/ 20 docs</td><td>15.4</td><td>35.3</td><td>44.8</td><td>55.6</td>
  </tr>
  <tr>
    <td>w/ 30 docs</td><td>16.0</td><td>36.0</td><td>46.1</td><td>56.7</td>
  </tr>
  <tr>
    <td>w/ 40 docs</td><td>16.9</td><td>37.8</td><td>47.2</td><td>58.1</td>
  </tr>
</table>

Table 5: Effectiveness of LONGFAITH on LongBench as context length increases.

Impact of LLM Selection for Synthesis. We experimented with different LLMs for synthesis, including smaller open-source LLMs such as Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, and larger open-source models like Llama-3.1-70B-Instruct, as well as a closed-source API model, GPT-4o mini, to synthesize LONGFAITH-SFT and LONGFAITH-PO for training Llama-3.1-8B-Instruct. The performance test results are presented in Tab.4. Using stronger closed-source API models to synthesize LONGFAITH-SFT led to a stronger performance boost, which aligns with intuition and previous work (Chen et al., 2024). However, an interesting finding is that the LONGFAITH-PO synthesized with different base LLMs did not show significant performance differences in preference optimization. Even smaller model like Qwen2.5-7B-Instruct, are able to synthesize high-quality reasoning chains, with performance on some datasets matching or even surpassing GPT-4o mini. This highlights the robustness of the LONGFAITH pipeline, which is capable of modeling fine-grained preferences to synthesize high-quality instructions.

Effectiveness of LONGFAITH as Context Length Increases. We further investigate the impact of increasing context length on the performance of
models trained with the LONGFAITH pipeline. Specifically, we manipulate the number of documents included in the training set MuSiQue to simulate varying context lengths during training. By incrementally increasing the number of documents from 10 to 40, we assess how the model’s reasoning ability scales when exposed to longer input contexts. The results, as reported in Tab. 5, show a consistent improvement in both EM and F1 scores across both LongBench(S) and Long-Bench(M) as the context length increases. This indicates that LONGFAITH effectively leverages additional contextual information, enhancing the model’s comprehension and reasoning capabilities. These findings validate the scalability of the LONGFAITH framework in handling long-context scenarios, highlighting its potential for applications that require deep reasoning over extensive inputs.

5 Conclusion

This paper addresses the challenge of questionable faithfulness in data synthesis approaches for long-context LLMs. We propose LONGFAITH, a novel pipeline synthesizing faithful long-context reasoning datasets through ground truth integration and citation-based reasoning prompts. Experiments demonstrate its effectiveness, with ablation studies confirming the adaptability of the LONGFAITH-SFT and LONGFAITH-PO datasets across diverse long-context scenarios.

Limitations

While LONGFAITH demonstrates significant improvements in long-context reasoning tasks, its scalability and generalization to other LLMs remain an open question. Our experiments focused on a single model, and thus, the performance of LONGFAITH on other general-purpose LLMs still needs further validation. Additionally, while the synthesized instruction sets with lengths of approximately 10,000 tokens successfully generalized to long-context reasoning tasks, future work will explore the extension of LONGFAITH to generate instructions with even longer contexts and evaluate the impact on model performance. Finally, LONGFAITH currently concentrates on reasoning tasks, and we plan to explore its generalization to other tasks such as summarization, dialogue generation, and others, to assess its broader applicability.

References

Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. 2024. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740.

Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian-Guang Lou. 2024. Make your llm fully utilize the context. arXiv preprint arXiv:2404.16811.

Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. 2024a. Longalign: A recipe for long context alignment of large language models. arXiv preprint arXiv:2401.18058.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508.

Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, et al. 2024b. Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.15204.

Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.

Moshe Berchansky, Daniel Fleischer, Moshe Wasserblat, and Peter Izsak. 2024. Cotar: Chain-of-thought attribution reasoning with multi-level granularity. arXiv preprint arXiv:2404.10513.

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023a. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595.

Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2023b. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307.

Zhi Chen, Qiguang Chen, Libo Qin, Qipeng Guo, Haijun Lv, Yicheng Zou, Wanxiang Che, Hang Yan, Kai Chen, and Dahua Lin. 2024. What are the essential factors in crafting effective long context multi-hop instruction datasets? insights and best practices. arXiv preprint arXiv:2409.01893.

Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. Quac: Question answering in context. arXiv preprint arXiv:1808.07036.

Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. arXiv preprint arXiv:2105.03011.
Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. 2023. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02486.

Yiran Ding, Li Lyna Zhang, Chengrudong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. Longrope: Extending llm context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. International Conference on Machine Learning.

Constanza Fierro, Reinald Kim Amplayo, Fantine Huot, Nicola De Cao, Joshua Maynez, Shashi Narayan, and Mirella Lapata. 2024. Learning to plan and generate text with citations. arXiv preprint arXiv:2404.03381.

Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. 2024. Data engineering for scaling language models to 128k context. arXiv preprint arXiv:2402.10171.

Tianyu Gao, Alexander Wetting, Howard Yen, and Danqi Chen. 2024. How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660.

Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. arXiv preprint arXiv:2305.14627.

Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594.

Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. 2024. Lm-infinite: Zero-shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3991–4008.

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060.

Jiwoo Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170–11189.

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shan-tanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

Lei Huang, Xiaocheng Feng, Weitao Ma, Yuxuan Gu, Weihong Zhong, Xiachong Feng, Weijiang Yu, Weihua Peng, Duyu Tang, Dandan Tu, et al. 2024. Learning fine-grained grounded citations for attributed large language models. arXiv preprint arXiv:2408.04568.

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, Al Os-trow, Akila Welibinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Xuhui Jiang, Shengjie Ma, Chengjin Xu, Cehao Yang, Liyu Zhang, and Jian Guo. 2025. Synthesize-on-graph: Knowledgeable synthetic data generation for continue pre-training of large language models. arXiv preprint arXiv:2505.00979.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.

Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint arXiv:2402.14848.

Dongfang Li, Zetian Sun, Xinshuo Hu, Zhenyu Liu, Ziyang Chen, Baotian Hu, Aigou Wu, and Min Zhang. 2023. A survey of large language models attribution. arXiv preprint arXiv:2311.03731.

Siheng Li, Cheng Yang, Zesen Cheng, Lemao Liu, Mo Yu, Yujiu Yang, and Wai Lam. 2024a. Large language models can self-improve in long-context reasoning. arXiv preprint arXiv:2411.08147.

Yanyang Li, Shuo Liang, Michael R Lyu, and Liwei Wang. 2024b. Making long-context language models better multi-hop reasoners. arXiv preprint arXiv:2408.03246.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.

Shengjie Ma, Chengjin Xu, Xuhui Jiang, Muzhi Li, Huaren Qu, Cehao Yang, Jiaxin Mao, and Jian Guo. 2024. Think-on-graph 2.0: Deep and faithful large language model reasoning with knowledge-guided retrieval augmented generation. arXiv preprint arXiv:2407.10805.

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shipplee. 2023. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.

Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210–31227. PMLR.

Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhao Wang, Chen Lin, Yeyun Gong, Heung-Yeung Shum, and Jian Guo. 2023. Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph. arXiv preprint arXiv:2307.07697.

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022a. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprint arXiv:2212.10509.

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022b. Musique: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alissa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Siye Wu, Jian Xie, Jiangjie Chen, Tinghui Zhu, Kai Zhang, and Yanghua Xiao. 2024. How easily do irrelevant inputs skew the responses of large language models? arXiv preprint arXiv:2404.03302.

Wenhan Xiong, Jingyu Liu, Igor Molyboha, Hejia Zhang, Prajiwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. 2023. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039.

Rongwu Xu, Zehan Qi, Zhijiang Guo, Cuxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. 2024. Knowledge conflicts for llms: A survey. arXiv preprint arXiv:2403.08319.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600.

Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. 2024. Helmet: How to evaluate long-context language models effectively and thoroughly. arXiv preprint arXiv:2410.02694.

Xiang Yue, Boshi Wang, Ziru Chen, Kai Zhang, Yu Su, and Huan Sun. 2023. Automatic evaluation of attribution by large language models. arXiv preprint arXiv:2305.06311.

Jiahao Zhang, Haiyang Zhang, Dongmei Zhang, Liu Yong, and Shen Huang. 2024a. End-to-end beam retrieval for multi-hop question answering. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1718–1731.

Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, et al. 2024b. Longcite: Enabling llms to generate fine-grained citations in long-context qa. arXiv preprint arXiv:2409.02897.

Jiajie Zhang, Zhongni Hou, Xin Lv, Shulin Cao, Zhenyu Hou, Yilin Niu, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. 2024c. Longeward: Improving long-context large language models with ai feedback. arXiv preprint arXiv:2410.21252.

Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. 2024d. \( \infty \)Bench: Extending long context evaluation beyond 100K tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15262–15277.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023. Siren’s song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219.

Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372.

Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wen-hao Wu, Furu Wei, and Sujian Li. 2023. Pose: Efficient context window extension of llms via positional skip-wise training. arXiv preprint arXiv:2309.10400.
A  An Example of Synthesized LONGFAITH-SFT and LONGFAITH-PO Datasets

Construction of LongFaith-SFT and LongFaith-PO using Chosen Sample and Rejected Samples

Instruction
You are provided with documents, a complex logical reasoning question, and the correct answer.
You must refer to the documents to perform step-by-step logical reasoning and reach the correct answer.
Each reasoning step must be on a separate line, ending with a newline character.
Cite the document properly during reasoning, e.g., [1], [2], etc.
End your reasoning with: 'The answer is' followed by the correct answer.

DOCUMENTS:
[1] Muammar Gaddafi: From childhood, Gaddafi was aware of the involvement of European colonialists in Libya……
[2] Muammar Gaddafi: On 16 April 1973, Gaddafi proclaimed the start of a "Popular Revolution" in a Zuwarah speech……
[9] Bogotá: Bogotá / bɔɡoʊtə /, / bɔɡa tə /, boo - tə. Spanish pronunciation: (boyo ta) (listen)……
[10] Alange: Alange is a corregimiento in Alange District, Chiriquí Province, Panama. It is the seat of Alange District……
[20] All Saints Church, Lockerbie: All Saints Church is in Ashgrove Terrace, Lockerbie, Dumfries and Galloway, Scotland……

QUESTION:
When was the country containing Alanje colonized by the country housing the All Saints Church in the town known for the bombing by Gaddafi's Libya?

Chosen Output (Sample: Highest Faithfulness)
Step 1: Identify the country where Alanje is located.
From Document [10], Alanje is in the Alange District, Chiriquí Province, Panama. Thus, Alanje is located in Panama.

Step 2: Identify the country where All Saints Church is located.
……
The answer is 1698.

Rejected Output (Sample: Misinformation due to Lack of Verification)
The All Saints Church is located in Lockerbie, Scotland, which is part of the United Kingdom.

The UK was involved in colonizing various territories, including Panama, where Alanje is located. [10]
……
The answer is 1903.

Rejected Output (Sample: Reasoning without Attribution)
Step 1: Identify the country containing Alanje.
From the document on Alanje, it is stated that Alanje is located in Panama.

Step 2: Identify the country housing the All Saints Church in Lockerbie.
……
The answer is 1698.

Rejected Output (Sample: Potential Knowledge Conflicts)
Alanje is a location, so we need to determine which country contains Alanje. Alanje is located in Panama.
……
However, Panama was not colonized by the United Kingdom; Panama was colonized by Spain.
So, we need to determine the connection between the United Kingdom and Spain regarding the colonization of Panama.
……
The answer is 1698.

Figure 6: An Example of synthesized LONGFAITH-SFT and LONGFAITH-PO datasets.
B Prompt Templates

We present the prompt templates that are used to synthesize the datasets. The core prompt template that generates long-context reasoning chains guided by ground truth using chain-of-citation is shown in Fig. 7. The samples are used in LONGFAITH-SFT dataset and are chosen as positive in LONGFAITH-PO dataset, since they are of the highest faithfulness. The other three prompt templates synthesize rejected samples for LONGFAITH-PO dataset, corresponding to (1) Misinformation due to lack of verification (Fig. 8), (2) Reasoning without attribution (Fig. 9), and (3) Potential knowledge conflicts (Fig. 10).

Synthesis of Long-Context Reasoning Chains Guided by Ground Truth Using Chain-of-Citation

Prompt for Synthesis
You are provided with documents, a complex logical reasoning question, and the correct answer.
You must refer to the documents to perform step-by-step logical reasoning and reach the correct answer.
Each reasoning step must be on a separate line, ending with a newline character.
Cite the document properly during reasoning, e.g., [1], [2], etc.
End your reasoning with 'The answer is' followed by the correct answer.

DOCUMENTS:
[10] Alanje: Alanje is a corregimiento in Alajue District, Chiriqui Province, Panama………
[14] British Empire: In 1695, the Scottish Parliament granted a charter to the Company of Scotland……
[16] Muammar Gaddafi: In 1977, Gaddafi dissolved the Republic and created a new socialist state, ……
[20] All Saints Church, Lockerbie: All Saints Church is in Ashgrove Terrace, Lockerbie, ……

QUESTION:
When was the country containing Alanje colonized by the country housing the All Saints Church in the town known for the bombing by Gaddafi’s Libya?

ANSWER:
1698

Synthesized Reasoning Chains
Step 1: Identify the country where Alanje is located.
From Document [10], Alanje is in the Alajue District, Chiriqui Province, Panama. Thus, Alanje is located in Panama.

Step 2: Identify the country where All Saints Church is located.
From Document [20], All Saints Church is in Lockerbie, Dumfries and Galloway, Scotland. Thus, All Saints Church is in Scotland.

Step 3: Determine the historical event that connects Panama and Scotland.
From Document [14], in 1698, the Company of Scotland established a settlement on the isthmus of Panama. This settlement was part of the Darien scheme, a Scottish colonial venture.

Step 4: Confirm the timeline of the colonization.
From Document [14], the Scottish settlement on the isthmus of Panama occurred in 1698.

Step 5: Verify the connection to Gaddafi’s Libya and Lockerbie.
From Document [16], Gaddafi’s Libya was responsible for the Lockerbie bombing. This connects the country housing All Saints Church (Scotland) to the logical context of the question.

Step 6: Synthesize the information.
The country containing Alanje (Panama) was colonized by the country housing All Saints Church (Scotland) in 1698.

The answer is 1698.

Chosen as Positive Sample: Highest Faithfulness

Figure 7: An example of synthesized chosen reasoning chain. The current reasoning chain must arrive at a correct answer, and reasoning with proper citation proposes more faithfulness and interpretability. Therefore, LONGFAITH will choose it in supervised fine-tuning and preference optimization as positive sample.
Synthesis of Long-Context Reasoning Chains Using Chain-of-Citation

Prompt for Synthesis
You are provided with documents, a complex logical reasoning question, and the correct answer.
You must refer to the documents to perform step-by-step logical reasoning and reach the correct answer.
Each reasoning step must be on a separate line, ending with a newline character.
Cite the document properly during reasoning, e.g.: [1], [2], etc.
End your reasoning with 'The answer is' followed by the correct answer.

DOCUMENTS:
[1] Muammar Gaddafi: From childhood, Gaddafi was aware of the involvement of European colonialists in Libya……
[2] Muammar Gaddafi: On 16 April 1973, Gaddafi proclaimed the start of a “Popular Revolution” in a Zuwarah speech……
……
[9] Bogotá: Bogotá (/ˈboʊɡətɑː/; /bɒɡoʊˈtaː/; boʊ - tɑː; Spanish pronunciation: (boyoˈta) (listen))……
[10] Alanje: Alanje is a corregimiento in Alanje District, Chiriquí Province, Panama. It is the seat of Alanje District……
……
[20] All Saints Church, Lockerbie: All Saints Church is in Ashgrove Terrace, Lockerbie, Dumfries and Galloway, Scotland……

QUESTION:
When was the country containing Alanje colonized by the country housing the All Saints Church in the town known for the bombing by Gaddafi’s Libya?

Synthesized Reasoning Chains
The All Saints Church is located in Lockerbie, Scotland, which is part of the United Kingdom.

The UK was involved in colonizing various territories, including Panama, where Alanje is located. [10]

The British Empire’s colonization of Panama through the Darien scheme began in 1698, although this attempt was unsuccessful. [14]

The formal British colonization of Panama occurred later, after the separation of Panama from Colombia in 1903, when the UK recognized Panama’s independence.

The answer is 1903.

Chosen as Rejected Sample: Misinformation due to Lack of Verification

Figure 8: An example of synthesized rejected reasoning chain. Misinformation due to lack of verification will cause more hallucination if we use current reasoning chain to fine-tune a LLM. Therefore, LONGFAITH will reject it in preference optimization.
Synthesis of Long-Context Reasoning Chains Guided by Ground Truth Using Chain-of-Thought

Prompt for Synthesis
You are provided with documents, a complex logical reasoning question, and the correct answer.
You must refer to the documents to perform step-by-step logical reasoning and reach the correct answer.
Each reasoning step must be on a separate line, ending with a newline character.
End your reasoning with 'The answer is' followed by the correct answer.

DOCUMENTS.
Alanje: Alanje is a corregimiento in Alajje District, Chiriqui Province, Panama………
British Empire: In 1695, the Scottish Parliament granted a charter to the Company of Scotland………
Muammar Gaddafi: In 1977, Gaddafi dissolved the Republic and created a new socialist state, …….
All Saints Church, Lockerbie: All Saints Church is in Ashgrove Terrace, Lockerbie, ……

QUESTION:
When was the country containing Alanje colonized by the country housing the All Saints Church in the town known for the bombing by Gaddafi’s Libya?

ANSWER:
1698

Synthesized Reasoning Chains
Step 1: Identify the country containing Alanje.
From the document on Alanje, it is stated that Alanje is located in Panama.

Step 2: Identify the country housing the All Saints Church in Lockerbie
The document on All Saints Church specifies that it is in Lockerbie, Scotland.

Step 3: Determine the relationship between Panama and Scotland.
The document on the British Empire mentions that in 1698, the Scottish Parliament established a settlement on the isthmus of Panama.

Step 4: Verify if the settlement was during a colonization attempt.
The British Empire document states that the Scottish settlement on the isthmus of Panama was part of the Darien scheme, an effort by Scotland to establish its own overseas empire, which qualifies as a colonization attempt.

Step 5: Confirm the year of colonization.
The British Empire document explicitly states that the settlement in Panama by Scotland occurred in 1698.

The answer is 1698.

Chosen as Rejected Sample: Reasoning without Attribution

Figure 9: An example of synthesized rejected reasoning chain. As it mentioned in previous work, lack of attribution will lead to much more interpretability and faithfulness, and response with citation is encouraged. Therefore, LONGFAITH will reject it in preference optimization.
Synthesis of Long-Context Reasoning Chains Guided by Ground Truth Without Documents Using CoT

Prompt for Synthesis
You are provided with a complex logical reasoning question and the correct answer. You must perform step-by-step logical reasoning and reach the correct answer. Each reasoning step must be on a separate line, ending with a newline character. End your reasoning with 'The answer is' followed by the correct answer.

QUESTION:
When was the country containing Alanje colonized by the country housing the All Saints Church in the town known for the bombing by Gaddafi's Libya?

ANSWER:
1698

Synthesized Reasoning Chains
Let's break this down step by step:

Alanje is a location, so we need to determine which country contains Alanje. Alanje is located in Panama.

Next, we need to identify the country housing the All Saints Church in the town known for the bombing by Gaddafi's Libya.

The bombing by Gaddafi's Libya refers to an event in 1986 in Loughborough, England.

Loughborough, England, is in the United Kingdom, and it indeed has an All Saints Church.

Now we know we are looking for when Panama (containing Alanje) was colonized by the United Kingdom (housing the All Saints Church in Loughborough).

However, Panama was not colonized by the United Kingdom; Panama was colonized by Spain.

So, we need to determine the connection between the United Kingdom and Spain regarding the colonization of Panama.

The United Kingdom is not directly related to the colonization of Panama, so the question appears to involve a broader historical context.

Spain colonized Panama in 1501, but colonization efforts and territorial control fluctuated.

The specific colonization of Alanje, a town in Panama, is tied to the Spanish colonization of the region.

The date 1698 aligns with significant historical colonization activities in the Americas.

Thus, the colonization being referred to in the question likely took place in 1698.

The answer is 1698.

Chosen as Rejected Sample: Potential Knowledge Conflicts

Figure 10: An example of synthesized rejected reasoning chain. Information related to "Panama was colonized by Spain" is not mentioned in documents provided, which will cause knowledge conflicts if using current reasoning chain for fine-tuning. Therefore, LONGFAITH will reject it in preference optimization.
C  Case Study of Metric "Hack" on SubEM comparing previous study with LONGFAITH

Who was the first elected president of the country where, along with the nation where Ágnes Konkoly is from, many expelled French Jews relocated to?

[1] Middle Ages: Jewish communities in England expelled from England in 1290 and France in 1306. Although some……
[2] Konkoly, Ágnes Konkoly (born 22 July 1987) is a Hungarian model, wedding planner and beauty pageant……
Correct Answer: Gabriel Narutowicz

SeaLong
To find the answer, we need to identify the country where Ágnes Konkoly is from and the country where many expelled French Jews relocated to.

Step 1: Identify the country where Ágnes Konkoly is from.
Ágnes Konkoly is a Hungarian model, wedding planner, and beauty pageant titleholder, so she is from Hungary.

Step 2: Identify the country where many expelled French Jews relocated to.
From the given documents, we can see that many expelled French Jews relocated to Poland and Hungary.

Step 3: Find the first elected president of the country where Ágnes Konkoly is from (Hungary).
The first president of Hungary was Lajos Kossuth, but he was not elected president. The first freely elected president of Hungary was János Áder.

However, the problem asks for the first elected president of the country where Ágnes Konkoly is from, which is Hungary. The first freely elected president of Hungary was János Áder, but the first president of the country was Lajos Kossuth, who was not elected.

Step 4: Determine the first elected president of the country where many expelled French Jews relocated to (Poland).
The first president of Poland was Gabriel Narutowicz, who was elected in 1922.

However, the problem asks for the first elected president of the country where Ágnes Konkoly is from, which is Hungary. We already found that the first freely elected president of Hungary was János Áder.

The answer is János Áder.

LongFaith
To determine the first elected president of the country where many expelled French Jews relocated to, we need to analyze the information provided in the documents.

First, we identify the country where Ágnes Konkoly is from. Ágnes Konkoly is a Hungarian model, and Hungary is her country of origin [18].

Next, we look for information regarding the relocation of French Jews. Document [19] states that after France expelled its original Jewish population in the Middle Ages, many of these Jews settled in areas that are now part of Germany, specifically in Alsace, which was formerly part of Germany.

Now, we need to find the first elected president of Poland, as Poland is the country where many of these French Jews relocated. According to document [9], the first president of Poland was Gabriel Narutowicz, who was sworn in on 11 December 1922.

Thus, the first elected president of Poland, where many expelled French Jews relocated, is indeed Gabriel Narutowicz.

The answer is Gabriel Narutowicz.

Figure 11: A case study that SubEM metric is "hacked" by previous study, which conduct more exploration with redundancy in response. LONGFAITH can arrive at the final correct answer with shorter response.
D Post-Training

In this section, we present two post-training algorithms—Supervised Fine-Tuning (SFT) and Preference Optimization (PO)—to better leverage synthetic data for efficiently enhancing model performance. Specifically, the model performs supervised fine-tuning on high-quality faithful outputs or is trained through reinforcement learning using synthetic preference pairs.

Supervised Fine-tuning on Faithful Outputs   We minimize the negative log-likelihood of the output as follows:

\[
\mathcal{L}_{\text{SFT}} = - \frac{1}{|y|} \log \pi_\theta(y \mid x)
\]
\[
= - \frac{1}{|y|} \sum_{i=1}^{|y|} \log \pi_\theta(y_i \mid x, y_{<i})
\]

where \( y \) denotes the high-quality faithful outputs, which are synthesized in Section 3.

Reinforcement Learning from Synthetic Preference   Additionally, we can leverage synthetic preference pairs for reinforcement learning (RL) to fine-tune the model toward generating faithful outputs while reducing the likelihood of low-scoring outputs. Standard RL algorithms for optimizing LLMs include Proximal Policy Optimization (PPO) (Schulman et al., 2017), RLOO (Ahmadian et al., 2024). However, these methods incur high computational costs. Recent approaches such as Direct Preference Optimization (DPO) (Rafailov et al., 2024), Kahneman-Tversky Optimization (KTO) (Ethayarajh et al., 2024), and Odds Ratio Preference Optimization (ORPO) (Hong et al., 2024) have been proposed to mitigate both computational and data requirements. In this work, we adopt the ORPO algorithm, which achieves an optimal balance between computational efficiency and model performance.

ORPO introduces an odds ratio loss \( \mathcal{L}_{OR} \) that minimizes the negative log odds ratio between preferred ("win" \( y_w \)) and dispreferred ("lose" \( y_l \)) outputs:

\[
\mathcal{L}_{OR} = - \log \sigma \left( \log \frac{\operatorname{odds}_\theta(y_w|x)}{\operatorname{odds}_\theta(y_l|x)} \right)
\]

where \( \sigma \) denotes the sigmoid function and \( \operatorname{odds}_\theta(y|x) = \frac{\pi_\theta(y|x)}{1-\pi_\theta(y|x)} \) measures how much more likely \( y \) is to be generated. The final objective of ORPO is to combine the SFT loss and the OR loss while controlling their relative importance through a hyperparameter \( \beta \):

\[
\mathcal{L}_{\text{ORPO}} = \mathcal{L}_{\text{SFT}} + \beta \cdot \mathcal{L}_{OR}
\]

In this paper, the chosen output \( y_w \) is synthesized by LongFaith through comprehensive consideration of Supporting Docs, Chain-of-Citation (CoC), and Ground Truth (GT), and is consequently assigned a high score. Conversely, the rejected output \( y_l \) refers to synthesized outputs that lack at least one of these three critical elements (Supporting Docs, CoC, or GT), which are deemed low-scoring due to insufficient design considerations.

E Statistics of Main Experiments

<table>
  <tr>
    <th>MuSiQue</th>
    <th>#2-Hop</th>
    <th>#3-Hop</th>
    <th>#4-Hop</th>
  </tr>
  <tr>
    <td>1K</td>
    <td>0</td>
    <td>512</td>
    <td>512</td>
  </tr>
  <tr>
    <td>2K</td>
    <td>512</td>
    <td>512</td>
    <td>1024</td>
  </tr>
  <tr>
    <td>4K</td>
    <td>1024</td>
    <td>2048</td>
    <td>1024</td>
  </tr>
  <tr>
    <td>8K</td>
    <td>3072</td>
    <td>4096</td>
    <td>1024</td>
  </tr>
</table>

Table 6: Statistics of train set for synthesis in different size sampled from MuSiQue (Trivedi et al., 2022b).
<table>
  <tr>
    <th>Datasets</th>
    <th>#Count</th>
    <th>Avg. L.</th>
    <th>Max L.</th>
  </tr>
  <tr>
    <th colspan="4">Multi-Hop Reasoning</th>
  </tr>
  <tr>
    <td>MuSiQue</td>
    <td>500</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>2-Hop</td>
    <td>254</td>
    <td>10843.3</td>
    <td>17560</td>
  </tr>
  <tr>
    <td>3-Hop</td>
    <td>154</td>
    <td>11456.5</td>
    <td>19225</td>
  </tr>
  <tr>
    <td>4-Hop</td>
    <td>92</td>
    <td>11224.3</td>
    <td>16756</td>
  </tr>
  <tr>
    <td>2WikiMultiHopQA</td>
    <td>500</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>2-Hop</td>
    <td>395</td>
    <td>4449.5</td>
    <td>10631</td>
  </tr>
  <tr>
    <td>4-Hop</td>
    <td>105</td>
    <td>4041.4</td>
    <td>9365</td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td>500</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Bridge</td>
    <td>412</td>
    <td>6301.0</td>
    <td>15702</td>
  </tr>
  <tr>
    <td>Comparison</td>
    <td>88</td>
    <td>5777.6</td>
    <td>11939</td>
  </tr>
  <tr>
    <th colspan="4">LongBench</th>
  </tr>
  <tr>
    <td>Qasper (S)</td>
    <td>200</td>
    <td>24262.3</td>
    <td>101636</td>
  </tr>
  <tr>
    <td>MultiFieldQA-En (S)</td>
    <td>150</td>
    <td>29583.7</td>
    <td>64751</td>
  </tr>
  <tr>
    <td>MuSiQue (M)</td>
    <td>200</td>
    <td>69876.8</td>
    <td>82338</td>
  </tr>
  <tr>
    <td>2WikiMHQA (M)</td>
    <td>200</td>
    <td>30076.5</td>
    <td>72971</td>
  </tr>
  <tr>
    <td>HotpotQA (M)</td>
    <td>200</td>
    <td>57041.4</td>
    <td>81815</td>
  </tr>
</table>

Table 7: Statistics of test sets including three long-context multi-hop reasoning datasets sampled by (Trivedi et al., 2022a) and five long-context QA datasets from LongBench (Bai et al., 2023). Avg. L. and Max L. refer to the average length and max length of input prompts for test samples. (S) and (M) refer to Single-doc QA and Multi-doc QA in LongBench.

<table>
  <tr>
    <th>Datasets</th>
    <th>Instruction</th>
    <th>Output(Chosen)</th>
    <th>Rejected</th>
  </tr>
  <tr>
    <td>LongAlpaca</td>
    <td>52043.2</td>
    <td>620.7</td>
    <td>0</td>
  </tr>
  <tr>
    <td>LongAlign</td>
    <td>36307.2</td>
    <td>1412.6</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MuSiQue-Attribute</td>
    <td>11395.0</td>
    <td>343.7</td>
    <td>0</td>
  </tr>
  <tr>
    <td>LongMIT</td>
    <td>280808.9</td>
    <td>825.2</td>
    <td>0</td>
  </tr>
  <tr>
    <td>LongReward</td>
    <td>72892.2</td>
    <td>913.4</td>
    <td>960.6</td>
  </tr>
  <tr>
    <td>SeaLONG</td>
    <td>82248.6</td>
    <td>1156.5</td>
    <td>1139.1</td>
  </tr>
  <tr>
    <td>LONGFAITH</td>
    <td>11542.1</td>
    <td>1029.6</td>
    <td>896.7</td>
  </tr>
</table>

Table 8: Average text length of baseline datasets and LONGFAITH in main experiments in Tab. 2. All of them has \(2K\) examples.

<table>
  <tr>
    <th>Datasets</th>
    <th>MuSiQue</th>
    <th>2WikiMultiHopQA</th>
    <th>HotpotQA</th>
    <th>Qasper</th>
    <th>MultiFieldQA</th>
    <th>Avg.L</th>
  </tr>
  <tr>
    <td>LongAlpaca</td>
    <td>365.62</td>
    <td>372.97</td>
    <td>319.60</td>
    <td>657.34</td>
    <td>511.25</td>
    <td>445.36</td>
  </tr>
  <tr>
    <td>LongAlign</td>
    <td>493.56</td>
    <td>349.65</td>
    <td>371.77</td>
    <td>651.76</td>
    <td>623.15</td>
    <td>497.98</td>
  </tr>
  <tr>
    <td>MuSiQue-Attribute</td>
    <td>99.61</td>
    <td>168.74</td>
    <td>164.24</td>
    <td>317.75</td>
    <td>252.95</td>
    <td>200.66</td>
  </tr>
  <tr>
    <td>LongMIT</td>
    <td>138.03</td>
    <td>159.16</td>
    <td>116.40</td>
    <td>194.41</td>
    <td>196.43</td>
    <td>160.89</td>
  </tr>
  <tr>
    <td>LongReward-SFT</td>
    <td>285.20</td>
    <td>241.47</td>
    <td>178.26</td>
    <td>750.83</td>
    <td>537.95</td>
    <td>398.74</td>
  </tr>
  <tr>
    <td>SeaLong-SFT</td>
    <td>1091.54</td>
    <td>776.01</td>
    <td>926.29</td>
    <td>1035.77</td>
    <td>822.82</td>
    <td>930.49</td>
  </tr>
  <tr>
    <td>LongFaith-SFT</td>
    <td>820.04</td>
    <td>619.18</td>
    <td>771.68</td>
    <td>1056.55</td>
    <td>941.13</td>
    <td>841.72</td>
  </tr>
  <tr>
    <td>LongReward-PO</td>
    <td>219.90</td>
    <td>253.41</td>
    <td>179.44</td>
    <td>616.11</td>
    <td>460.40</td>
    <td>345.85</td>
  </tr>
  <tr>
    <td>SeaLong-PO</td>
    <td>961.51</td>
    <td>740.14</td>
    <td>891.75</td>
    <td>946.68</td>
    <td>826.77</td>
    <td>873.37</td>
  </tr>
  <tr>
    <td>LongFaith-PO</td>
    <td>831.17</td>
    <td>669.76</td>
    <td>786.77</td>
    <td>1034.71</td>
    <td>917.11</td>
    <td>847.90</td>
  </tr>
</table>

Table 9: Average length of model output in test sets trained on different synthesized instruction.
F  Hyperparameters

<table>
  <tr>
    <th>Hyperparameters</th>
    <th>Value</th>
  </tr>
  <tr>
    <td># GPUs used</td>
    <td>4</td>
  </tr>
  <tr>
    <td>Learning rate</td>
    <td>5e-5</td>
  </tr>
  <tr>
    <td>Per-device batch size</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Gradient accumulation steps</td>
    <td>8</td>
  </tr>
  <tr>
    <td>LoRA rank</td>
    <td>32</td>
  </tr>
  <tr>
    <td>LoRA alpha</td>
    <td>64</td>
  </tr>
  <tr>
    <td>LoRA dropout</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>ORPO beta</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>Warm-up ratio</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>Epochs</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Precision</td>
    <td>bfloat16</td>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>AdamW</td>
  </tr>
</table>

Table 10: Hyperparameter settings of fine-tuning and preference optimization.

Reviewer paper summaries: The paper introduces LongFaith, which is designed to address faithfulness issues in synthetic data for long-context LLMs. By integrating ground truth and citation-based reasoning prompts, LongFaith improves reasoning accuracy and reduces reliance on verification. Two datasets (LongFaith-SFT and LongFaith-PO) are open-sourced, addressing verified reasoning, attribution, and contextual grounding. Experiments show that models fine-tuned on these datasets achieve significant performance gains, demonstrating the scalability and broad applicability.

The paper proposes a novel pipeline LongFaith for creating synthetic long-context datasets with high faithfulness characterized by a verified reasoning process, source attribution and contextual grounding. The pipeline is subsequently used to construct and open-source two datasets for model fine-tuning: LongFaith-SFT (for SFT) and LongFaith-PO (for RL). The paper also conducts extensive fine-tuning experiments to understand the value of the datasets and the individual aspects of faithfulness that comprise the approach.

The paper "LONGFAITH: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data" introduces a novel pipeline, LONGFAITH, designed to improve long-context reasoning in large language models (LLMs) by synthesizing faithful synthetic datasets. Addressing limitations in existing data-centric approaches—such as misinformation, lack of attribution, and knowledge conflicts—the pipeline integrates ground truth and Chain-of-Citation (CoC) prompting to generate verified, attributable, and contextually grounded reasoning chains. It produces two open-source datasets: LONGFAITH-SFT for supervised fine-tuning and LONGFAITH-PO for preference optimization, rejecting samples that fail faithfulness criteria (e.g., those with errors, no citations, or reliance on parametric knowledge) using hard rules rather than neural methods. Extensive experiments on multi-hop reasoning datasets and LongBench show significant performance gains when fine-tuning Llama-3.1-8B-Instruct, with ablation studies confirming the pipeline’s scalability and adaptability, though its generalization to other LLMs and tasks like summarization remains a future exploration.
Summary of strengths: 1. The paper addresses a significant issue of long-context reasoning that the synthetic data could be unfaithful.
2. The method appears to be novel and does not require additional human involvement.
3. The experiments are very thorough, demonstrating the effectiveness of LongFaith across multiple datasets.

**Well suited for increasing contextual faithfulness in multi-hop QA tasks:** The approach and datasets are designed around verifiable reasoning and grounding in multi-hop question-answering tasks and provide a richer ground truth for training on such models. This is also evident in the performance on these benchmarks when models are fine-tuned on these datasets. Each component in the proposed pipeline is well ablated (Table 3).

**More robust compared to existing similar datasets:** As shown in Table 2 and Appendix C, LongFaith produces more robust responses (e.g. much higher F1 scores at similar SubEM scores) than similar previous datasets e.g. LongAlpaca, SeaLong on MultiHop tasks.

The strength of the paper "LONGFAITH: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data" lies in its innovative LONGFAITH pipeline, which significantly enhances long-context reasoning in large language models by leveraging Chain-of-Citation (CoC) prompting to address key weaknesses like misinformation, knowledge conflicts, and lack of attribution, outperforming traditional Chain-of-Thought (CoT) methods. By integrating ground truth and contextual grounding, it ensures faithful synthetic data, systematically rejecting flawed samples using hard rules, thus avoiding costly neural verification processes. The authors demonstrate its effectiveness by intentionally selecting smaller models like Qwen2.5-7B-Instruct, which typically struggle with long-context attribution, and showing substantial performance improvements after training on LONGFAITH-SFT and LONGFAITH-PO datasets—evidenced by gains in F1 and EM scores on multi-hop reasoning tasks and LongBench. Extensive experiments, ablation studies, and open-sourced datasets further highlight its robustness, scalability (from 1K to 8K samples), and adaptability, proving its value even with underperforming models, while the positive correlation between attribution capability and performance underscores CoC’s superiority.
Summary of weaknesses: 1. The method may be unreliable due to the potential for hallucinations, which could generate incorrect citations.
2. In Table 2, LongFaith does not show performance improvements over previous methods in many settings, raising questions about the effectiveness of the approach that requires further validation.

Since I am not an expert in the long context task, the decision on whether this paper is accepted may depend more on the opinions of other reviewers.

**Synthesis lacks quality assurance:** The synthesis of high quality reasoning chains depends on prompts (Appendix B) and providing controlled information in inputs. However, there is no quality assurance mechanism to verify the logical consistency of the produced reasoning chains (e.g. groundings are valid and in the right sequence) which makes it completely dependent on the instruction following capability of the synthesizing LLM. This will also affect the quality of the generated preference dataset for RL. 

**Value not clear for Long-Context:** It is not clear how the proposed approach helps with longer contexts. Training dataset specifications don’t mention input context length and evaluations do not study the effectiveness of fine-tuning on LongFaith datasets as context length increases say from 8k -> 128k. Performance is measured on LongBench but that doesn’t ablate how approach’s effectiveness varies with increasing context length. 

**Imposes very specific bias on the right kind of answer:** The proposed approach seems very specific to MuSiQue type datasets (multi-hop QAs with clean composable individually groundable hops) and the synthesized reasoning chains explicitly impose that structure in model output via a highly tailored prompt (Fig 7). While this can help boost performance in multi-hop tasks, it is not clear training on these datasets improves general long-context reasoning capabilities in models. 

**Can make models prone to false / conflicting information in context:** The counter side of fitting the model on the synthesized chains with focus on grounding in every step overestimates the accuracy of provided context and can struggle with filtering out easily refutable / conflicting information within context in real-world tasks (e.g. search augmented large language models like Perplexity). For large models where parametric knowledge can help it counter false information in context, fine-tuning on this dataset can have the opposite effect on factuality despite improving grounding. 

**Paper contributions lean on the incremental side:** While the pipeline proposed in the paper and the corresponding datasets are novel, almost all of the core components come from existing literature (Musique dataset, Chain-of-Citation etc). The synthesized chains have no vetting process and the approach cannot be easily used to create new datasets either (need a base dataset with similar multi-hop structure and supporting paragraphs like Musique).

As noted in the "Limitations" section (Page 9), the pipeline currently emphasizes synthesizing datasets for reasoning, and while it excels in multi-hop reasoning and question-answering benchmarks like LongBench and MuSiQue, the authors admit that its generalization to diverse tasks remains unexplored. This limitation stands out because long-context LLMs are often deployed in varied scenarios beyond reasoning, and without evidence of LONGFAITH’s effectiveness in these areas, its utility as a comprehensive solution for enhancing long-context capabilities in LLMs is uncertain, potentially requiring significant adaptation or additional research for broader use cases.
Comments, suggestions and typos: 1. More experiments on more models could make the conclusion of the paper more reliable.

**Typo, Line 282:** s/samples/sampled

**Typo, Line 360:** s/Instructand/Instruct and

The extensive experiments, ablation studies, and detailed tables (e.g., Tables 3 and 4) provide strong empirical support for the pipeline’s scalability and effectiveness, while the focus on hard rules over neural methods is a practical and transparent approach.
Overall assessment scores: 3

3

4
Metareview score: 3
Metareview: This paper proposes a new pipeline for creating synthetic long-context data with multi-hop reasoning steps, Chain-of-Citation, and grounding to the input to increase faithfulness for standard multi-hop QA downstream tasks. This pipeline is then used to create two publicly available datasets for SFT and RL. Finally, the authors conduct a large number of experiments based on two open-source architectures with different context lengths and ablations of their data on 3 multi-hop datasets.
Summary of reasons to publish: - The proposed pipeline is building upon the strengths of several previous works to use chain-of-thought reasoning steps and grounding via citations to the input and can be beneficial to the NLP community both in terms of using the datasets for training other models straight off-the-self, but more importantly taking this idea and applying it on similar-looking long-context tasks such as summarisation, long-form generation, etc.
- The experimental results show a significant improvement with the use of the datasets, and are validated in a very rigorous manner, which further justifies the current direction of models towards reasoning, aka 'thinking', intermediate steps as a way to improve on faithfulness.