ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities
Authors: Peng Xu
Wei Ping
Xianchao Wu
Chejian Xu
Zihan Liu
Mohammad Shoeybi
Bryan Catanzaro
Venue: ICLR 2025 Poster (2025)
Keywords: Long Context LLM
Retrieval-augmented generation
Primary area: foundation or frontier models, including LLMs

Abstract: In this work, we introduce ChatQA 2, an Llama 3.0-based model with a 128K
context window, designed to bridge the gap between open-source LLMs and
leading proprietary models (e.g., GPT-4-Turbo-2024-04-09) in long context un-
derstanding and retrieval-augmented generation (RAG) capabilities. These two
capabilities are complementary to each other and essential for LLMs to process
large volumes of information that cannot fit into a single prompt. We present
a detailed continued training recipe to extend the context window of Llama3-
70B-base from 8K to 128K tokens, along with a three-stage instruction tun-
ing process to enhance the model’s instruction-following, RAG performance,
and long-context understanding capabilities. Our results demonstrate that the
Llama3-ChatQA-2-70B model outperforms most existing state-of-the-art models,
including GPT-4-Turbo-2024-04-09, Qwen2-72B-Instruct, and Llama3.1-70B-
Instruct, on ultra-long tasks beyond 100K tokens, as well as on the RAG benchmark
using only a 4K context window, showing the strong long context capability across
varying sequence lengths. We further provide extensive comparisons between
direct long-context and RAG solutions using the same state-of-the-art long-context
LLMs. Interestingly, we find that the performance of strong long-context LLMs
using RAG improves when retrieving a larger number of chunks. With a large set
of top-k chunks, RAG consistently outperforms direct long-context solution using
the same state-of-the-art long-context models (e.g., Llama3-ChatQA-2-70B and
Qwen2-72B-Instruct) on both 32K and 128K benchmarks. We open-source the
model weights, training data, and the evaluation setup for the for the community:
https://chatqa2-project.github.io/

Published as a conference paper at ICLR 2025
CHATQA 2: BRIDGING THE GAP TO PROPRIETARY LLMS IN LONG CONTEXT AND RAG CAPABILITIES
Peng Xu"
Wei Ping"
Xianchao Wu
Chejian Xu Bryan Catanzaro
Zihan Liu
Mohammad Shoeybi
NVIDIA
(pengx, wping @nvidia..om
ABSTRACT
In this work, we introduce ChatQA an Llama 3.0-based model with a 128K context window, designed to bridge the gap between open-source LLMs and leading proprietary models (e.g., Turbo--urbo-0024-044
long context un- derstanding and retrieval-augmentte generation (RAG) capabilities. These two capabilities are complementary to each other and essential for LLMs to process large volumes of information that cannot fit into single prompt. We present a detailed continued training recipe to extend the context window of Llama3- 70B-base from 8K to 128K tokens, along with three-stage instruction tun- ing process to enhance the model's instruction-followin and long-context understanding capabilities. Our results demonstrate that the Llama3-ChattAA---708 model outperforms most existing state-of-the-arr models, including Tur-44440ob-2224-04--999 Qwen2-72B- IInttrcct,, and Llama3. 1-70B- Instruct, on ultra-long tasks beyond 100K tokens, as well as on the RAG benchmark using only a 4K context window, showing the strong long context capability across varying sequence lengths. We further provide extensive comparisons between direct long-context. and RAG solutions using the same sate-offtheear long-context LLMs. Interestingly, we find that the performance of strong long-context LLMs using RAG improves when retrieving a larger number of chunks. With large set of top-k chunks, RAG consistently outperforms direct long-context solution using the same ofeee-----hhtt art long-conter models (e.g.. 2lama3-ChatQA- ChatQA-2 2-0B and Qwen2-72B (nstrrutt)) on both 32K and 128K benchmarks. We open-source the model weights, training data, and the evaluation setup for the for the community: https: chaathattt thaatqaa-aa -projecc........... g thub.io/
RAG performance,
INTRODUCTION
The open LLM community has made significant progress in advancing the capabilities of open-access large language models (LLLMs), including Jlama-3-70B-Instruel (Meta-AI, 2024), Qwen2-72B- Instruct (Alibaha-QWen, 2024), Bemotronn-43400-nnnsttee
(NVIDIA et al., 2024), and Mixtral- 8x22B-Instruct-vd (Mistral, 2024). However, performance gaps compared frontier proprietary models, c.g., GPT-4-Turbo (OpenAI, 2023), still exist in many domains. Additionally, open-access models focused key domains have been developed, such DeepSeck-Coder- V2 (Zhu al., 2024) for coding and math, ChatQA 1.5 (Liu et al., 2024) for conversational QA and retrieval-augmente generation (RAG), and VL 1.5 (Chen al., 2024) for vision- lisionelanguage tasks, which can be on par with GPPT-4-Turbo-2224-04444 (OpenAI, 2023) in the certain domains.
In recent developments, the trend of extending the context window length in L.MMs has gained remarkable traction within both the industrial and research communities. All leading proprietary LLMs support very large context window, allowing them to accommodate several hundred pages of text in single prompt. For example, GPT-4 Turbo (OpenAI, 2023) and Claude 3.5 Sonnet offer 128K and 200K context window, respectively. Meanwhile, Gemini 1.5 Pro (Gemini- -eami, 2024) impressively supports up to 10M context. Open-access LLMs have also made significant strides to keep up (01.AI et al., 2024; Alibaba- QWen, 2024). For instance, Qwen2-72B Instruct (Alibaba- QWen, 2024) and Yi-34B (01.AI et al., 2024) support 128K and 200K context windows, respectively.

---

Published as a conference paper at ICLR 2025
Ultra-long (>100K) tasks
Long (32K)
Short (4K)
Model Type
Long Context LLM (128K)
6 tasks
9 tasks (RAG)
Proprietary
UPT-4-Turbo-20244-4444
33.16
51.93
54.72
Qwen2-72B-Instee
39.77
49.94
54.06
ACCESS
Llama3. 1araaa3.--00bb0 1-770B-nnstttct
39.81
49.92
52.12
Open Llama3-ChatQA-2-T0B
41.04
48.15
56.30
ofate-of-the-an long context LLMs across tasks with varying context lengths. Notably, we open-source the training data and reproduction recipe for building 128K long-context LLMs from pretrained base models with 8K context, resources that are not currently available for open-access (open-weights) 128K context models like Qwen2 and 1 1. Our model achieves the highest average score on four real-world ultra-long context (beyond 100K) tasks from InfiniteBench, and nine short context (within 4K) tasks from CHATRAG BENCH.
Table I: Samma--haatQA-2-11 2-70B
The concurrent work Llama-3.1- 12.1-700-Instrre (Meta-Al, 2024) also supports 128K context window. However, the absence of training data and reproduction recipe makes replicating these models challenging. In addition, these models have mostly been evaluated on synthetic tasks, like Needle in a Haystack (Kamradt, 2023) test, which does not accurately represent real-world downstream task performance. In this work, we aim to create an open source long context recipe that poses the capability at the level of proprietary models (e.g.. BPT-4-Turbb-2224--4444 while focusing on real-world long-context understanding tasks.
On the other hand, the long-contex. capability of LLMs is sometimes considered a rival technique to retrieval- augmented generation (RAG). In fact, RAG and long-context techniques complement each other from a pragmatic perspective. An LLM with a long context window can either process large volumes of text prompt or utilize retrieval methods to efficiently extract relevant information from the extensive text, depending on the downstream tasks and accuracy vs. efficiency trade-offs. RAG has efficiency advantages and easily retrieve relevant contexts for query-based tasks (e.g, QA) from billions of tokens, feat that long context models cannot achieve so far. Meanwhile, long context models are good at tasks such as summarizing entire documents, where RAG can not perform. As a result, the state-of-the-art (SOTA) L.M needs to excel at both capabilities, providing options for different downstream tasks and meeting different accuracy-efficiency trade-off requirements. To compare the solution between long context and RAG, Xu et al. (2024) extended the context window of Llama2 (Touvron et al., 2023b) to 16K and 32K tokens, and studied the interplay between RAG and long-context LLMs. It demonstrates that the RAG method can improve the generation accuracy and inference efficiency a GPT-3.5-turbe 16k level long context model for QA and query- based summarization tasks.
In this work, we present ChatQA 2, which extends the study of long-context vs RAG by pushing long-context L.MSs to the GPT-4-Turbo level capability (with 128K context window), and combines with state-of-the-ar long-context retriever for RAG. See Table for an overview of the results. The training data and reproduction recipe for this work are publicly accessible.
Specifically, we make the following contributions:
1. We present a two-step approach to establish the long context capability of Llama3-70B. First, we extend Llama3-70B base's context window from 8K 128K by continually pretraining it on mix of SlimPajama (Soboleva et al., 2023) with upsampled long sequences (Fu al.. 2024). Then, we apply a three-stage instruction tuning process on curated datasets to enhance the instruction-folloiin RAG, and long context understanding capabilities at each respective stage. We find that stage-wise innstruction-tuning by incorporating previous datasets, simplifies experimentation and hyperparameter tuning.
2. We demonstrate that the resulting 1lama3-ChattAA227700 with 128K context window achieves better accuracy than most existing -fate-of-the-ae art models, including GPT-4-Turbo- 2024-04-09, Qwen2-72B -2n-stutt-t Instruct, exceeding 100K tokens and the RAG benchmark for conversational QA tasks within a 4K context window.
and Alama3.1-708 -nsttuutt both long-context tasks
2

---

Published as a conference paper at ICLR 2025
3. Previous RAG approachs has two potential issues that undermine downstream task accuracy: i) top-k chunk-wise retrieval introduces fragmentation of context, and ti) a small top-k leads to low recall, while larger k introduces too much irrelevant context to the L.MM (e.g., see the analysis in Figure of Yu et al., 2024). In contrast, we find that the RAG performance of our Llama3-ChatQA. Sat--hhttQA-------- 2-BBBB is highly robust variations in chunk size when using a congecontext retriever (Wang et al., 2023c; Lee et al., 2024). Even more promising, accuracy consistently improves as the total number of retrieved tokens for input increases.
4. We conduct extensive comparisons between direct long-context and RAG solutions using the same state-of-the-at long-context LLMs. We find that RAG can still outperform GPT-4-Turbo- level long-context models (including our Llama3 Llama3-ChatQA-2 ChatQA-2-70B) and real-world 128K tasks, given a sufficient number of top-k chunks.
both 32K benchmarks
We organize the rest of the paper as follows. We discuss related work in $ 2. We introduce the continued pretraining for context window extension in $ 3.1 and the three-stage instruction tuning in 3.2. We report results in $ 5 and conclude the paper in 7.
2
RELATED WORK
2.1 LONG CONTEXT LLM
The trend of extending the context window in L.L starts by Claude with 100K token context (An- thropic, 2023). Although the underlying long context techniques behind proprietary models are unclear, the open LLM and research community has developed many methods to extend the context window of LLMs through continued training or fine-tuning (Kaiokendev, 2023; Nijkamp et al., 2023; Chen et al., 2023a; Tworkowski et al., 2023; Chen et al., 2023b; Peng et al., 2023; Xiong et al., 2023; Fu et al., 2024), especially for open-access LLMs (Touvron et al., 2023a;b) based on rotary position embedding (RoPE) (Su et al., 2024).
There two popular approaches to adapt RoPE for long-context inputs: position interpolation (Chen et al., 2023a) and increasing the hase frequency 0 of RoPE (Xiong et al., 2023; Liu et al., 20236). Recently, Yi-34B (O1.AI al., 2024) was pretrained with a sequence length of 4K, and its context window was extended to 200K by increasing the RoPE 0 from 10,000 to 5M during continued pretraining. Qwen2-72B-Instreee extrapolated to a 128K context length using YaRN (Peng et al., 2023). Instead of extending the context window of the base model then applying instruction tuning, GradientAl (2024) directly fine-tunes the
Sutbaba-QWen. 2024) was trained on 32K-length contexts and
which uses NTK-aware interpolation (Peng et al., 2023) and the formula from Liu et al. (2023b) to scale up
Note that, the Llama-3.1 models (Dubey et al., 2024) with 128K context window were introduced after the release of this work. We add a comprehensive comparison between ChatQA-2 and Llama-3.1 in the experiments.
2.2 RETRIEVAL-AUGMENT GENERATION (RAG)
Retrieval with a standalone retriever (e.g., Karpukhin et al., 2020; Wang et al., 2022; Lin et al., 2023; Lee al., 2024) is long-standing solution for handling long texts that cannot fit into the context window of language models. In previous work, various retrieval-augmented language models have been proposed (Nakano al., 2021; Borgeaud et al., 2022: Wang et al., 2023b;a; Guu al., 2020; Izacard & Grave, 2021; Izacard et al., 2022; Lewis et al., 2020; Huang et al., 2023; Khandelwal et al., 2019: Liu et al., 2024).
In general, the input to retriever or dense embedding model is chunk of text (i.c., sequence of subword tokens), and the output an embedding vector. These dense embedding vectors are then indexed and retrieved using k-nearest-neighhe retrieval. Previous dense-embedding- based retrievers only supported limited chunk size (e.g.. 512 tokens) (e.g.. Karpukhin et al., 2020; Wang et al., 2022: Lin et al., 2023) due to the context window size of pretrained BERT base model. In top-k chunk-wise retrieval, the short chunk size increases context fragmentation. As a result, extending the context window of retrievers has become popular. For example, Jina Embeddings 2 2(Günther et al., 2023) and Nomic Embed (Nussbaum et al., 2024) support 8K tokens, while E5 misstrll--7 (Wang et al., 2023c) and NV-Embed (Lee et al., 2024) support 32K tokens.
3

---

Published as a conference paper at ICLR 2025
3 METHOD
3.1 EXTENDING CONTEXT WINDOW TO 128K
In this section, we present the method to extend the context window from 8K 128K for Llama3. Note that, the Llama-3.1 models with 128K context window are released after this work. We compare our Llama3-Cii Saataaa-hhat-A---11 2-70- and Llama3-ChatQA-2255 with -lama-3.1-70B-Inss. in experiments.
and
We prepare our long context pretraining corpus from the Slimpajama (Soboleva et al., 2023) following Fu et al. (2024). We upsample long-context documents with the hyperparameter set as 0.1 to produce 10 billion tokens with sequence length of 128k. Since Ilama3 pretrained with a much higher RoPE base frequency of 500,000 compared to Llama2, we increased the RoPE base frequency to 150M accordingly. We set the batch size to 32 to fit 4 million tokens in a batch and use a learning rate of 3e-5 to train 2000 steps (8B tokens in total).
Interestingly, we found it more effective to separate different documents using special characters, such as "
", rather than the reserved beginning and ending tokens <BOS> and <EOS>. We hypothesize that the <BOS> and <HOS> tokens in Llama3 signal the model to ignore previous chunks of text after pretraining, which is not helpful for the L.MSs to adapt for longer context inputs.
3.2 INSTRUCTION- -RRUUCTION-TUNING WITH LONG CONTEXT DATA
In this section, we present the instruction tuning method designed to enhance both long context understanding capability and RAG performance.
Specifically, we implement three stages of instruction-tuning, For the first two stages, we follow ChaiQA 1.5 (Liu al., 2024), where the model initially trained on a high-quality instruction- following datasets, and then trained on blend of conversational QA data with provided context. However, these two stages involve relatively short contexts, with a maximum sequence length of only 4K tokens. To enhance our model's capability to handle very long context sequences up to 128k tokens, we collect a long SFT dataset.
This dataset is collected through two categories: 1) For SFT data sequences less than 32k: We leverage existing long-context datasets from AaagAlpaca12K, GPT-4 samples from Open Orca 1 and Long Data Collections 2. 2) For sequence lengths between 32k and 128k: Since it is challenging to collect such SFT samples, we rely on synthetic datasets. We utilize NarrativeQA (Kočiský et al., 2018) 8), which includes summary paragraphs, questions, answers, and source long web pages. The summaries are human-generated based on the source web pages, while the question-answer pairs are annotated by humans using the summaries. To extend the context length, we inserted a summary into the corresponding long web page document at a random location, ensuring that sentence structure was not disrupted. This approach maintains the grounding of the question-anse -nnswer pairs within the augmented long documents.. Since we used NarrativeQA for synthetic data generation, we intentionally excluded it from the evaluation benchmarks to avoid any potential data contamination. Both the full long SFT dataset and the short SFT dataset from the first two stages are then blended for training. We set the learning rate at 3e-5 and the batch size at 32.
3.3 LONG CONTEXT RETRIEVER MEETS LONG CONTEXT LLM
As we mentioned in previous section, the current RAG pipeline for LLM has the following issues: i) The top-k chunk-wise retrieval introduces non-negligible fragmentation of context for generating accurate answers. For example, previous state-of- the censedembedding based retrievers (e.g.. Li ofte--of-the-att
et al., 2023; Lin et al., 2023) only support 512 tokens. ii) Small top-k (e.g., 5 or 10) usually leads to relatively low recall, while much larger & (e.g., 100) can lead to worse generation (see Table Xu et al. (2024)) as the previous LLMS could not utilize too many chunked context very well (Liu et al., 2023a). To address the issue, we propose to use the most recent long-contex. retriever (Wang et al., 2023c; Lee al., 2024), which can support thousands of tokens. In our setting, we use the E5-mistral
http::///gnnngigeeeee/erresssser Dataa sectioon httpeer///ogiigggee.ec///rsseees
veqa
4

---

Published as a conference paper at ICLR 2025
embedding model (Wang et al., 2023c) the retriever. The input of E5-Mistral model is chunk of text represented by sequence of subword tokens, and the output is an embedding vector. The long documents corpus are chunked and embedded into set of dense embedding vectors. Such dense embedding vectors are indexed and retrieved by k-nearest-neighhor retrieval at inference time, where the embedding of prompt/ question is treated as query.
BASELINES AND EVALUATION BENCHMARKS
Long context models We compare our models against SOTA long context models: 1) GPT-4-Turbo- 2024-04-09 (128K context window) (OpenAI, 2023), 2) Swen2-72B-Instruul (128K context win- dow) Alibaba-QWen 2024), 3) Llama-3-70 --nn-----0B--- Instruct-Gradiee 262k (256K context window) (Gra- dientAI, 2024), 4) Llama-3. 1-8B-Instruce and 1ama-3.1-70B-Instin (128K context window). Note that, the Llama-3.1 models are released after this work.
Retrieval-augmented generation (RAG) We apply RAG to our ChatQA models and baseline long-conters LLMs when it is applicable to the downstream tasks, such those with informative prompts or queries. By default, we use the E5-mistral retriever (Wang et al., 2023c), as it supports both short and long contexts. In experiments, we chunk size of 1200 and select the top chunks, as this configuration provides a good trade-off between cost and performance, as demonstrated the ablation study Section 6.
To give comprehensive study of different context lengths, our evaluation benchmarks covers three categories, 1) ultra-long context benchmarks beyond 100K tokens, 2) long context benchmarks within 32K tokens, and 3) short context benchmarks within 4K tokens.
4.1 ULTRA-LONG CONTEXT BENCHMARKS BEYOND 100K TOKENS
InfiniteBench (Zhang et al., 2024b) is proposed to evaluate the ultra-long context capability of LLMs over 100K sequence length. As we focus on real-world english tasks, we only take the four related tasks from the InfiniteBench, i.e. longbook summarization (En.Sum), longbook qa (En.QA), longbook multiple choice (En.MC), and longhook dialogue (En.Dia). En.Sum is a task that requires models to generate concise summary of the given novel and is evaluated using the ROUGE-L-Sum metric (Lin, 2004). En.QA is annotated by a pipeline that ensures the questions' necessitating of long-range dependencies and reasoning, beyond simple short passage retrieval. Aggregation reasoning and filtering reasoning are the two primary reasoning categories. Fl score used to evaluate the quality of the answer. En.MC is annotated with the same pipeline of En.QA except that four answer choices are provided and exact matching scores are reported. En. leverages movie and drama scripts from designated online database with long, multi-role dialogues. Exact matching score is used again to evaluate the prediction accuracy.
4.2 LONG CONTEXT BENCHMARKS WITHIN 32K TOKENS
We use the long context datasets (except Narrative as it is included in our training) from Xu et al. (2024) as our benchmark for long datasets within 32K. There are six datasets in total, where QMSum (QM), Qasper (QASP), QUALITY (QLTY) are token from SCROLLS (Shaham et al., 2022) and HotpotQA (HQA) MuSiQue (MSQ), MultiFieldQA
(MFQA) are token from LongBench (Bai et al., 2023). Following the official metrics, we report the geometric mean of ROUGE scores (i.e., ROUGE1/2/L) (Lin, 2004) for QM, the exact matching (EM) score for QLTY, and scores for the remaining four datasets QASP, MSQ, HQA and MFQA.
4.3 SHORT CONTEXT WITHIN 4K TOKENS
We ChatRAG Bench (Liu al., 2024) as benchmark for short context within 4k. ChatRAG bench consists setup of Liu et al. (2024), for Doc2Dial (D2D), QuAC, and QRcCC task with long documents, each document is divided into segments of roughly 300 words. The top 5 relevant chunks are then retrieved
datasets and we exclude HDial as it is included in our training. Following the
hhtpps: madb. com
5

---

Published as a conference paper at ICLR 2025
Pressure Testing Liama-3 3 1.1-88 -88-Innstrutt
Pressure Testing namaa-.1-708--mme
Fact Retrieval Across Context Lengths ('Nendle In A HayStack")
Fact Retrieval Across Context Lengths ("Nendle In A HayStark"),
0.0
L0
0.0
10
Percert.
11.0
33.
64.0
16
46.0
26
$6.0
36.0
05.0
89.
100.0
100.
5252
16410
2035
40007
41221
(2359
73457
didtty
asti
214077
12
2214be
7200ld
20129
42007
25799
6319
89278
101
Joatin
1214t0
Dian Lime
bien Lime
(a) Llama3. I-Instruct 8B
(b) Llama3. I-Instruct 70B
Pressure Testing Liama3 ChetQA
Pressure Testing Lama3 Chat@A 708
Fact Retrieval Across Context Lengths ('Needle HayStack")
Fact Retrieval Across Context Lengths ('Nerdle HayStack))
22.0
25.0
230
44.0
44.0
56.0
36.0
m..
78.0
32
29.0
70.0
02
190.
100.0
1840
18110
42607
402211
55799
20179
Jocko
120000
2232
10410
J010
42007
55.91
774887
15179
100300
134t72
121400
Token
(c) Allama3-ChaQQA-2
(d) Llama3- -hamaa--aaaQQ-2-008
Figure 1: Needle In A Haystack test for (a) Llama3.1 Instrrctt 8B, (b) Llama3. 1- 1-Instruct-70B, (c)
Slama3-ChatAA-2-8B and (d) -Jamma-ChatQAA22-000 A---7BBBBB up 128K context window. We show the result using the same needle: "The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day."
as context for each user question. For TopiOCQA and INSCIT, top-20 chunks were retrieved to obtain similar context length to the first three datasets. The other four datasets are CoQA, DoQA, ConvFinQA (CFQA), and SQA, which cover wide range of domains like finance, children's 's stories, literature, mid/high school exams, news, Wikipedia and etc. We use F1 score as the metric to evaluate the generations and report the average without HDial as it is fair zero-shot comparisons different models.
5 RESULTS
In this section, we present the results and comparisons from extensive benchmark evaluations. We begin with the synthetic Needle in a Haystack test, then focus on real-world long context understanding and RAG tasks.
5.1 NEEDLE IN A HAYSTACK
and Llama3-ChatQA-2288 model on the Needle In A Haystack (NIAH) test (Kamradt, 2023). This synthetic task is popular for testing the long- congccttext capability of LLMs, and can be considered as a threshold level evaluation. We use the needle test: "The best thing to do in San Francisco is eaf sandwich and sit in Dolores Park on sumny day." We find this is generally a harder pressure test than the "passkey" in haystack test, which is commonly used by other long-context LLM works. Figure 3 demonstrates the performance of our models up to 128K tokens. shows that our model achieves 100% accuracy under both test cases. This test confirms model's 's excellent long-co cong-context retrieval capability.
We evaluate our Allmaa-ChatQAA222
In addition, we also include the NIAH test for Llama3. -Instruct 8B and Llama3. From Figures la and Ib, find that neither model could pass the NIAH test with the needle: "The best thing to do in San Francisco..." although they could pass the easier Passkey retrieval test, as shown in Figures 4a and 4b in Appendix.
1etra33.IIInnttuee I-Instruct-706

---

Published as a conference paper at ICLR 2025
Model
Avg.
En.Sum
En.QA
En.MC
Dn.
GPT-4--Tubb-200244-40000
33.16
17.62
19.29
77.73
18.00
Claude 2
33.96
14,50
11.97
62.88
46.50
Yi-34B-200K
< 15.15
<5
12.17
38.43
<
w/ RAG
N/A
N/A
17.69
77.29
N/A
Qwen2-72B-Instrene
39.77
31,66
21.46
82.97
23.00
w/ RAG
N/A
N/A
16.48
76.86
N/A
32.57
14.27
29.52
69.00
17.50
Llama3. --8t-Instt
33.17
29.20
31.51
58.95
13.00
w/ RAG
N/A
NIA
15.41
64.63
N/A
Llama3. 1-m03.1-70B--.
39.81
30.94
38.49
75.55
14.29
w/ RAG
N/A
N/A
23.44
79.04
N/A
L1 1amma-ChaatQ4-2-BB-
35.59
17.14
43.51
64.19
17,50
w/ RAG
N/A
N/A
41.48
63.76
N/A
Ll 1lamm3-Chaare Camaa---aatQA-2-OBB
41.04
16.08
48.22
80.35
19.50
w/ RAG
N/A
N/A
40.66
73.80
N/A
Table Evaluation results on InfiniteBench includes real-world long-context understanding tasks beyond 100K context window. For RAG, we use top-5 retrieved chunks, each with 1200 tokens from E5-mistral retriever (Wang et al., 2023c).
5.2 ULTRA-LONG CONTEXT EVALUATION BEYOND 100K TOKENS
In this subsection, we evaluate the ultra-long context capability beyond 100K tokens on the real-world tasks from InfiniteBench (Zhang et al., 2024a). Table 2 shows that our model (41.04) outperforms many existing state-of-the-an models, such as T7T44-uubo-2024-01 Instruct (39.81), Qwen2-72B
(33.16), 3lama3.1-70B- (32.57) and Claude 2 (33.96), confirming the effectiveness of our recipe and state-of of-the-art long- long-context capability of our model. We find that our model performs extremely well at QA tasks while performs relatively low for summarization. This can be attributed to the lack of summarization data our SFT recipe. Additionally, compare Llama3- Llama3-ChatQA-2 A-2BB to Llama3. Instruutt and model again shows better results. The RAG setting shows it is worse than the full long context scores, when we employed the default RAG evaluation setting: top-5 retrieved chunks with a chunk size of 1200 tokens, so overall 5 x 1, 200 = 6, 000 are fed into LLMs. In Section 6 and Table 5, we show that RAG scores can be better than full long context solutions we increase the number of chunks in the prompt.
(39.77). Llama-3-708 B0O---nstuut----- Seattruc-rrraient-2000
5.3 LONG CONTEXT EVALUATION WITHIN 32K TOKENS
In this subsection, we evaluate the long context capability within 32K tokens. Table 3 shows that achieves the highest score of 51.93 among all models. Our model scores but is bit lower than Qwen2-72B- (49.92) This difference can attributed to the extensive
48.15, which is higher than 3-------------------- Searruct-rraiiet--222 Instruct (49.94) and Llama3.1 32K pretraining and large SFT data implemented by Qwen2-77B-Bhe Instructtttttttt and Llama3. 1-70BB Instructt while we used a much smaller continued pretraining corpus and SFT datasets.
Additionally, found that the default RAG solution (top-5 with chunk size as 1200) perform worse than the top long context solution (e.g. T09T-4-Turbo2202--44-442 Llama3- Llama3-ChatQA-2 Catt-hatttt-------- which suggests that SOTA long context LLMs can handle 32K tokens well within their context window.
Owwen--7B--nnsii
5.4 CHATRAG BENCH: SHORT CONTEXT EVALUATION WITHIN 4K TOKENS
In this subsection, we evaluate the models on the short context tasks within 4K tokens from CHATRAG BENCH (Liu et al., 2024). The results can found in Table 4. Our model achieves an average score of 56.30, outperforming most existing SOTA models with 128K context window, including GPT-4-urbo-2001 notrbo-2224-44-99 Qwen2-72BB 72eens-tt-tssttrutt.. and L 1--0B-Instruce It is also comparable, though slightly behind, Llama3- -aataa---hh------- 1.5--A---1--7- (57.14). which only supports 4K context length. This suggests that extending chort-c models to long-context has trade-offs. Exploring how to effectively extend the context window to even larger scales, e.g., millions of tokens in Gemini
7

---

Published as a conference paper at ICLR 2025
Model
Avg.
QM
QASP
QLTY
MSQ
HQA
MFQA
51.93
16.37
38.96
88.45
44.88
70.65
52.26
w/ RAG
49.84
16.07
36.18
85.85
42.17
67.85
50.94
Owen2-72B-Instren
49.94
17.06
34.84
84.85
46.80
65.98
50.12
w/ RAG
48.08
17.63
35.19
83.05
39.92
64.58
48.10
Llama-3-708B Ios-Insttuct--oe Instruct-Gradient Gradient 262k
40.51
20.72
30.64
74.35
20.20
45.82
51.33
w/ RAG
40.57
20.04
30.68
72.35
22.19
46.85
51.31
42.42
15.16
36.59
68.20
25.89
52.42
56.25
w/ RAG
41.35
14.08
35.05
65.35
28.30
50.76
54.54
1
49.92
14.85
35.75
87.40
45.65
59.64
56.23
w/ RAG
47.63
13.59
32.11
85.65
39.53
58.30
56.58
Llama3-Cha Samaa-ChatQAA--8B
39.41
11.64
28.85
63.35
27.81
53.81
$1.02
w/ RAG
40.31
13.20
28.85
61.10
29.77
57.81
51.15
Llama3-hhatQ-22-708
48.15
15.99
32.84
82,20
39.26
62.24
56.39
w/ RAG
47.59
16.16
33.27
81,70
38.16
61.13
55.14
Table 3: Evaluation results the long context benchmarks within 32K tokens. For RAG,
top-5 retrieved
chunks, each with 1200 tokens from ES-mistral retriever (Wang et al., 2023c). Note that here, we intend to optimize the RAG scores by increasing the number of total retrieved tokens and we leave the ablation in Scction 6.
Avg. wlo HiDil
Models
D2D
QuAC
QReCC
CoQA
DeQA
CFQA
SQA
TOQA
INSCIT
GPT-3.5-Turbo-063
50.69
34.83
37.17
50.46
79.33
41.11
73.15
60.63
44.30
35.
34.16
77.42
43.39
20444Turbo-2224-44.4oo
54.72
35.35
40.10
51.46
77.73
41.60
84.16
79.98
48.32
33,75
Command R+ 104B
51.40
33.51
34.16
49.77
69.71
40.67
71.21
74.07
53.77
35.76
Qwen2.72B-
54.06
35.76
38.48
51.21
85.04
33.89
77.52
77.06
51.64
35.90
Llama2-Chat-
44.64
36.87
32.47
49.40
80.41
38.97
46.85
37.62
44,31
34.88
Llama3-70B Sart--nsttat-Gradien---a-
45,20
34,30
24.01
49.60
73.45
25.76
54,70
63.80
46.30
34,89
Jamma3-CaaQQA-L555
57.14
41.26
38.82
51.40
78.44
50.76
81.88
83.82
55.63
32.31
Llama3.1 Instruct
35.24
74.32
29.84
33,85
Llama3, Llama3,1-70B-Imm Inss-Instuut
52.12
35.80
33.92
51,30
73,46
34.14
77.72
77.45
50.62
34.69
Llams3-Chat0A-2-88 I.aaa3-chatt0A-2-77
52.50
39.04
37.85
44.76
76.14
49.3
75.37
68. 68.25
$1.76
29.99
40.28
81.38
$2.08
Table 4: Evaluation results CHATRAG BENCH with 9 datasets. Following Liu al. (2024), exclude
HDial as included in the instruction tuning datasets. maximum context lengths are 4K tokens.
RAG (top-X)
Long context
Llama3-ChattQ--2-700
64.55 (k=30)
64.29
Qwen2-72B-Insst Instruct
52.95 (k=20)
52.22
Table 5:
We compare RAG long context evaluation using Dllama-ChhatAA-2-777 tasks beyond 100k. Here we use average accuracy of En. QA and En.MC tasks that can apply RAG. The RAG-based result still better than long context evaluation.
1.5 Pro (Gemini-Teanee 2024), without degrading performance on regular short-context tasks is an exciting research direction.
5.5
ABLATION STUDY
Retriever
Long (32K)
Fn.MC.
Fn.QA
E5-Mistral
47.59
73.80
40.66
NV-emb-v2
46.43
74.24
41.19
Table 6: Ablation of different retrievers for Llama3-Chatee Sa-aata-----0000 using the top-5 and chunk size as 1200.
To understand the choice of different retrievers, we also report the performance using NV-emb-v2 for L1 11lam3--haatt0---70,, the top-1 retriever in the MTEB leaderboard 5, Table shows there
8

---

Published as a conference paper at ICLR 2025
48.2
48.0
47.6
Average
47.4
X
47.2
RAG: chunk size 300
47.0
RAG: chunk_size, 600
46.8
X
RAG: chunk size 1200
Full long context
3000
6000
12000
18000
24000
context window
Figure 2: Ablation of Llama3- Sat.a---htt----70000 with RAG given different top-k = {5, 10, 20, 40) retrieval, and chunk-size (300, 600, 1200) long context benchmarks within 32K tokens (see Section 4.2 for more details). The accuracy can be monotonically improved with more retrieved tokens (i.e., kx chunk size) the context window.
are relatively small performance variations when switching to another high-performing retriever and (ii) there is no clear overall winner, with NV-Embed-v? showing better performance on ultra-long tasks.
All-in-one-stage
Threc-stage
Short (4K)
48.55
52.50
Long (32K)
40.69
39.41
Ultra-long (>100K)
35.07
35.59
Table 7: Comparison between the three-stage training and all-in-one-stage training.
To study the effect of three stage recipe, we blended all the instruction-tuning data from all three stages and trained new model based on Llama-3-8B base in single stage. The results are shown in Table we observed that the three-stage training process outperforms the all-in-one stage training on RAG by significant margin, while showing marginally better or worse performance ultra-long and long tasks.
To further improve the performance of our long (32K) setting, collect another SFT data which con- tains 1 1.6M samples covering a wide range topics. Table shows that our Saanaa-ChatQA-2--0 (new) comparable Llama3. 1--.e..---B--mnaee Instructt Long (32K) tasks while being significantly better Short (4K) and Ultra-long (>100K) tasks. Furthermore, shows that (i) our model outperforms Llama3. 8B-Instruct on the math benchmark GSM8K (ii) it underperforms and the coding benchmark HumanEyal. Since it has not been aligned by DPO PPO process as Llama3. Lomaa.1--B-Inn -ntttsruuut,, the MT-Bench score is lower. We could apply the RLHF process improve its MT-Bench score in future.
the knowledge-intensive MMLU
RAG VERSUS LONG CONTEXT
In Figure 2 and Table 5, compare direct long-conteds and RAG solutions on 32K and beyond 100K benchmarks. For sequences beyond 100K tokens, we report only the average scores for En.QA and En.MC, as the RAG setting is not directly applicable to En. Sum and En. Dia.

---

Published as a conference paper at ICLR 2025
Llama3-ChatQQ4-28B (new)
1amma..--B-Instiee
Shart (ChatRAG)
53.79
48.79
Long (32K)
42,05
42.42
Ultra long (>100K)
35.18
33.17
HumanEval
66.46
70.73
MMIJU
65.73
67.59
MT-bench
8.09
8.42
GSM8K (0-shot)
87.41
83.70
Table 8: Comparison between Llama3.1 1-8---1-----aa -8B-Instruct and new (aaaama--haaa----00000 model trained by 1.6M
SFT dataset
Figure 2 compares different chunk sizes for top-k retrieval and the total number of tokens fed into our long context LLM. Comparing total tokens from 3000 24000, found that more tokens consistently yield better results, confirming the strong long- long-context capability of our model. Also, for downstream tasks with sequence lengths up to 32K, RAG can still marginally outperform the long-context solution if sufficient number of top-k chunks are used, i.e., at least 12,000 tokens as shown in Figure Therefore, the hyperparameter k can selected based on the tradeoff between efficiency and accuracy in real-world applications.
For context lengths beyond 100K in Table 5, RAG (using top-30 chunks for our Lllama3-ChatQA-2-705 and top-20 for solution. In such scenarios, RAG is recommended for better accuracy and much lower inference cost, provided it is applicable the downstream tasks.
again outperforms the full long-context
CONCLUSION
We introduce -amaa-ChattQA22- ties for both understanding up to 128K long contexts and utilizing retrieved contexts for generation. This provides the flexible options for different downstream tasks with specific accuracy and efficiency requirements. We present a detailed and reproducible technical recipe for building and evaluating the model, including the methods, training data, and evaluation benchmarks. In particular, evaluate ChatQA 2 on short-context RAG benchmark (ChatRAG) (within 4K tokens), long context tasks from SCROLLS and LongBench (within 32K tokens), and ultra-long context tasks from In- finiteBench (beyond 100K tokens). Our results demonstrate that the Llama3- Llama3-ChatQA-2 ChatQA-2-70B can achieve GPT-4-Turbo-20 04-09 level accuracy on these benchmarks across different sequence lengths.
long context LLM that possess GPT-4 Turbo-level capabili-
REFERENCES
OLAII, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint S0tvv:240330466555 2024.
Libabb-QWen Qwen2 technical report. 2024. URL qwen2/.
Anthropic. Introducing 100k context windows.
1ookkcontextto.. windows, 2023.
Yushi Bai, Xin Lv, Jiajie Zhang. Hongchang Lyu. Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308. /4508, 2023.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Bapting Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In ICML, 2022.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.1553 15595, 2023a.
10

---

Published as a conference paper at ICLR 2025
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tune tine-tuning long-content large language models. arXiv preprint vrXiv:2399.23300 2023b.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404. 16821, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprini arXiv:2407.21783 2024.
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. arXiv preprint arXiv:2402.10177 2024.
Gemini-Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
arXiv preprint arXiv:2310.07713, 2024.
GradientAL.
Scaling rotational 2024.
embeddings
for long-context lan-
guage models,
URL
Michael Günther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Moham- mad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, et al. Jina embeddings 8192-token general purposse text embeddings for long documents. arXiv preprint vrXiv:23101199.. 2023.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. REALM: Retrieval augmented language model pre-training. In ICML, 2020.
Jie Huang, Wei Ping, Peng Xu, Mohammad Shoeybi, Kevin Chen-Chuan Chang, and Bryan Catanzaro. Raven: In-context learning with retrieval augmented encoder-decoder language models. arXiv preprint arXiv:2308.0777 079222, 2023.
Gautier Izacard and Édouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In EACL, 2021.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yie Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave, Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299, 2022.
Kaiokendev. Things I'm leamning while training SuperHOT. https://kaiokendee 1o/til44ettenddng-coo wwwp:eiddoo-oooooootooooooteoeee 20233. 2023.
thub.
Gregory Kamradt. Needle in a haystack pressure testing llms. SttrraadtLLLMMeett____. Itr.reett_eeeeeeeaaaaaaatttttee 2023.
com/
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. EMNLP, 2020.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv: 2019.
Tomá's Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Ny-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint 4rXiv:2405.1 17428, 2024.
11

---

Published as a conference paper at ICLR 2025
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmmeted genera- tion for knowledge-intenen intensive nlp tasks. NeurIPS, 2020.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03200 2023.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL httph::////// /ale:///llllle /aa/aclaannn www.annnooooyyy. org. --11113..
Sheng-Chich Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. arXiv preprint 2021v:230..07452, 2023.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172 2023a.
Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of rope-based extrapolation. arXiv preprint arXiv:2310.05209, 2023b.
Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro. ChatQA: Surpassing GPT-4 on conversational QA and RAG. arXiv preprint arXiv:2401.10225, 2024.
Meta-Al. Llama model card. 2024. URL WRLLttss///etttbb.com///e 1lama/llamaa blob/main/Monl CARD_CARR.... md.
Mistral. Mixtral 8x22b. 2024. URL /rL.ittpss/mmmttral..a//eee//
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju. William Saunders, al. WebGPT: Browser-assisted question-answerrig with human feedback. arXiv preprint arXiv: 222.033322 2021.
Erik Nijkamp, Hiroaki Hayashi, Tian Xie, Congying Xia, Bo Pang, Congying Xia, and al. Long sequence modeling with XGen: A 7b L.M trained on 8k input sequence length. https:// blog. sa les forceoocaaicoooo resea rchh com com/xgen/. 2023.
Zach Nussbaum, John Morris, Brandon Duderstadt, and Andriy Mulyar. Nomic embed: Training a reproducible long context text embedder. arXiv preprint arXiv:2402.01613, 2024.
NVIDIA, Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay. Jonathan Cohen, et al. Nemotron-4 340b technical report. arXiv preprint arXiv:2406.11700 2024.
OpenAL.
GPT-4 turbo with 128k context.
https://openai.comm 2023. 2023.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint Sxkiv:2309.007//1 2023.
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, and Omer Levy. SCROLLS: Standardized CompaRison over long language sequences. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 12007-12021, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://eaee http::s:///llaahhloogg.o org/2022. amrlp-main. 823.
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama, 2023, URL https: htts::////ggggggg//e co/ /ao//taeee datasets.......oo wwt.crereeaaeeeeeeee 1 i jama- 627B.
12

---

Published as a conference paper at ICLR 2025
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiigg 568: 127063, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Szymon Tworkowski, Konrad Staniszewski, Mikotaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Mitoš. Focused transformer: Contrastive training for context scaling. arXiv preprint arXiv:2307.03170, 2023.
Boxin Wang, Wei Ping, Lawrence McAfee, Peng Xu, Bo Li, Mohammad Shoeybi, and Bryan Catanzaro. InstructRetro: Instruction tuning post retrieval-augmentt pretraining. arXiv preprint arXiv:2310.07713, 2023a.
Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Olcksii Kuchaiev, Bo Li, Chaowei Xiao, et al. Shall pretrain autoregressive language models with retrieval? a comprehensive study, arXiv preprint wrXiv:2304.06762 2023b.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by seakly-s contrastive pre-training. arXiv preprini arXiv:2212.03533, 2022.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. arXiv preprint (xXiv:2401.0036. 2023c.
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.1600 2023.
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. In ICLR, 2024.
Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. Rankrag: Unifying context ranking with retrivvll-uummented generation in LLMs. arXiv e-prints, pp. arXiv-2407, 2024.
Xinrong Zhang. Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. oobench: Extending long context evaluation beyond 100k tokens, 2024a.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. oobench: Extending long context evaluation beyond 100k tokens, 2024b. URL https: /ttpp::///aiiii orgg abs 2022244222200 13718.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepscck-coder-v Breaking the barrier of closed-souree models in code intelligence. arXiv preprint arXiv:2406. 1131,,12000 2024.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
Pressure Testing Llama-3 1-88-Instruct
Pressure Testing Uamm-331-708-nnstee
Passkey Retrieval Across Context Actri
Lengths
Passkey Retrival Across Context Lengths
0.8
10
0.0
10
11.0
11.0
04.0
36
46.0
67.0
56.0
2A
56.0
67.0
100.0
3967
18810
J01e7
18015
201744
126000
20
100.0
3237
16610
2214
2013
120000
3072
Token Limi
bien Limit
(a) Llama3.1 L Instruct 8B (passkey retrieval)
(b) Llama3.1-1 Instrrct 70B (passkey retrieval)
Pressure Testing Liama ChatQA-2-88 2.88
Pressure Testing Liama Lama3-Chatta-22 ChatGA
Foct Retricval Across Context Lengths ('Needle In A HayStack")
Fact Retricval Across Context Lengths ("Nendle In HayStack")
0.0
0.0
10
Percert.
33.0
64.0
16
46.0
26
$6.0
56.0
05.0
100.0
100.0
3252
11410
29358
43221
(2559
69022
73451
18015
a5170
1012
2083000
114877
720090
2293
42007
25790
10
05779
126050
Dien Lime
Taken Limit
(c) Sammaa-ChatQA-2-8B1 (passkey retrieval)
(d) Chaaa--CatQQA-2-00B (passkey retrieval)
Figure 3: Needle in A Haystack for Llama3.1 I-Istrucc 8B and Llama3. 1 I-Instruct-Te 70B up 128K context window. We show the Passkey retrieval results here. The needle is set as "The pass key is 385243. Remember 385243 the pass key." " with the question asking "What the pass key?" ?"
To further confirm the superrior retrieval performance, we evaluate our models on the three retrieval based tasks InfinitBBench in Table 9, Aligned with our superior NIAH scores, all our models get 100% accuracy for number_string and passkey tasks and our Llama3-ChhatQA-2-700 get better score than GPT-4-Turbo on kv kv_retrieval task.
model
kv retrieval
numbe string
passkey
GPT-4-Turbo
89
100
100
91
100
100
Llama3-ChaatQA-2-88
72
100
100
Table 9: Llama3-ChattQA-2-700 demonstrates better performance on three retrieval tasks In- finiteBench.
B APPENDIX
We trained our model on 2B tokens with different separators and the NIAH below shows that using document breaker "
" is much better than <EOS> <BOS>.
14

---

Published as a conference paper at ICLR 2025
NIAH test using as the separator)
(b) NIAH test using <EOS> <BOS> as the separator)
Figure 4: NIAH shows that using document breaker "
" is much better than LEOS><BOSS
C APPENDIX
The three-stage SFT process involves blending multiple datasets, with each dataset potentially being trained for a different number of epochs, depending on its blending ratio and number of samples. For instance, the statistics for the datasets used in Stage 2 provided below, which adds complexity to the analysis.
number of samples
epochs
drop Narrative QA
29195
0.68
39440
0,69
Quoref
10996
0.68
ROPES
10924
0.69
squadl.1
86863
0.32
squad2.0
86092
0.32
newsqa
76560
0.36
convqa
5413
3.73
nvolve_4e
55458
1.20
scale_cqa
18224
2.37
instruct_ __
1411
4.50
hybriddial
19190
2.70
convfinqa
1162
6.70
tatqa
3176
5.45
corner_caa casess
1720
4.19
Table 10: Comparison between the three-stage training and all-in-one-stage training.
15


Reviews:
[{"review_id": "8lak17nFkh", "reviewer": "Reviewer_BLGw", "summary": "This paper presents a way to train a short context base model to handle long context. In involves positional encoding (RoPE) to allow long context handling, pretraining on long context data and RAG. The trained performance on llama 70b shows comparable results to commercial LLMs in long context.", "questions": "- More details in the method. The 3.1 and 3.2 looks quite common. Since the authors emphasizes RAG, what specific changes have the authors made? And how RAG is used in the training?\n- I am a bit confused by the highlight of this paper. I suppose the authors want to present a training pipeline for long context. But I don't quite understand the ablation of why certain choices are made.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "The performance looks good.", "weaknesses": "- The method is not clearly presented. I don't understand what fundamental changes of RAG or finetuning that makes the long context performance better\n- The ablation study of the training process / design is missing. I don't understand the takeaway message from this paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4kX8KQqugD", "reviewer": "Reviewer_UtmQ", "summary": "This paper introduces ChatQA 2, an open-source long-context language model with a 128K token context window, aiming to achieve performance comparable to GPT-4-Turbo. By leveraging a continue pretraining and a three-stage instruction tuning process, ChatQA 2 demonstrates improved accuracy and robustness in handling large volumes of context and consistently outperforms direct long-context models when using RAG with sufficient top-k chunks.", "questions": "1. Will the collected long SFT dataset utilized in the three-stage instruction tuning be open source?\n\n2. Will the ChatQA 2 model be open source?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces a significant enhancement to Llama3-70B’s long-context processing abilities, extending its context window from 8K to 128K tokens. Compared to Llama3.1with 128K context window, the proposed ChatQA2 achieves better performance. \n\n2. This paper proposes a three-stage instruction tuning, and collect a long SFT dataset to enhance the model's capability to handle very long context sequences up to 128k tokens. \n\n3. The authors propose utilizing the latest long-context retriever to address challenges in the current RAG pipeline for LLMs, specifically the limitations of top-k chunk-wise retrieval.", "weaknesses": "1. Compared to Llama3.1, ChatQA 2 shows improved performance on InfiniteBench, which includes real-world long-context understanding tasks beyond a 100K context window. However, on long-context benchmarks within 32K tokens, ChatQA 2 does not outperform Llama3.1, possibly due to its instruction tuning being focused on long SFT dataset. The author should also pay attention to the performance of 32K tokens.\n\n2. The long-context open-source model ChatQA 2 presented in this paper achieves strong performance, though it lacks some innovation. The synthetic dataset used in the long SFT dataset could be described in more detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xbSDRsqdzn", "reviewer": "Reviewer_teFx", "summary": "This work develops ChatQA 2, a long context LLM based on llama3-8k model that matches the performance of the April version of OpenAI GPT4 models and outperform SOTA open-source models. This work introduces a two-stage adaptation pipeline including long context continued training and multi-stage instruction finetuning. The developed model is then evaluated on short-context RAG, medium context up to 32k and long context above 100k benchmarks to show the superior performances than the existing open-source models.", "questions": "1. At several places, e.g. line 144, 168, 222, in this manuscript, the authors mentioned \"the first version of this work.\", what does that mean? Please clarify.\n\n2. What are the key new insights that this work offer, considering that the Llama3.1-128k model was published late July? I am not complaining about anything, but mainly want to make sure the technical contributions of this work. I will appreciate the answer from the authors.\n\n3. Around line #188, the authors mentioned \"where the model is initially trained on 128k high-quality instruction following datasets\", while also mentioning \" these two stages involve relatively short contexts\", what does that mean, why 128k high-quality datasets are short contexts? Is there a typo?\n\n4. In the long sequence SFT data, the authors used NarrativeQA for synthetic data generation. Are there potential data contamination in the benchmarks?\n\n\n5. Minor questions: line # 263: why are the documents segmented into 300-word chunks? Does the size matter for the benchmark results?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The developed ChatQA2 model shows promising performances than existing open-source baselines and the April version of OpenAI GPT model. \n\n- The evaluation from short to long contexts are comprehensive and convincing.\n\n- This work presents a detailed data and training recipe that enables a good-performing long-context model, bridging the gap between the commercial sector and open-source community for long context modeling.", "weaknesses": "- The authors should clearly articulate the relationship of this work and ChatQA 1.5, right now it was described as both a baseline (as shown in Table 4) and confusingly as \"the first version of this work\", i.e. the same work. If ChatQA1.5 is indeed a baseline, then it should be listed in all the tables and result analysis.\n\n- Some details of the computational experiment are missing in section 3: how many epochs and tokens were trained in each stage? How much does this impact the results? For example, there is a discussion to the potential reason of low longbench scores due to under-training. The authors should present the trend of higher scores with more training epochs (in supplementary information).  Please also provide the data, and training for reproducibility purposes.\n\n- The authors should improve the comprehensiveness of their ablation studies and results analysis: (1) The authors should also evaluate the retrieval-based tasks in infinitebench to further confirm the NIAH superior performances as shown at the beginning. (2) The section 5.2 is a bit lacking: the RAG numbers seem not being discussed. (3) The authors should add an ablation study to support their claim that started at line #177, i.e. using an alternative document breaker helps the training.\n\n- Minor comments: (1) the authors should add citations of scrolls and longbench when they are mentioned in line # 252 and line # 253. (2) There seems to be a typo in line #261: \"as it is vasincluded\" seems a bit confusing. (3) The authors should keep the bold texts consistent in all tables, right now some Llama-3.1-128k models are made bold but some are not. For example, I am not sure why 48.15 is made bold in Table 3, but not 49.92 (for 3.1-70b-instruct).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qbKRab3S5r", "reviewer": "Reviewer_TcQ9", "summary": "This paper introduces a recipe to extend Llama-3.0 models to long context windows (128K). The recipe involves increasing the RoPE frequency, continual pretraining on longer length corpus, then supervised fine-tuning on a mix of short and long context length instructions. They benchmark the model on tasks (synthetic and real-world) with various context lengths and found consistent gains over the original Llama-3 models, with the 70B model exceeding the performance of GPT-4-Turbo on some tasks. Ablation studies demonstrate tradeoffs involved with extending LLMs to handle long-context inputs.", "questions": "See weakness section for questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "## Originality\nThe technique of extending the context lengths of LLMs is not new. The most originality lies in the creation of the synthetic, long-context instruction dataset highlighted in Section 3.2. The authors seem to suggest that utilizing long-context retrievers for RAG is a contribution in Section 3.3.\n\n## Quality\nThe paper is well written. Experiments cover a wide range of tasks and reflect actual usage of long-context LLMs in real life. Baselines covered are extensive. Nothing to pick here.\n\n## Clarity\nThe recipe is simple and presented clearly in the paper. Particularly, technical details are well-documented in the Method Section (Sec 3).\n\n## Significance\nThe topic of extending LLMs to support longer context than it was original pretrained is of itself significant, which in turn makes potential solutions significant.", "weaknesses": "## Discussion on the key factors influencing performance tradeoff amongst tasks with different context lengths in long-context models\n\nA major part of the paper is dedicated to compare the performance of long-context models on various (shorter) context lengths, specifically 4K, 32K, and 100K+. As the authors stated in P8L429 \"This suggests that extending short-context models to long-context has trade-offs.\", it is important to pinpoint what exactly controls the trade-off. The authors claimed in P7L372 that \"This difference (in 32K performance) can be attributed to the extensive 32K pretraining and large SFT data implemented by Qwen2-72B-Instruct and Llama3.1-70B-Instruct.\", which seems to hint at the data distribution being mainly responsible for the tradeoff. However, to the extent of my knowledge, Llama3.1 didn't released their training recipe nor data distribution. Can the tradeoff be fully attributed to the context-length distribution in the continual pretraining and SFT dataset? More discussion would greatly benefit the work.\n\n## Relevance of stage-wise instruction-tuning\n\nThe authors highlighted in P2L98 and P4L187 that the instruction-tuning is split into 3 stages: instruction-following on short context, QA on short context, then a mix of long context instruction-following and synthetic QA. Stated in P4L199 \"Both the full long SFT dataset and the short SFT dataset from the first two stages are then blended for training.\" What is the relevance of breaking down the instruction-tuning into stages when the final stage includes all data from the first two stages? An ablation study on training the model all-at-once with all the available data would help justify the multi-stage approach. Perhaps there are benefits in gradually increasing the context-length when instruction-finetuning and is actually key to the success of this recipe.\n\n## Evaluating tradeoffs with other LLM capabilities with standard, general benchmarks\n\nSection 5.2 highlights the difference in performance of the proposed model on QA and summarization. Quote \"We find that our model performs extremely well at QA tasks while performs relatively low for summarization. This can be attributed to the lack of summarization data in our SFT recipe.\" Indeed the SFT data is heavy on QA and thus accounting for the performance bias. It is concerning that the good performance on this model on the benchmarked tasks might merely be a tradeoff with other LLM capabilities (e.g. reasoning, ICL, commonsense, creative writing, coding). Providing results on more general benchmarks (e.g. MT-Bench, MMLU, HumanEval) would help users evaluate the entire spectrum of tradeoffs of the proposed recipe.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7oF28zXmXC", "reviewer": "Reviewer_GdeC", "summary": "This paper presents Llama3-ChatQA-2, a model based on LLama-3-base that is trained with an extended context window. The authors first collect a long SFT dataset for instruction-tuning and train the model with modified PE. On top of this, they integrate a long-context retriever to implement RAG. Extensive testing are conducted across a variety of tasks involving short, long, and ultra-long token lengths, achieving state-of-the-art (SOTA) results.", "questions": "1. Under what circumstances does RAG improve LLM performance? For instance, why does RAG cause a performance drop for Llama3-ChatQA-2 on En.MC in Table 2, while improving the performance of Llama3.1-Instruct? The SOTA results in Tables 2 and 3 were achieved without RAG. Does this suggest that the effectiveness of RAG depends on the specific benchmark rather than the context length?\n2. Could you test the results with retrievers other than E5-mistral?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 1, "contribution": 4, "strengths": "1. The paper provides open access to the trained model and training recipe, which facilitates further research in the field of LLM + RAG.\n2. The paper focuses on increasing the context length of LLMs and has conducted extensive testing across benchmarks of varying lengths. It compares the performance with both open-source and proprietary models, achieving promising results.", "weaknesses": "Some sections of the paper are too brief, presuming the reader’s familiarity with certain concepts, particularly those related to RAG. For example, the meaning of “chunk size” and a simple introduction to the inputs and outputs of the E5-mistral retriever are not sufficiently explained. Since the paper emphasizes *open-source training data and reproduction recipe* compared to the open-access (Open Weights) Llama3.1-70B-Instruct, it would benefit from providing more detailed explanations, like [1].\n\n> [1] The llama 3 herd of models. *arXiv preprint arXiv:2407.21783*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)