Venue: NAACL 2024

# Effective Long-Context Scaling of Foundation Models

Wenhan Xiong\({}^{|*}\), Jingyu Liu\({}^{i}\), Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang\({}^{*}\), Hao Ma\({}^{*}\)

###### Abstract

We present an effective recipe to train strong long-context LLMs that are capable of utilizing massive context windows of up to 32,000 tokens. Our models are built through continual pretraining from Llama 2 checkpoints with longer text sequences and on a dataset where long texts are upsampled. We perform extensive evaluation using language modeling, synthetic context probing tasks, and a wide range of downstream benchmarks. Across all evaluations, our models achieve consistent improvements on most regular-context tasks and significant improvements on long-context tasks over Llama 2. Moreover, with a cost-effective instruction tuning procedure that is free of expensive annotation, the presented models can already surpass gpt-3.5-turbo-16k's overall performance on long-context benchmarks. Alongside these results, we provide an in-depth analysis on each individual component of our method. We delve into Llama's position encodings and discuss its key limitation in modeling long data. We examine the impact of various design choices in the pretraining process, including the data mix and the training curriculum of sequence lengths - ablation results suggest that having abundant long texts in the pretrain dataset is _not_ the key to achieving strong performance, and we empirically verify that long context continual pretraining is more efficient and similarly effective compared to pretraining from scratch with long sequences.

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4643–4663

June 16-21, 2024 0204 Association for Computational Linguistics

## 1 Introduction

Large language models (LLMs), trained with an unprecedented magnitude of data and compute, hold the promise of fundamentally improving the way we interact with the digital world. As LLMs get rapidly deployed and continue to evolve through scaling, we envision these models to serve more intricate and complex use cases, such as analyzing dense knowledge-rich documents, powering more genuine and engaging chatbot experiences, and adding human users in iterative creation processes such as coding and design, etc. A crucial feature supporting this evolution is the ability to effectively process long-context inputs.

LLMs with robust long-context capabilities are primarily provided through proprietary LLM APIs Anthropic (2023); OpenAI (2023) and there is no open recipe for building long-context model that can demonstrate on-par downstream performance as these proprietary models. Moreover, existing open-sourced long-context models Tworkowski et al. (2023); Chen et al. (2023); Mohtashami and Jaggi (2023); MosacML (2023) often fall short on evaluations and primarily measure long-context capabilities with the language modeling loss and synthetic tasks, which do not comprehensively demonstrate their effectiveness in diverse, real-world scenarios. Additionally, these models often overlap the necessity of maintaining strong performance on standard short-context tasks, either bypassing the evaluations or reporting degenerated perfor

Figure 1: We show that our model’s validation loss can be fit as a function of the context length: \(L(c)=(\frac{\alpha}{\varphi})^{\beta}+\gamma\) with a different set of \(\alpha,\beta,\gamma\) for each model size. This power-law relationship suggests that context length is another important axis of scaling LLMs and our model can continually improve its performance as we increase the context length up to 32,768 tokens.



mance (Peng et al., 2023; Chen et al., 2023).

In this work, we present a method to build long-context LLMs with superior performance over existing open-sourced models. We build our models by continually pretraining from Llama 2 checkpoints with additional 400 billion tokens formed as long training sequences. Among the model series, the smaller 7B/13B variants are trained with 32,768-token sequences while the 34B/70B variants with 16,384-token sequences. In contrast to the limited evaluation performed by existing studies, we extensively evaluate our models across various types of benchmarks. On language modeling, our model demonstrates a clear power-law scaling behavior with respect to context lengths. This scaling behavior, as shown in Figure 1, not only shows our models' ability to consistently benefit from more contexts but also suggest that context length is another importance axis of scaling LLMs. When comparing our models to Llama 2 on standard benchmarks, we not only observe significant improvements on long-context tasks but also modest improvements on standard short-context tasks, especially on coding, math, and knowledge benchmarks. We explored using a simple and cost-effective procedure to instruction finetune our continually pretrained long models without any human-annotated data. The end result is a chat model that can achieve stronger overall performance than gpt-3.5-turbo-16k on a series of long-context benchmarks covering question answering, summarization, and multi-document aggregation tasks.

In the remaining part of this paper, we begin by presenting the continual pretraining approach and a lightweight instruction tuning procedure, followed by detailed results on a range of short and long context tasks. To facilitate future research, we complement our results with a thorough analysis discussing how the design of positional encodings, the length distribution of the dataset, and the training curriculum contribute to the final performance.

## 2 Method

### Continual Pretraining

Training with longer sequences can introduce significant computational overhead due to quadratic attention calculations. This motivates us to take a continual training approach instead of training from scratch. The underlying hypothesis that similar long-context capabilities can be learned by continually pretraining from a short-context model is later validated in Section 4.4. We keep the original Llama 2 architecture nearly intact during continual pretraining and only make a necessary modification to the positional encoding that is crucial for the model to attend long context. We also choose not to apply sparse attention Child et al. (2019) in this work, since given Llama 2 7OB's model dimension (\(h\) = 8192), the cost of attention matrix calculation and value aggregation only becomes a computation bottleneck when the sequence length exceeds 49,152 (\(6h\)) tokens Narayanan et al. (2021).1

Footnote 1: While sparse attention might be useful for reducing the key/value cache size at inference time when trading off performance, it can complicate the inference pipeline and the LSTM’s output.

Positional EncodingThrough early experiments at the 7B scale, we identified a key limitation of Llama 2's positional encoding (PE) that prevents the attention module from aggregating information of distant tokens. We adopt a minimal yet necessary modification to the RoPE positional encoding Su et al. (2022) for long-context modeling - decreasing the rotation angle (controlled by the hyperparameter "base frequency \(b\)"), which reduces the decaying effect of RoPE for distant tokens. In Section 4.1, we show this simple method outperforms a concurrent approach Chen et al. (2023) for extending Llama's context length.

Data MixOn top of the working model with the modified PE, we further explored different pretrain data mixes in Section 4.2 for improving long-context abilities, either by adjusting the ratio of Llama 2's pretraining data or adding new long text data. We found that often the quality of the data plays a more critical role than the length of texts for long-context continual pretraining.

Optimization DetailsWe continually pretrain Llama 2 checkpoints with increased sequence length while using the same number of tokens per batch as in Llama 2. We train all models for a total of 400B tokens over 100,000 steps. With FlashAttention Dao et al. (2022), there is negligible GPU memory overhead as we increase the sequence length and we observe around \(17\%\) speed loss when increasing the sequence length from 4,096 to 16,384 for the 70B model. For the 7B/13B models, we use learning rate \(2e^{-5}\) and a cosine learning rate schedule with 2000 warm-up steps. For the larger 34B/70B models, we find it important to set a smaller learning rate (\(1e^{-5}\)) to achieve monotonically decreasing validation losses.

 



### Instruction Tuning

Collecting human demonstration and preference labels for LLM alignment is a cumbersome and expensive process Ouyang et al. (2022); Touvron et al. (2023). The challenge and cost are more pronounced under long-context scenarios, which often involve complex information flow and specialized knowledge, e.g., processing dense legal/scientific documents, making the annotation task nontrivial even for skilled annotators. In fact, most existing open-source instruction datasets Conover et al. (2023); Kopf et al. (2023) predominantly consist of short samples.

In this work, we found that a simple and cheap approach which leverages a pre-built large and diverse short-prompt dataset works surprisingly well on long-context benchmarks. Specifically, we take an in-house short-context finetuning dataset (referred as "FT-Short") and augment it with synthetic self-instruct Wang et al. (2022) long data generated by Llama 2 Chat, in the hope that the model can learn a diverse set of skills through the large amount of short-context data and transfer that knowledge to long-context scenarios. The data generation process focuses on QA-format tasks: starting from a long document in our pretraining corpus, we select a random chunk and prompt Llama 2 Chat to write question-answer pairs based on information in the text chunk. We collect both long and short form answers with different prompts. After that, we also adopt a self-critique step where we prompt Llama 2 Chat to verify the model-generated answers. Given a generated QA pair, we use the original long document (truncated to fit the model's maximum context length) as the context to build a training instance.

For short instruction data, we concatenate them as 16,384-token sequences. For long instruction data, we add padding tokens on the right so that models can process each long instance individually without truncation. While standard instruction tuning only calculates loss on the output tokens, we find it particularly beneficial to also calculate the language modeling loss on the long input prompts, which gives consistent improvements on downstream tasks (Section 4.3).

## 3 Main Results

### Pretrained Model Evaluation

Short TasksTo make long-context LLMs universally useful, an important desiderata is to ensure robust performance on standard short-context tasks. We verify our models' performance on a series of common benchmarks following the previous work Touvron et al. (2023). The aggregated results are shown in Table 1. Overall, we observe _on-par and, in most cases, stronger results_ than Llama 2. Notably, we observe significantly improved results on coding, math, and knowledge intensive tasks such as MMLU. As shown in Table 2, our model outperforms both GPT-3.5 and PaLM Chowdhery et al. (2023) on MMLU and GSM8k. This is in contrast to a previous work Chen et al. (2023) which observes degradation on short tasks. We attribute the improvements to additional training FLOPs and the knowledge learned from newly introduced data.

Long TasksDifferent from previous works Chen et al. (2023); Mohtashami and

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Model** & Size & Coding & Math & MMLU & Commonsene & OpenQA \\ \hline  & 7B & 16.8 & 8.55 & 45.3 & 63.9 & 48.9 \\ Llama 2 & 13B & 24.5 & 16.3 & 54.8 & 66.9 & 55.4 \\  & 34B & 27.8 & 24.2 & 62.6 & 69.9 & 58.7 \\  & 70B & 37.4 & 35.2 & 68.9 & 71.9 & 63.6 \\ \hline  & 7B & 20.6 & 10.5 & 47.8 & 64.9 & 51.0 \\ Llama 2 Long & 13B & 25.7 & 21.5 & 60.1 & 67.8 & 56.8 \\  & 34B & 29.9 & 29.0 & 65.0 & 70.9 & 60.3 \\  & 70B & **39.9** & **41.3** & **71.7** & **72.7** & **64.0** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Performance on standard short-context benchmarks. We report _Coding_ score as the average of pass@1 of HumanEval Chen et al. (2021) and MBPP Austin et al. (2021); Math score as the average of top-1 accuracy of 8-shot GSM8K Cobbe et al. (2021) and 4-shot MATH Hendrycks et al. (2021); OpenQA score as the average of 5-shot performance on NaturalQuestions Kwiatkowski et al. (2019) and TriviaQA Joshi et al. (2017); _Commonsense_ score as the average of PIQA Bisk et al. (2020), SIQA Sap et al. (2019), HellaSwag Zellers et al. (2019), WinoGrande Sakaguchi et al. (2021), ARC easy and challenge Clark et al. (2018), OpenBookQA Mihaylov et al. (2018) and CommonsenSEQA Talmor et al. (2018).



Jaggi, 2023) that mostly rely on perplexity and synthetic tasks to gauge long-context performance, we perform long-context evaluation using real-world language tasks. We evaluate 0-shot performance on NarrativeQA Kocisky et al. (2018), 2-shot on QuALITY Pang et al. (2022) and Qasper Dasigi et al. (2021), and 1-shot on QMSum Zhong et al. (2021). The number of shots are decided based on the average sample length of each dataset (i.e., samples in Qasper and QuALITY are often much shorter than those of NarrativeQA). We focus these QA-style tasks because of the ease of prompt engineering2 and less biased automatic evaluations The input prompts are truncated from the left side if the prompts exceed the maximum input length of the model or 16,384 tokens. We compare with open-source long-context models available in Huggingface Transformers, namely Focused Transformer Tworkowski et al. (2023b), YaRN Peng et al. (2023), Xgen Nijkamp et al. (2023), MPT MosaicML (2023b, a) and Together's Llama 2 forkTogether (2023). As shown in Table 3, our models achieve superior performance compared to these models. At the 7B scale, only "Together-7B-32k" can match our model's performance. Note that this model is not a purely self-supervised model and has been finetuned using a large supervised dataset to improve its few-shot results. As the 7/13B variants of our models have been trained with 32k-token sequences, we also perform comparisons using 32,768 maximum prompts lengths and the results are consistent, as shown in Table 11.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline

- & 29.3 & **37.5** & 33.0 & 35.7 \\ GSMSK (8-shot) & 57.1 & **92.0** & 56.5 & 80.7 & 56.8 & 65.4 \\ HumanEval (0-shot) & 48.1 & **67.0** & 26.2 &
- & 29.9 & 32.9 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison with closed models on standard short tasks.

Figure 2: Performance on long-context tasks as the maximum context lengths of prompts increase.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & NarrativeQA & Qasper & QuALITY & QMSum \\  & F1 (0-shot) & F1 (2-shot) & EM (2-shot) & ROUGE-geo* (1-shot) \\ \hline Focused Transformer (3B) & 16.3 & 15.4 & 20.5 & 10.6 \\ Yarn-7B-128k & 20.9 & 26.2 & 32.3 & 11.4 \\ Together-7B-32k† & 23.3 & 27.3 & 41.2 & 12.6 \\ Xgen-7B-sk-base & 17.4 & 20.5 & 21.0 & 6.79 \\ MPI-7B-8k & 18.8 & 24.7 & 23.7 & 8.78 \\ Yarn-13B-128k & 23.4 & 27.1 & 46.4 & 11.9 \\ MPT-30B-8k & 22.9 & 29.0 & 41.5 & 10.3 \\ \hline Llama 2 70B & 25.7 & 27.5 & 53.0 & 11.9 \\ \hline Llama 2 Long 7B & 21.9 & 27.8 & 43.2 & 14.9 \\ Llama 2 Long 13B & 25.6 & 31.2 & 57.6 & 15.7 \\ Llama 2 Long 34B & 29.4 & 33.7 & 65.7 & 15.9 \\ Llama 2 Long 70B & **30.9** & **35.7** & **79.7** & **16.5** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison with open-source long-context models on research benchmarks. †: “together-7B-32k” is not a purely pretrained model and has been trained using supervised datasets which can improve its few-shot results. *: ROUGE-geo is the geometric mean of ROUGE-1, 2 and L. All numbers are validation results and the maximum allowed prompt length is set to 16,384 tokens.

 

Effective Context UtilizationTo validate that our models can effectively use increased context window, we first show in Figure 2 that the results on each long task improve monotonically as we increase the context lengths. Inspired by Kaplan et al. (2020); Hoffmann et al. (2022), we also found that the language modeling loss of our model follows a power-law plus constant scaling relationship with the context length (Figure 1), suggesting: 1) our model continues to show gain in language modeling up to 32,768 tokens of text, despite having diminishing returns (e.g., for the 70B model, if we double the context length, we can expect the loss to be reduced by a factor of \(2^{-\beta}\approx 0.7\) plus a model specific constant \((1-2^{-\beta})\cdot\gamma\)), and 2) larger models can leverage the contexts more effectively, indicated by the larger \(\beta\) value of the curves.

### Instruction Tuning Results

We test our instruction tuned model on ZeroSCRLLS Shaham et al. (2023) which bundles 10 long-context datasets spanning from summarization, question answering, to multi-document aggregation tasks. For a fair comparison, we use the same configuration (prompts, truncation strategy, and maximum generation lengths, etc) as specified by the benchmark. As shown in Table 4, without using any human annotated long context data, our 70B chat model is able to outperform gpt-3.5-turbo-16k on 7 out of the 10 tasks. In addition, we run evaluations on six new long tasks introduced in L-Eval An et al. (2023) and again observe strong results, as shown in Table 15 in the Appendix. We see that the finetuned model is particularly good at QA tasks which is the main theme of the self-instruct data. We expect the performance to be further improved if more diverse data are used for finetuning.

It is worth mentioning that evaluating long-context LLMs is a nontrivial task. The automatic metrics used in these benchmarks are limited in many ways. For instance, the summarization tasks only come with a single ground-truth summary and the \(n\)-gram matching metrics do not necessarily align with human preference. For QA and aggregation tasks, where the metric is less of a concern, truncating the input context might also remove the information necessary to answer the question. Another important caveat is that most proprietary models do not share their training data details, which makes it hard to take into consideration the potential leakage during public benchmark evaluation.

## 4 Analysis

### Positional Encoding for Long Text

Our early experiments used a synthetic "first-sentence-retrieval" task to probe the effective context window of the pretrained models where we simply prompt the model to return the first sentence of the input. Our initial task results suggest that, with the original Llama 2 architecture untouched, our model was unable to effectively attend beyond \(4,000\) - \(6,000\) tokens even after extensive long-context continual pretraining. We hypothesize that this bottleneck comes from the RoPE positional encoding used in Llama 2 series which imposes a heavy decay on the attention scores3 for distant tokens. We propose a simple modification to the default RoPE encoding to reduce the decaying effect - increasing the "base frequency

\begin{table}
\begin{tabular}{l|c c c|c c c c c|c|c} \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c|}{Summarization} & \multicolumn{3}{c|}{Question answering} & \multicolumn{3}{c|}{Aggregation} \\  & GR & SS & QM & SQAL & Qspr & Nrtv & QALT & MuSQ & SpDg & BkSS & Avg \\ \hline GPT-3.5-turbo (4k) & 21.3 & 16.1 & 15.6 & 20.4 & 49.3 & 25.1 & 66.6 & 27.1 & 49.1 & 49.8 & 34.0 \\ GPT-3.5-turbo-16k\({}^{\dagger}\) & 24.3 & 16.2 & 17.4 & 21.4 & 50.0 & 29.5 & 72.0 & 27.0 & 54.1 & 54.6 & 36.7 \\ Claude (8k) & 24.2 & 16.1 & 14.6 & 21.0 & 52.3 & 32.6 & 84.8 & 36.1 & 61.6 & 47.4 & 39.1 \\ GPT4 (8k) & 26.3 & 17.3 & 18.5 & 22.6 & 50.7 & 27.6 & 89.2 & 41.1 & 62.8 & 60.5 & 41.7 \\ \hline \end{tabular}
\end{table}
Table 4: ZeroSCROLLS long-context leaderboard results. \({}^{\dagger}\)Evaluated as of 8/7/2023. The GPT-4 and Claude results are directly copied from the leaderboard. Underscored are the 7/10 tasks where our model outperforms gpt-3.5-turbo-16k.

\begin{table}
\begin{tabular}{l l l l} \hline \hline
**PE** & Books & CC & Wikipedia \\ \hline RoPE & 6.548 & 6.816 & 3.802 \\ \hline RoPE PI & 6.341 & 6.786 & 3.775 \\ RoPE ABF & **6.323** & **6.780** & **3.771** \\ xPos ABF & 6.331 & **6.780** & **3.771** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Validation perplexity of models with different positional encoding variants. All samples are \(32,768\)-token sequences (CC: CommonCrawl).

 

\(b\)" of RoPE from \(10,000\) to \(500,000\), which essentially reduces the rotation angles of each dimension. The idea is also concurrently suggested in the Reddit (r/LocalLLaMa) community and Roziere et al. (2023). The effect of the base frequency change is visualized in Figure 4. Another concurrent approach named "position interpolation" (PI) Chen et al. (2023) proposes to linearly scale the input positions such that the positions of tokens in the long sequences will to mapped to the model's original position range. As shown by the figure, it also implicitly achieves a decay reduction effect.

Another interesting observation from the visualization is that RoPE introduces large "oscillation" in the long-range regions, which could be undesirable for language modeling Sun et al. (2022). To investigate whether this effect hurts performance, we also explored another recently proposed variant of rotary encoding, xPosSun et al. (2022), which smooths the high-frequency component. Note that xPos with the default parameters suffers from the same decaying issue as RoPE and therefore, we also applied a similar decay fix to xPos.

Specifically, we empirically compare the following methods: the RoPE baseline, PI, our proposed RoPE with adjusted base frequency (denoted as RoPE ABF), and xPos ABF (visual comparisons in Figure 4). We report results on 1) long-sequence validation perplexity in Table 5 and Figure 2(a), and 2) the first-sentence-retrieval context probing task4 in Figure 2(b). All model variants are continually pretrained from the 7B Llama 2 checkpoint with additional 80B tokens organized as 32,768-token long sequences. Results on these evaluations suggest that RoPE ABF performs the best among all explored variants. In particular, we see that RoPE ABF is the only variant that can maintain its performance up to the full 32,768-token context window on the first-sentence-retrieval task. We found that xPos ABF with less oscillation does not lead to substantial gains, suggesting that these artifacts are not detrimental to language modeling. While xPos is claimed to possess better extrapolation property Sun et al. (2022), we show xPos does not extrapolate better than RoPE after the base frequency modification (see Appendix C). We provide a theoretical analysis of RoPE ABF and its difference to PI in Appendix B.

Footnote 4: We also test on the PassKey task as used in Mohtashami and Jaggi (2023). All the model variants except RoPE can achieve perfect accuracy. We believe this task is overly simple for context probing.

### Pretraining Data Mix

The data used to continually pretrain our model combines existing datasets used by Llama 2 and new long text data. We also adjusted the data source mix ratio to up-weight long data samples. Our early experiments with 7B models confirms the significant improvements using this data mix for long-context tasks, as shown in the first two rows of Table 6. In this section, we aim to rigorously investigate the source of improvements. In particu

Figure 4: Decaying raw attention scores for distant tokens of explored positional encoding variants (assuming keys and queries are all-ones vectors).

Figure 3: Comparison of positional encoding variants on synthetic sentence retrieval task and validation perplexity evolution during continual pretraining.

 

lar, we are interested in differentiating the effects of the data length distribution and the quality of the corpus itself.

We perform two additional ablations using Llama 2's pretrain datasets: 1) we remove the long text data from the Llama 2 dataset and continually pretrain our model with mostly short documents; 2) we increase the sample weights of existing long text data to be similar to the long text ratio used by proposed new model. Interestingly, even with most of the long texts removed, the model can still obtain most of the performance gain over Llama 2. We also find that there is no clear and consistent advantage as we greatly increase the long data ratio (the third row v.s. the fourth row in Table 6 and Table 7). We observe similar results on the first-sentence-retrieval task as shown by Figure 6 in the Appendix.

Based on the above ablations, we can see that adjusting the length distribution of the pretrain data does not provide major benefits. However, as we evaluate these model variants' performance on standard short-context tasks, we find that new data mix also leads to large improvements in many cases, especially knowledge-intensive tasks like MMLU, as shown in Table 7. These results suggest that _long-context LLMs can be effectively trained even with very limited long data_ and the improvements of our pretrain data over the one used by Llama 2 mostly come from the quality of the data itself, instead of the length distribution difference.

### Instruction Tuning

We explored various strategies to instruction-finetune the pre-trained long context model which do not require any supervised long data. We start with only finetuning the models with the in-house short instruction data (FT-Short) and then blend in with some pretrain data to avoid forgetting of previous long context continual pretraining. As demonstrated in Table 8, using only short instruction data can already produce a decent long model that significantly outperforms Llama 2 on various long-context tasks. On top of this dataset that only includes short prompts, we see that adding pretrain data (calculating language modeling loss on the whole sequence) can further boost the performance on most datasets. Inspired by this, we add the LM loss over the long context inputs when we finetune with self-instruct data. This simple trick makes learning more stable when we have unbalanced input and output lengths5, which gives significant improvements on most of the tested tasks (the last two rows of Table 8).

Footnote 5: In our cases, the output lengths of most samples are a lot shorter than the those of the long-context inputs.

### Training Curriculum

Continual pretraining has demonstrated its efficacy in our experiments, but an open question still remains: does pretraining from scratch with long sequences yield better performance than continual pretraining? In this section, we study different training curricula and try to investigate if continual pretraining can offer competitive performance with less computation budget. We start off by pretraining a 7B model with 32,768 sequence length from start to the end. Then we explored various two-stage training curricula where we begin with 4096 sequence length and switch to 32,768 when the model completes 20%, 40%, 80% of whole training process. For all cases, we keep the same number of total training tokens and make sure the number of tokens per each gradient update remains

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
* remove long text data & 19.48\% & 39.14\% & 67.1\% & 36.60\% \\
- upsample existing long text data & 22.15\% & 36.82\% & 65.0\% & 42.83\% \\ \hline \hline \end{tabular}
\end{table}
Table 6: Comparison of different pretraining data mix on long-context tasks. Instead of showing the absolute performance, we report relative improvements over the 7B Llama 2 which has a 4,096-token context window. All models are evaluated with prompts truncated at 16,384 tokens.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
* remove long text data & 17.07 & 3.57 & 46.25 & **76.76** & 65.90 \\
- upsample existing long text data & 17.07 & 3.53 & 46.25 & 76.74 & 66.04 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Standard short task performance of long-context models with different pretrain data mix.
 

constant (4 million tokens) by adjusting the batch size and sequence length accordingly.

We evaluate our models on the long-text QA tasks used in Section 4.2. As shown in Table 9, continual pretraining from short context models can save around 40% FLOPs while imposing almost no loss on performance. These results also align with the validation perplexity and training loss curves that we observed from each run (shown in Figure 5 in the Appendix A) - the models can quickly adapt to the increased sequence length and get to similar loss scale.

## 5 Related Work

Long Sequence Models.A long line of research in handling long data focus on building more efficient attention variants. Popular methods include limiting the attention to a subset of the context Beltagy et al. (2020); Tay et al. (2020); Roy et al. (2021) and decomposing the quadratic computation by removing the softmax normalization Peng et al. (2021); Katharopoulos et al. (2020). However, the efficiency improvements largely diminish as the model size grows. We do not explore this direction since the attention computation is not the major bottleneck at the studied scale. We also leave the investigation of completely new modeling paradigms, e.g., state-space models Gu et al. (2021) or hybrid architectures Dao et al. (2022b), to future work, which would require customized training framework including hardware-friendly kernels and optimized training parallelisms.

Position Encoding.In contemporary Transformer models, relative position encoding has emerged as the standard due to its superior performance over early absolute variants Vaswani et al. (2017). Initial designs of relative encoding Shaw et al. (2018); Raffel et al. (2020) integrated position information by adding an additional learnable term to the attention scores. Press et al. (2022) proposes a simplified design with manually defined position terms and show improved length extrapolation results. Rotation-based methods Su et al. (2022); Sun et al. (2022) maintain the core principle of relative encoding but directly incorporate the position information into query and key vectors. This design was originally motivated by the compatibility with linear attention methods but later popularized through their adoption in LLMs Chowdhery et al. (2023); Touvron et al. (2023). We investigate this line of method due to the constraint of the based models and focus on evaluating their performance in modeling long text.

## 6 Conclusion

We present a series of long-context LLMs that leverage a simple yet necessary position encoding refinement and continual pretraining to achieve strong long-context performance. Our long context scaling is performed by continually pretraining from Llama 2 with additional 400B tokens and outperform Llama 2 on both short and long-context tasks. Our models also demonstrate superior performance compared to existing open-source long-context models and compare favorably against gpt-3.5-turbo-16k on a suite of long-context tasks after a simple instruction finetuning procedure without human supervision. We complement our results with a comprehensive analysis, providing insights on the influences of various factors including the nuances of position encodings, the data mix, and the pretraining curriculum on the final performance. We hope our study could make

\begin{table}
\begin{tabular}{l|c c c c c} \hline \hline
**Pretrain Curriculum** & **FLOPs** & NarrativeQA & Qasper & Quality & QMSum \\  & & F1 & F1 & EM & ROUGE-geo \\ \hline
32k from scratch & \(3.783\times 10^{22}\) & 18.5 & 28.6 & 37.9 & 11.46 \\
4k\(\rightarrow\)32k @ 20\% & \(3.405\times 10^{22}\) & 20.0 & 28.1 & 38.8 & 12.09 \\
4k\(\rightarrow\)32k @ 40\% & \(3.026\times 10^{22}\) & 20.1 & 27.0 & 37.4 & 12.44 \\
4k\(\rightarrow\)32k @ 80\% & \(2.270\times 10^{22}\) & 18.5 & 25.0 & 38.3 & 11.00 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Comparison of models with different training curricula on long context QA tasks.

 

long-context LLMs more accessible and facilitate further advancements in this field.

## 7 Limitations

Limited Functionality.The our model proposed in this paper has not yet been finetuned for a wide range of long-context applications, such as creative writing that require long-form outputs. Applying existing alignment recipes, e.g., RLHF, for various scenarios is expensive and nontrivial. Even skilled annotators may struggle to the intricate details in dense texts. In this regard, we consider developing efficient alignment methods for long LLMs to be a very valuable direction for future research.

Tokenizer Efficiency.While the proposed our model series can consume contexts up to 32,768 tokens, the actually number of words our model can take is largely affected by the tokenizer behaviour. The tokenizer used by the Llama series has a relatively small vocabulary (32k symbols) and often produces longer sequences compare to the sequences given by GPT-3.5's tokenizer - we observe our tokenizer often produce \(10\%\) more tokens on average. Additionally, the tokenizer we use also cannot efficiently handle whitespace, making it inefficient to process long code data.

Hallucination.Like other LLMs, we have observed hallucination issue when testing the proposed our model. While this issue is common for short-context models, tackling with this problem for long-context models can be more pronounced because of the dense information they consume and the insufficient alignment process.

## References

* Almazrouei et al. [2023] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Albahamsmi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: an open large language model with state-of-the-art performance.
* An et al. [2023] Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. L-eval: Instituting standardized evaluation for long context language models. _arXiv preprint arXiv:2307.11088_.
* Anthropic [2023] Anthropic. 2023. Introducing 100K Context Windows.
* Austin et al. [2021] Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henry Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program synthesis with large language models. _arXiv:abs/2108.07732_.
* Beltagy et al. [2020] Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. abs/2004.05150.
* Bisk et al. [2020] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In _Proceedings of the AAAI conference on artificial intelligence_, volume 34, pages 7432-7439.
* Chen et al. [2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. _arXiv preprint arXiv:2107.03374_.
* Chen et al. [2023] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation.
* Child et al. [2019] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. _arXiv preprint arXiv:1904.10509_.
* Chowdherey et al. [2023] Aakanksha Chowdherey, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. _Journal of Machine Learning Research_, 24(240):1-113.
* Clark et al. [2018] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tarjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. _arXiv preprint arXiv:1803.05457_.
* Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reitichiro Nakano, et al. 2021. Training verifiers to solve math word problems. _arXiv preprint arXiv:2110.14168_.
* Conover et al. [2023] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world's first truly open instruction-tuned llvm.
* Dao et al. [2022a] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Re. 2022a. Flashattention: Fast and memory-efficient exact attention with io-awareness. In _NeurlPS_.
* Dao et al. [2022b] Tri Dao, Daniel Y Fu, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Re. 2022b. Hungry huppos: Towards language modeling with state space models. _arXiv preprint arXiv:2212.14052_ 

Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 4599-4610. External Links: Link Cited by: SS3.
* J. Dhamala, T. Sun, V. Kumar, S. Krishna, Y. Pruksachatkun, Kai-Wei Chang, and R. Gupta (2021)Bold: Dataset and metrics for measuring biases in open-ended language generation. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 862-872. Cited by: SS3.
* J. Ding, S. Ma, Li Dong, X. Zhang, S. Huang, W. Wang, and F. Wei (2023)Longnet: scaling transformers to 1,000,000,000 tokens. External Links: Link Cited by: SS3.
* K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, Thorsten Holz, and M. Fritz (2023)Not what you've signed up for: compromising real-world llm-integrated applications with indirect prompt injection. arXiv preprint arXiv:2302.12173. Cited by: SS3.
* A. Gu, K. Goel, and C. Re (2021)Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396. Cited by: SS3.
* T. Hartvigsen, Saadia Gabriel, H. Palangi, M. Sap, D. Ray, and E. Kamar (2022)Toxigen: a large-scale machine-generated dataset for adversarial and implicit hate speech detection. arXiv preprint arXiv:2203.09509. Cited by: SS3.
* D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021)Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874. Cited by: SS3.
* J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. Anne Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guyon, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre (2022)Training compute-optimal large language models. External Links: 2204.10077 Cited by: SS3.
* C. Hutto and E. Gilbert (2014)Vader: a parsimonious rule-based model for sentiment analysis of social media text. In Proceedings of the international AAAI conference on web and social media, Vol. 8, pp. 216-225. Cited by: SS3.
* Z. Ji, N. Lee, R. Freiske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. Jin Bang, A. Madotto, and P. Fung (2023)Survey of hallucination in natural language generation. ACM Computing Surveys55 (12), pp. 1-38. Cited by: SS3.
* M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer (2017)Triviaqa: a large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551. Cited by: SS3.
* J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei (2020)Scaling laws for neural language models. External Links: 2004.10077 Cited by: SS3.
* A. Katharopoulos, A. Vyas, N. Papapas, and F. Fleuret (2020)Transformers are rnns: fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, Vol. 119 of Proceedings of Machine Learning Research, pp. 5156-5165. Cited by: SS3.
* T. Kocisky, J. Schwarz, P. Blunsom, C. Dyer, K. Moritz Hermann, G. Melis, and E. Grefenstette (2018)The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics6, pp. 317-328. Cited by: SS3.
* A. Kopf, Y. Kilcher, D. von Ratte, S. Anagnostidis, Z. Tam, K. Stevens, A. Barhoum, N. Minh Duc, O. Stanley, R. Nagyfi, et al. (2023)Openassistant conversations--democratizing large language model alignment. arXiv preprint arXiv:2304.07327. Cited by: SS3.
* T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, et al. (2019)Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics7, pp. 453-466. Cited by: SS3.
* S. Lin, J. Hilton, and O. Evans (2021)Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958. Cited by: SS3.
* Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019)Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: SS3.
* T. Mihalov, P. Clark, T. Khot, and A. Sabharwal (2018)Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789. Cited by: SS3.
* A. Mohtashami and M. Jaggi (2023)Landmark attention: random-access infinite context length for transformers. arXiv preprint arXiv:2305.16300. Cited by: SS3.
* MosaicML (2023a)Introducing mpt-30b: raising the bar for open-source foundation models. Note: Accessed: 2023-06-22 Cited by: SS3.
* MosaicML (2023b)Introducing mpt-7b: a new standard for open-source, ly usable llms. External Links: 2305.16300 Cited by: SS3.

 

Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Pretrui Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. 2021. Efficient large-scale language model training on gpu clusters using megatron-lm. In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_, pages 1-15.
* Nijkamp et al. (2023) Erik Nijkamp, Tian Xie, Hiroaki Hayashi, Bo Pang, Congying Xia, Chen Xing, Jesse Vig, Semih Yavuz, Philippe Laban, Ben Krause, et al. 2023. Long sequence modeling with xgen: A 7b llm trained on 8k input sequence length. _Salesforce AI Research Blog_. OpenAI. 2023. Gpt-4 technical report.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:27730-27744.
* Pang et al. (2022) Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nagia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. 2022. QuALITY: Question answering with long input texts, yes! In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5336-5358, Seattle, United States. Association for Computational Linguistics.
* Peng et al. (2023) Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models.
* Peng et al. (2021) Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong. 2021. Random feature attention. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
* Press et al. (2022) Ofir Press, Noah A. Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21(140):1-67.
* LaMa (2020) r/LocalLLaMa. NTK-Aware Scaled RoPE allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation. [https://www.reddit.com/r/LocallLAMA/](https://www.reddit.com/r/LocallLAMA/) comments/14l2fj5/ntkkaware_scaled_rope_allows_lllama_models_to_have/. Accessed: 2023-08-25.
* Roy et al. (2021) Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. 2021. Efficient content-based sparse attention with routing transformers. _Transactions of the Association for Computational Linguistics_, 9:53-68.
* Roziere et al. (2023) Bapitste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jeremy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Defossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023. Code llama: Open foundation models for code.
* Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winograde: An adversarial winograd schema challenge at scale. _Communications of the ACM_, 64(9):99-106.
* Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiqa: Commonsense reasoning about social interactions. _arXiv preprint arXiv:1904.09728_.
* Shaham et al. (2023) Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023. Zeroscrolls: A zero-shot benchmark for long text understanding.
* Shaw et al. (2018) Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 464-468, New Orleans, Louisiana. Association for Computational Linguistics.
* Su et al. (2022) Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2022. Roformer: Enhanced transformer with rotary position embedding.
* Sun et al. (2022) Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benham, Vishrav Chaudhry, Xia Song, and Furu Wei. 2022. A length-extrapolatable transformer.
* Talmor et al. (2018) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. _arXiv preprint arXiv:1811.00937_.
* Tay et al. (2020) Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. 2020. Sparse sinkhorn attention. In _ICML_, volume 119 of _Proceedings of Machine Learning Research_, pages 9438-9447. PMLR.
* Together (2023)Together. 2023. LLama-2-7b-32kx-instruct -- and finentuning for llama-2 models with together api.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gaudtier Izacard, Xavier Martinet, Marie-Anne Lacchaux, Timothelec Lacroix, Bapitste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models 

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Tworkowski et al. (2023a) Szymon Tworkowski, Konrad Staniszewski, Mikolaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Milos. 2023a. Focused transformer: Contrastive training for context scaling.
* Tworkowski et al. (2023b) Szymon Tworkowski, Konrad Staniszewski, Mikolaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Milos. 2023b. Focused transformer: Contrastive training for context scaling.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.
* Wang et al. (2022) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishizi. 2022. Self-instruct: Aligning language model with self generated instructions. _arXiv preprint arXiv:2212.10560_.
* Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence?
* Zhong et al. (2021) Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutthia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: A new benchmark for query-based multi-domain meeting summarization. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5905-5921, Online. Association for Computational Linguistics.

 

More Results

 

distance between the old and the new images of the embedded vectors:

\[d(f,\hat{f})=\max_{x\in\mathcal{X}}\min_{k\in\{0,..N-1\}}\text{dist}[f(x,k), \hat{f}(x,j)],\]

where \(\mathcal{X}\subset\mathbb{R}^{d}\) is the set of vectors that would need to be positionally embedded. (Chen et al., 2023) computed this distance through the magnitude of the attention scores, but still argued for the efficiency of their method "position interpolation" due to its reduced value of the distance to the original RoPE images when compared to the naive extrapolation of the positional embedding.

With this in mind, we consider two different methods to extend the sequence length of a trained transformer: Position Interpolation (PI) parameterized with \(\alpha\), and Adjusted Base Frequency (ABF) parameterized with \(\beta\). These two methods correspond to the following embedding curves:

\begin{table}
\begin{tabular}{l|c c|c c|c c} \hline \hline
**Model** & HumanEval & MBPP & MATH & GSM8K & NQ & TQA \\ \hline Llama 2 Long 7B & 18.3 & 23.0 & 4.22 & 16.8 & 27.5 & 74.4 \\ Llama 2 Long 13B & 19.5 & 31.8 & 8.38 & 34.6 & 32.5 & 81.1 \\ Llama 2 Long 34B & 22.6 & 37.2 & 10.6 & 47.4 & 35.0 & 85.6 \\ Llama 2 Long 70B & 32.9 & 46.8 & 17.2 & 65.4 & 39.8 & 88.2 \\ \hline \hline \end{tabular}
\end{table}
Table 13: Results on HumanEval (0-shot), MBPP (3-shot), MATH (4-shot), GSM8K (8-shot), NaturalQuestions (5-shot) and TriviaQA-wiki (5-shot).

Figure 6: first-sentence-retrieval performance of models trained with different data mixes.

\begin{table}
\begin{tabular}{l l l l l l l l l} \hline \hline
**Model** & PIQA & SIQA & HellaSwag & WinoGrande & ARC-e & ARC-c & OBQA & CSQA \\ \hline Llama 2 Long 7B & 78.9 & 48.7 & 77.8 & 70.4 & 76.2 & 52.0 & 59.0 & 61.0 \\ Llama 2 Long 13B & 81.6 & 50.7 & 81.2 & 74.1 & 77.7 & 51.4 & 55.6 & 70.4 \\ Llama 2 Long 34B & 82.6 & 51.7 & 83.8 & 77.5 & 79.7 & 54.8 & 60.2 & 77.0 \\ Llama 2 Long 70B & 83.3 & 52.8 & 85.7 & 79.6 & 80.3 & 58.4 & 59.6 & 81.9 \\ \hline \hline \end{tabular}
\end{table}
Table 14: Commonsense reasoning decomposed results. We use the same number of shots and evaluation metrics for all tasks as Llama 2.

 

\[f^{RoPE+PI}(x,t)_{j} =\left(x_{2j}+ix_{2j+1}\right)e^{i\alpha\cdot(b^{-\frac{2j}{d}})t}\] \[f^{RoPE+ABF}(x,t)_{j} =\left(x_{2j}+ix_{2j+1}\right)e^{i(gb)^{-\frac{2j}{d}}t}\]

Evaluating a positional embedding a-priori, we should consider the degree of granularity with which the embedding images are being distributed over the embedding space. Comparing alternative positional embeddings \(\hat{f}\) mapping \(\mathbb{R}^{d}\times\mathbb{N}\) into \(\mathbb{C}^{d/2},\) we should prefer the one with the maximal value of the distance between the two closest images:

\[q(\hat{f})=\min_{x\in\mathcal{X};k\neq j\in\{0..\hat{N}-1\}}\text{dist}[\hat{ f}(x,k),\hat{f}(x,j)].\]

This leaves us with a multi-objective decision selecting the positional embedding for a model with extended context: on one hand, \(\hat{f}\) should be chosen so that it minimizes \(d(f,\hat{f}),\) while on the other hand its value of \(q(\hat{f})\) should be big enough.

Before proceeding to the explanation on how we make this multi-objective decision, we would like to provide a geometric intuition for the positional embeddings considered here. While it is difficult to visualize a mapping \(\mathbb{R}^{d}\times\mathbb{N}\rightarrow\mathbb{C}^{d/2},\) we can consider \(x\in\mathbb{R}^{d}\) to be fixed and visualize the projection \(\mathbb{R}\rightarrow\mathbb{R}^{3}.\) To get the intuition behind PI and ABF, let us consider the helix that is formed by Re\(\big{[}f^{RoPE}(x,t)_{0}\big{]},\) Im\(\big{[}f^{RoPE}(x,t)_{0}\big{]}\) and Re\(\big{[}f^{RoPE}(x,t)_{j}\big{]}.\) The example on the Figure 6(a) depicts a black helix line given with the system

\[x=\text{cos $t$};y=\text{sin $t$};z=\text{sin $at$}.\]

The red dots on the line correspond to \(11\) integer values of \(t.\)

Figure 6(b) aims to illustrate the impact of Position Interpolation on the relative position of the mapped vectors. The distance between the consecutive points got reduced considerably compered to Figure 6(a). The impact of Adjusted Base Frequency is illustrated on Figure 6(c). The distance between the consecutive points remained almost the same as on Figure 6(a), although the minimal distance between points got considerably reduced due to the increased frequency of the helix. This effect of increased frequency of the helix would be reduced in the high dimension setting. The value of the coefficient \(a\) for the helix depicted on Figure 6(a) is two times larger than the value of the coefficient \(a\) for the helix depicted on Figure 6(c). If the dimension of the input of the attention mechanism is \(d=128,\) then the difference between \(\theta_{1}=b^{-\frac{2j}{d}}\) at \(b=10,000\) and \(\theta_{1}=b^{-\frac{2j}{d}}\) at \(b=500,000\) is only \(6\%.\) Thus, we further focus specifically on the distance between the consecutive images of the embeddings.

We make a formal comparison between Positional Interpolation and Adjusted Base Frequency by analytically comparing the pairwise distances between the images given by \(f^{RoPE+PI}\) and \(f^{RoPE+ABF}\) for consecutive integer values of \(t.\) This corresponds to the evaluation of \(q(\hat{f})\) discussed earlier. We will measure the distance between embedding images in terms of the Euclidean sine similarity metric since all versions of RoPE are norm-preserving.

\[\sin\angle(a,b)=\dfrac{Im\langle a,b\rangle}{\|a\|\|b\|}\]

\begin{table}
\begin{tabular}{l|c c c c c c} \hline \hline

- \\ gpt-3.5-turbo-16k & 59.7 & 69.9 & 69.3 & 45.4 & 24.9 & 45.9 \\ \hline \multicolumn{7}{l}{_Best open models reported in An et al. (2023)_} \\ longchat-13b-16k & 36.8 & 55.4 & 33.3 & 37.9 & 21.1 & 22.8 \\ chatglm2-6b-8k & 47.2 & 54.6 & 10.0 & 34.8 & 16.4 & 17.6 \\ \hline Llama 2 Long Chat & 52.9 & 81.8 & 76.0 & 47.3 & 25.5 & 66.7 \\ \hline \hline \end{tabular}
\end{table}
Table 15: Evaluation on additional long-context tasks from L-Eval. We report the official metrics defined in An et al. (2023) and the results of compared models are directly token from the paper.
 

The following result states that in a high-dimensional space, the sine similarity \(\sin{\angle}(f^{RoPE+ABF}(x,n~{}+~{}1),f^{RoPE+ABF}(x,n))\) between two consecutive embedding images of a vector \(x\) can be bounded with a value proportional to \((\log b+\log\beta)^{-1}.\) Moreover, the similarity \(\sin{\angle}(f^{RoPE+PI}(x,n+1),f^{RoPE+PI}(x,n))\) can be bounded using \(\alpha(\log b)^{-1}.\)

**Theorem 1**.: _For \(x\in\mathbb{R}^{d}\) and \(n\in\mathbb{N},\) the Euclidean sine similarity between the two consecutive images of a positional embedding can be bounded as_

\(\frac{\min_{k}x_{k}^{2}}{\|x\|^{2}}K_{d}\leq\sin{\angle}(f(x,n+1),f(x,n))\leq \frac{\max_{k}x_{k}^{2}}{\|x\|^{2}}C_{d}\)__

_where \(\lim_{d\to\infty}C_{d}\approx\begin{cases}(\log b+\log\beta)^{-1}\text{ if $ f=f^{RoPE+ABF}$ under the assumptions of $\alpha\ll 1$ and $b\gg 1$}.\\ \end{cases}\)_

Proof.: Let us begin the proof by writing down the expressions for the inner product between two images of RoPE variants.

\[\langle f^{RoPE+PI}(x,m),f^{RoPE+PI}(x,n)\rangle=\sum_{j=0}^{\frac{d}{2}-1} \left(x_{2j}^{2}+x_{2j+1}^{2}\right)e^{ib^{-\frac{2j}{d}}\alpha(m-n)}\]

\[\langle f^{RoPE+ABF}(x,m),f^{RoPE+ABF}(x,n)\rangle=\sum_{j=0}^{\frac{d}{2}-1} \left(x_{2j}^{2}+x_{2j+1}^{2}\right)e^{ib^{-\frac{2j}{d}}\beta^{-\frac{2j}{d} }(m-n)}\]

From them, we can derive the expressions for the Euclidean sine similarity between the images of the positional embeddings:

\[\sin{\angle}(f^{RoPE+PI}(x,m),f^{RoPE+PI}(x,n))=\frac{\sum_{j=0}^{\frac{d}{2}- 1}\left(x_{2j}^{2}+x_{2j+1}^{2}\right)\sin(b^{-\frac{2j}{d}}\alpha(m-n))}{ \sum_{j=0}^{d-1}x_{j}^{2}}\]

\[\sin{\angle}(f^{RoPE+ABF}(x,m),f^{RoPE+ABF}(x,n))=\frac{\sum_{j=0}^{\frac{d}{2 }-1}\left(x_{2j}^{2}+x_{2j+1}^{2}\right)\sin(b^{-\frac{2j}{d}}\beta^{-\frac{2 j}{d}}(m-n))}{\sum_{j=0}^{d-1}x_{j}^{2}}\]

Let's put \(m=n+1\) to compare the distance between the two consecutive positional embedding images of the same vector \(x.\)

\(\|x\|^{2}\sin{\angle}(f^{RoPE+PI}(x,n+1),f^{RoPE+PI}(x,n))=\sum_{j=0}^{\frac{ d}{2}-1}\left(x_{2j}^{2}+x_{2j+1}^{2}\right)\sin(b^{-\frac{2j}{d}}\alpha)\)

\(\|x\|^{2}\sin{\angle}(f^{RoPE+ABF}(x,n+1),f^{RoPE+ABF}(x,n))=\sum_{j=0}^{\frac {d}{2}-1}\left(x_{2j}^{2}+x_{2j+1}^{2}\right)\sin(b^{-\frac{2j}{d}}\beta^{- \frac{2j}{d}})\) Due to the range of \(b,\alpha\) and \(\beta\) that is typically considered, we can bound the arguments of the sine functions as \(0<\alpha b^{-\frac{2j}{d}}\leq 1\) as well as \(0<(\beta b)^{-\frac{2j}{d}}\leq 1.\) Using that, we derive that \(\sin(b^{-\frac{2j}{d}}\beta^{-\frac{2j}{d}})\) and \(\sin(b^{-\frac{2j}{d}}\alpha)\) are non-negative as well as \(x_{j}^{2}\) for any \(j\in\{1,\ldots d\}.\) Thus, the following inequalities hold:

Figure 7: RoPE variants visualization as helices.

 

\[\sum_{j=0}^{\frac{d}{2}-1}\min_{k}x_{k}^{2}\sin(b^{-\frac{2j}{d}} \alpha)\leq\sum_{j=0}^{\frac{d}{2}-1}\left(x_{2j}^{2}+x_{2j+1}^{2}\right)\sin( b^{-\frac{2j}{d}}\alpha)\leq\sum_{j=0}^{\frac{d}{2}-1}\max_{k}x_{k}^{2}\sin(b^{- \frac{2j}{d}}\alpha).\]

Carrying \(\min_{k}x_{k}^{2}\) and \(\max_{k}x_{k}^{2}\) out of the summation signs, we obtain

\[\min_{k}x_{k}^{2}\sum_{j=0}^{\frac{d}{2}-1}\sin(b^{-\frac{2j}{d}} \beta^{-\frac{2j}{d}})\leq\sum_{j=0}^{\frac{d}{2}-1}\left(x_{2j}^{2}+x_{2j+1} ^{2}\right)\sin(b^{-\frac{2j}{d}}\beta^{-\frac{2j}{d}})\leq\max_{k}x_{k}^{2} \sum_{j=0}^{\frac{d}{2}-1}\sin(b^{-\frac{2j}{d}}\beta^{-\frac{2j}{d}}),\] \[\min_{k}x_{k}^{2}\sum_{j=0}^{\frac{d}{2}-1}\sin(b^{-\frac{2j}{d}} \alpha)\leq\sum_{j=0}^{\frac{d}{2}-1}\left(x_{2j}^{2}+x_{2j+1}^{2}\right)\sin (b^{-\frac{2j}{d}}\alpha)\leq\max_{k}x_{k}^{2}\sum_{j=0}^{\frac{d}{2}-1}\sin( b^{-\frac{2j}{d}}\alpha).\]

Introducing \(C_{d}^{ABF}=\sum_{j=0}^{\frac{d}{2}-1}\sin(b^{-\frac{2j}{d}}\beta^{-\frac{2j}{ d}})\) and \(C_{d}^{PI}=\sum_{j=0}^{\frac{d}{2}-1}\sin(b^{-\frac{2j}{d}}\alpha)\) proves the first part of the Theorem:

\[\frac{\min_{k}x_{k}^{2}}{\|x\|^{2}}C_{d}^{ABF}\leq\sin\angle(f^{ RoPE+ABF}(x,n+1),f^{RoPE+ABF}(x,n))\leq\frac{\max_{k}x_{k}^{2}}{\|x\|^{2}} C_{d}^{ABF},\]

\[\frac{\min_{k}x_{k}^{2}}{\|x\|^{2}}C_{d}^{PI}\leq\sin\angle(f^{ RoPE+PI}(x,n+1),f^{RoPE+PI}(x,n))\leq\frac{\max_{k}x_{k}^{2}}{\|x\|^{2}}C_{d}^{ PI}.\]

Now, considering the limit of \(C_{d},\) we notice that due to the inequalities on the arguments of the sines, the following bounds hold:

\[(b\beta)^{-\frac{2j}{d}}\left(1-(b\beta)^{-\frac{2j}{d}}/\pi\right)\leq\sin( b^{-\frac{2j}{d}}\beta^{-\frac{2j}{d}})\leq(b\beta)^{-\frac{2j}{d}},\]

\[\alpha b^{-\frac{2j}{d}}\left(1-\alpha b^{-\frac{2j}{d}}/\pi\right)\leq\sin(b ^{-\frac{2j}{d}}\alpha)\leq\alpha b^{-\frac{2j}{d}}\]

Using the formula of geometric sums and a corollary of the exponential (second) foundational limit, we establish the limits of the sums of these bounds as \(d\to\infty\):

\[\sum_{j=0}^{\frac{d}{2}-1}\alpha b^{-\frac{2j}{d}}=\frac{\alpha(b-1)b^{2/d}}{ b^{2/d+1}-b}\rightarrow\alpha\frac{b-1}{b\log b}\text{ as $d\to\infty$}\]

\[\sum_{j=0}^{\frac{d}{2}-1}\alpha^{2}b^{-\frac{4j}{d}}=\frac{\alpha^{2}(b^{2}-1 )b^{4/d}}{b^{4/d+2}-b^{2}}\rightarrow\alpha^{2}\frac{b^{2}-1}{b^{2}\log b} \text{ as $d\to\infty$}\]

\[\sum_{j=0}^{\frac{d}{2}-1}(b\beta)^{-\frac{2j}{d}}=\frac{(b\beta-1)(b\beta)^{ 2/d}}{(b\beta)^{2/d+1}-b\beta}\rightarrow\frac{(b\beta)-1}{(b\beta)\log(b \beta)}\text{ as $d\to\infty$}\]

\[\sum_{j=0}^{\frac{d}{2}-1}(b\beta)^{-\frac{4j}{d}}=\frac{(b^{2}\beta^{2}-1)(b \beta)^{4/d}}{(b\beta)^{4/d+2}-b^{2}\beta^{2}}\rightarrow\frac{(b\beta)^{2}-1 }{(b\beta)^{2}\log(b\beta)}\text{ as $d\to\infty$}\]

Substituting these into the bounds on \(\lim_{d\to\infty}C_{d},\) one achieves:

\[(\log b+\log\beta)^{-1}\left(\frac{(b\beta)-1}{(b\beta)}-\frac{(b\beta)^{2}-1 }{\pi(b\beta)^{2}}\right)\leq\lim_{d\to\infty}C_{d}^{ABF}\leq(\log b+\log \beta)^{-1}\frac{(b\beta)-1}{(b\beta)},\]

\[\alpha(\log b)^{-1}\left(\frac{b-1}{b}-\frac{\alpha}{\pi}\frac{b^{2}-1}{b^{2}} \right)\leq\lim_{d\to\infty}C_{d}^{PI}\leq\alpha(\log b)^{-1}\frac{b-1}{b}\]

From these bounds, one can see that in the setting considered within this paper, where \(b=10000\) and \(\alpha<1/4,\) the approximation of \(\lim_{d\to\infty}C_{d}\) used in the statement of the Theorem is of a high quality.

 

Based on this theoretical derivation, we return to the interpretation of our experimental results. On one hand, the experiments have shown that the model can adapt to the new sequence length with both RoPE PI (\(\alpha=1/4\) or \(\alpha=1/8\)) and RoPE ABF (\(\beta=50\)). Thus, we can conclude that the chosen hyperparameters provide a sufficient degree of approximation of RoPE images under \(b=10000\). In other words, both \(d(f,f^{RoPE+ABF})\) and \(d(f,f^{RoPE+PI})\) are small enough to allow rapid adaptation. On the other hand, comparing the expressions of \(C_{d}\) for RoPE ABF and RoPE PI, we can observe that for the values of \(\alpha=\frac{1}{4}\) or \(\alpha=\frac{1}{8}\) and \(b=10000\) that were used in our experiments, the granularity (the distance between two consecutive images of RoPE) is much lower for the RoPE PI (\(\alpha(\log b)^{-1}\approx 0.027\)) than for RoPE ABF (\((\log b+\log\beta)^{-1}\approx 0.076\)) with \(\beta=50\). We further hypothesise that the higher degree of granularity is related to the higher evaluation on the downstream tasks of the RoPE ABF variant compared to RoPE PI because it makes the task of distinguishing between the positional embedding images simpler for the model. In other words, this corresponds to the case of \(q(f^{RoPE+ABF})>q(f^{RoPE+PI})\).

Throughout this consideration we implicitly assumed that the distance between the consecutive images of an embedding is smaller than the distance between any other pair of the images. While this assumption is likely to hold true in a high-dimensional space, significantly increasing the parameter of \(\beta\) in RoPE ABF may violate this assumption due to the changed geometry of the embedding curve.

## Appendix C Length Extrapolation Results

Despite not the focus of this work, extrapolation is an important property for long context models. Extrapolation refers to a model's ability to conduct inference on input sequences that are longer than its training sequences. We evaluate how our 70B model extrapolates with two tasks:

* **Validation loss at each position**: In Figure 9(a), we visualize the average loss at each position of the 32,768 sequence length where the first 16,384 is the interpolation area (within training sequence length) and the second half is extrapolation. We use 50 batches of samples and average across them. To make plots smoother, we also take the mean of losses every 500 positions. As we can see, our 70B model with either RoPE ABF or xPos ABF maintain the loss in the extrapolation area. To contrast this, we also plot the result for Llama 2 with 4,096 context window: the loss explodes after the position goes beyond training sequence length, which suggests that Llama 2 does not extrapolate effectively.
* **Synthetic first-sentence-retrieval task**: To complement validation loss evaluation, we also test our 70B model with two different PEs on the context probing task. Unlike validation loss task where it is hard to find data samples that require very long range dependencies consistently, first-sentence-retrieval imposes a very strict requirement for models to attend with a specific length.

Figure 8: Evaluation on our 70B model’s extrapolation capabilities.

 

In Figure 8b, we visualize the results up to 32,768 where we do see some performance degradation when the model needs to extrapolate. In addition, we observe that, despite often considered as having better extrapolation properties, xPos ABF does not outperform RoPE ABF in our setting.

## Appendix D AI Safety

### Evaluation on Safety Benchmarks

Despite showing excellent performance on various of downstream tasks, large language models are prone to generating harmful, misinformative, and biased contents Lin et al. (2021); Hartvigsen et al. (2022); Dhamala et al. (2021); Ji et al. (2023). Long-context language models can process extended inputs in their context window, but at the same time, they also face a higher risk of jailbreak, especially through means such as prompt injection Greshake et al. (2023). In this section, we evaluate the safety capability of instruction fine-tuned model using three standard academic benchmarks including TruthfulQA Lin et al. (2021), ToxiGen Hartvigsen et al. (2022), and BOLD Dhamala et al. (2021), similar to Touvron et al. (2023). We focus on the largest instruction fine-tuned model variant (i.e., 70B) and compare its results with both open sourced LLMs Falcon-instruct Almazrouei et al. (2023), MPT-instruct MosaicML (2023)) and propriety LLMS (GPT-3.5, GPT-4 OpenAI (2023), Claude-2 Anthropic (2023)) in Table 16.

We observe that in general instruction fine-tuned model maintains similar safety performance compared to Llama 2 Chat and is safer and less biased compared to other open-source LLMs such as Falcon-instruct and MPT-instruct. AI safety is a complex domain and it can be extremely difficult to comprehensively evaluate all safety aspects of instruction fine-tuned model with three benchmarks. However, we hope our analysis can serve as a pilot study and provide directional signals on long-context large language models' safety performance, which are not discussed in other works on the same topic Tworkowski et al. (2023); Ding et al. (2023); Chen et al. (2023). Currently the community also lacks dedicated safety benchmarks for long-context large language model evaluation and we plan to invest in this direction in our future work.

TruthfulQARevate instruction fine-tuned model on TruthfulQA Lin et al. (2021) to benchmark its factuality. The benchmark consists of 817 questions covering 38 categories including health, law, finance, and politics Lin et al. (2021). Similar to Touvron et al. (2023), we use few-shot prompts with 6 random QA pairs for generation and then leverage two fine-tuned GPT-3 models to classify whether the generation is truthful and informative. We report the percentage of generations that are both truthful and informative as the final metric in Table 16.

ToxiGenWe measure the toxicity of instruction fine-tuned model using ToxiGen Hartvigsen et al. (2022) where we check the percentage of toxic and hateful generations against 13 minority groups. Following Touvron et al. (2023), we filtered out prompts where annotators disagree with each other on the target demographic group. We use the default ToxiGen classifier fine-tuned based on RoBERTa Liu et al. (2019) to evaluate the level of toxicity of the model's outputs. We report the percentage of toxic generations across all groups in Table 16.

BoldBias in Open-Ended Language Dataset (BOLD) Dhamala et al. (2021) is used in this work to quantify how biased the models are against people from different demographic groups. This dataset consists of 23,679 prompts extracted from English Wikipedia covering five domains including race, gender, religion, political ideology and profession with 43 subgroups in total. Following Touvron et al. (2023), we exclude prompts belonging to Hinduism and Atheism religious subgroups as they only feature 12 and 29 prompts, respectively. After generations are inferred from each model, we leverage the Valence Aware Dictionary and Sentiment Reasoner VADER Hutto and Gilbert (2014) to perform sentiment analysis with a score ranging between -1 and 1. A positive score corresponds to a positive sentiment towards the subgroup mentioned in the prompt and vice versa. A sentiment score close to 0 indicates neutral sentiment which is desired. We report the average sentiment score across 43 demographic subgroups as the final metric for BOLD in Table 16 



## Appendix E Self-Instruct Data

As described in Section 4.3, we use Llama2 Chat to bootstrap self-instruct data for instruct finetuning. In this section we describe the detailed procedure as well as providing the necessary prompts used for generating this dataset. The main challenge is that we need an automated process of generating long context instruct data with only short context models at hand. The core idea behind this is to split the long documents into chunks of texts that can fit into short model's context and apply self-instruct. We focus primarily on question answering dataset. We first split the long document into smaller chunks, and for each chunk we construct a prompt as in Figure 9 which gets fed into Llama 2 Chat to get a question-answer pair. To diversify the question types, we randomly choose between the two prompts that ask for either normal or short answers. Once we extract the question and answer from the response (using tags as required by the prompt), we can construct long question answering instruct data together with the original long document, using the templates in Figure 10 of the corresponding answer type.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & Model Size & TruthfulQA \(\uparrow\) & ToxiGen \(\downarrow\) & BOLD \(\downarrow\) \\ \hline GPT-3.5-turbo & - & 78.46 & 0.01 & 0.50 \\ GPT-3.5-turbo-16k & - & 75.15 & 0.07 & 0.49 \\ Claude-2 & - & 62.66 & 0.05 & 0.46 \\ GPT4 & - & **80.66** & 0.03 & 0.43 \\ Falcon-instruct & 40B & 57.41 & 3.3 & 0.39 \\ MPT-instruct & 30B & 42.71 & 16.85 & **0.34** \\ \hline Llama 2 Chat & 70B & 64.14 & 0.01 & 0.41 \\ \hline Llama 2 Long Chat & 70B & 60.95 & **0.00** & 0.40 \\ \hline \hline \end{tabular}
\end{table}
Table 16: Evaluation of fine-tuned LLMs on three safety benchmarks. For TruthfulQA, we present the percentage of generations that are both truthful and informative (the higher the better). For ToxiGen, we present the percentage of toxic generations across all groups (the smaller the better). For BOLD, we report the average sentiment score across 43 demographic groups (the closer to 0 the better).





**Normal Answer Prompt:**


 You are given a text chunk (delimited by triple quotes) taken from a long text. Write a question about this text and provide the correct answer. The answer needs to be based on the text. This question will later be used as a reading comprehension test over the entire document. Wrap the question and answer using XML tags (<question> and </question>, <answer> and </answer>). """ {TEXT_CHUNK} "" 


**Short Answer Prompt:**


 You are given a text chunk (delimited by triple quotes) from a long document. Based on information from the text, come up with a specific question *which can be answered in a few words or a single phrase* and provide the correct answer without explanation. The answer needs to be based on the text. This question will later be used as a reading comprehension test over the entire document. Wrap the question and answer using XML tags (<question> and </question>, <answer> and </answer>). Again, the answer needs to be short. """ (TEXT_CHUNK) """ L/INST]

Figure 10: Data templates for constructing long question-answer data. The question and answer pair is extracted from the response of Llama 2 Chat.

Figure 9: Prompts used for generating question and answer pairs by boostrapping Llama 2 Chat. We split the long documents into chunks and feed each chunk into one of the prompts with equal probability. We prompt the models to wrap the answer with XML tags, which enables more accurate answer extraction.