LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
Authors: Yukang Chen
Shengju Qian
Haotian Tang
Xin Lai
Zhijian Liu
Song Han
Jiaya Jia
Venue: ICLR 2024 oral (2024)
Keywords: Efficient fine-tuning
Long context
Large language model
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: LongLoRA is an efficient fine-tuning approach to extend the context lengths of pre-trained large language models.

Abstract: We present LongLoRA, an efficient fine-tuning approach that extends the context sizes of pre-trained large language models (LLMs), with limited computation cost.
Typically, training LLMs with long context sizes is computationally expensive, requiring extensive training hours and GPU resources. For example, training on the context length of 8192 needs 16x computational costs in self-attention layers as that of 2048. In this paper, we speed up the context extension of LLMs in two aspects. On the one hand, although dense global attention is needed during inference, fine-tuning the model can be effectively and efficiently done by sparse local attention. The proposed shifted sparse attention effectively enables context extension, leading to non-trivial computation saving with similar performance to fine-tuning with vanilla attention. Particularly, it can be implemented with only two lines of code in training, while being optional in inference. On the other hand, we revisit the parameter-efficient fine-tuning regime for context expansion. Notably, we find that LoRA for context extension works well under the premise of trainable embedding and normalization. LongLoRA combines this improved LoRA with S^2-Attn. LongLoRA demonstrates strong empirical results on various tasks on Llama2 models from 7B/13B to 70B. LongLoRA extends Llama2 7B from 4k context to 100k, or Llama2 70B to 32k on a single 8x A100 machine. LongLoRA extends models' context while retaining their original architectures, and is compatible with most existing techniques, like Flash-Attention2. In addition, we further conduct supervised fine-tuning with LongLoRA and our long instruction-following LongAlpaca dataset. All our code, models, dataset, and demo are available at https://github.com/dvlab-research/LongLoRA.

Published as a conference paper at ICLR 2024
LONGLORA: EFFICIENT FININGUNING OF LONG- CONTEXT LARGE LANGUAGE MODELS
Yukang Chen
Shengju Qian
Haotian Tang
Xin Lai
Zhijian Liu 2
Song Han 2,3
Jiaya Jia
CUHK
MIT
NVIDIA
ABSTRACT
We present LongLoRA, an efficient fine-tuning approach that extends the context sizes of pre-trained large language models (LLMs), with limited computation cost. Typically, training LLMs with long context sizes is computationally expensive, requiring extensive training hours and GPU resources. For example, training on the context length of 8192 needs 16x computational costs in self-attention layers as that of 2048. In this paper, we speed up the context extension LLMs aspects. On the hand, although dense global attention is needed during inference, fine-tuning the model can be effectively and efficiently done by sparse local attention. The proposed shifted sparse attention (S²-Attn effectively enables context extension, leading to non-trivial computation saving with similar perfor- mance fine-tuning with vanilla attention. Particularly, can be implemented with only two lines of code in training, while being optional in inference. On the other hand, we revisit the parameter-efficient fine-tuning regime for context expansion. Notably, we find that LoRA for context extension works well under the premise of trainable embedding and normalization. LongLoRA combines this improved LoRA with Atttt... LongLoRA demonstrates strong empirical results on various tasks on Llama2 models from 7B/13B to 70B. LongLoRA extends Llama2 7B from 4k context to 100k, Llama2 70B to 32k single 8x A100 machine. LongLoRA extends models' context while retaining their original archi- tectures, and is compatible with most existing techniques, like 2atst-Attention2.. In addition, we further conduct supervised fine-tuning with LongLoRA and our long instruction-following LongAlpaca dataset. All code, models, dataset, and demo are available at htt../////vllbbbrssrrrraooooaooo
Perplexity
GPU memory
Training hours
3.9
80
OOM
3.73
68.8
50
57.4
60
2.9
2.78
2.94
2.52
2.48
40
46.3
34.6
46.3
7.4
16.3
39.8
24.6
52.4
2.4
20
25.6
5.2
11.3
Context
Content 8192
16364
32768
65536
Content
-e- Full FT - LoRA
LongLoRA
Figure 1: LongLoRA closes the accuracy gap that between conventional LoRA and full fine-tuning, while still maintaining up to 1.8x lower memory cost than full fine-tuning. Furthermore, LongLoRA improves the training speed of LoRA by up to 1.8% with 52-Attn. Llama2-7B are fine-tuned to various context lengths with Flash Attention2 (Dao 2023) and DeepSpeed (Rasley et al., 2020) stage 2 and evaluated on the proof-pile (Azerbayev et al. 2022 test set in perplexity.
INTRODUCTION
Large language models (LLMs) are typically trained with a pre-defined context size, such as 2048 tokens for I.L.MA (Touvron et al. 2023a) and 4096 tokens for Llama2 Touvron al.) 2023b).

---

Published as a conference paper at ICLR 2024
Each pattern in half heads
Embedding
Normaar
Mutti-head
Lora
Norm your A
Feed Forward
w/
Combination
(a) Shifted sporse attention
(b) Low-ronk adopt
Figure 2: Overview of LongLoRA, We introduce Shifted Sparse Attention (S²-Attn) during fine-
tuning. The trained model retains original standard self-attention at inference time. In addition to training LoRA weights in linear layers, LongLoRA further makes embedding and normalization layers trainable. This extension is pivotal for context extension, and only introduces minimal number of additional trainable parameters.
However, the pre-defined size limits LLMs in many applications, like summarizing long documents or answering long questions. To resolve this limitation, some recent works (Chen et al. 2023
Tworkowski et al. 2023 Mohtashami & Jaggi 2023) train fine-tune LLMs to longer context. However, training an LLM from scratch with long sequences poses computational challenges, and fine-tuning existing pre- trained LLM is also considerably expensive. For instance, Position
2023) spent 32 A100 GPUs to extend LLaMA models from 2k to 8k context, and 128 A A100 GPUs for longer context fine-tuning. FOT (Tworkowski al. 2023) used 32 TPUs for standard transformer training and 128 TPUs for LongLLaMA, These computation resources are typically unaffordable for common researchers, which naturally leads us question: can we extend the context window of LLMs efficiently?
Interpolation (Chen et al.
One straightforward approach is fine-tune a pre-trained LLM via low-rank adaptation (LoRA) Hu et al. 2022). LoRA modifies the linear projection layers in self-attentio blocks by utilizing low-rank matrices, which are generally efficient and reduce the number of trainable parameters. However, our empirical findings indicate that training long context models in this manner is neither sufficiently effective nor efficient. terms of effectiveness, plain low-rank adaptation results in a high perplexity in long context extension, in Table 2 Increasing the rank higher value, e.g., rank = 256, does not alleviate this issue. In terms of efficiency, regardless computational cost increases dramatically self-attention mechanism (Vaswani et al. (017). As shown in Figure | even with LoRA, the training hours for the standard Llama2 model increase substantially when the context window expands.
whether LoRA is employed or not, the context size expands, primarily due to the standard
In this work, we introduce LongLoRA, an efficient fine-tuning approach that extends the context windows of pre-trained LLMss, e.g., Llama2 (Touvron et al.) (023b). LoRA (Hu al., (022) low-rank weight updates to approximate full fine-tuning. Similarly, we find that short attention is also able to approximate long context during training. We present shifted sparse attention (St-Attn) as an efficient substitute for standard self-attention. As shown in Figure 2 we split context length into several groups and conduct attention each group individually. the tokens by half group size, which ensures the information flow between neighboring groups. For example, we use A²tA with group size 2048 to approximate the total 8192 context length training. This shares high-level spirit with Swin Transformer (Liu et al. 2021)
half attention heads, we shift
Models fine-tuned via S2-Attn retain the original attention architecture during inference. This facilitates most existing optimization and infrastructure. Techniques for common LLMs also be applied to ours. For example, Flash-Attenn Aiahh-Attentio22 (Dao et al. 2022 Dao 2023) is compatible with our method in both training and inference time. The reason behind this that short attention resembles the attention scheme in the pre-training stage of LLMs. Other efficient attentions, e.g., dilated or sparse attention, have large gap to the standard style and do not work well like ours, in Table
We empirically show that learnable embedding and normalization layers are the key to unlocking long context LoRA fine-tuning, Table Embedding and normalization layers take up small
2

---

Published as a conference paper at ICLR 2024
Attention shift
Attention shift
Group
Stop
Split
Shift
repats
Pastens Parrumz
Pataamz
St.dtte
Vnto part
Figure 3: Illustration of A2tttt.. It involves three steps. First, it splits features along the head dimension into two chunks. Second, tokens in one of the chunks are shifted by half of the group size. Third, we split tokens into groups and reshape them into batch dimensions. Attention only computes in each group in while the information flows between groups via shifting. Potential information leakage might be introduced by shifting, while this is easy to prevent via small modification on the attention mask. We ablate this in the variant in Section B.3 the appendix.
proportion parameters in the entire LLM. For example, embedding has (< 2%) parameters, and normalization has (S 0.004%) parameters in Llama2 7B. This ratio decreases for even larger LLMs.
In experiments, we show that LongLoRA is effective and efficient. We present experimental results of extending the context window for Llama2 7B, 13B, and 70B. Following the experimental settings of Position Interpolation (Chen et al. 2023), we fine-tune models with proper position embeddings. The trained models achieve comparable performance the full-attention and fully fine-tuned results, while the computational cost is much less as shown in Figure I LongLoRA can fine-tune Llama2 7B up to 100k context, 70B model up to 32k, on single 8x A100 machine.
In addition, we present solution for supervised fine-tuning (SFT) with our self-collected long instruction-followingg dataset, LongAlpaca. Our LongLoRA models are further fine-tuned with long questions and the corresponding answers. We design various types of questions for technical papers, science fiction, and other books. SFT is important for improving the chat ability of LLMs. We introduce our SFT settings in Section B.6 in the appendix.
RELATED WORK
Long-context Transformers. A large body of research has been developed to increase the context length transformers. Some of these approaches are retrieval-based (Karpukhin et al. 2020 Izacard et al. 2022 Guu et al.) 2020), which augment language models via fetching related documents and including the retrieved results into contexts. Our work complementary to these works, as our attention mechanism is unmodified during inference. Many works modify multi-head attention be
approximated ones (Wang et al. 2020 Beltagy et al.) 2020 Zaheer et al. 22020: Kitaev et al. 2020
Bulatov al. 2022 Ding et al. 2023 Qiu al. 2020). They alleviate the quadratic complexity of
the sel-tteention computation. For example, Longformer (Beltagy et al. 2020) and BigBird (Zaheer
al.
2020) use sparse attention handle long sequences. Other works (Wu et al.. 2022 Bulatov
et al.
2022) utilize memory mechanisms
compression past inputs, to look up relevant tokens.
One limitation of these works is that these compressions have large gap to full attention, making it infeasible to fine-tune pre-trained L.MSs. Although our work also involves an approximation of attention mechanism, it has a similar shape and a small gap to standard attention. This enables fine-tuning pre-trained LLMs on Attn and maintain full attention during inference.
Long-context LIMss. LLMS are typically pre-trained with a pre-defined context length, such as 2048 for LLaMA (Touvron et al. 2023a) and 4096 for Llama2 (Touvron et al. 2023b). Training L.MMs with long context from scratch is prohibitively expensive for most researchers. Recently, several works have tried to extend the context length of LLMs via fine-tuning, Position Interpolation (Chen
et al. 2023) modifies rotary position encoding (Su et al. 2021) 1) and extends the context length of LLaMA M 32768. Focused Transformer (Tworkowski et al., 2023) utilizes contrastive learning to train LangLLaMA. Both of them rely on full fine-tuning which is computationally expensive
(128 100 GPUs / 128 TPUv3 for training). Landmark attention (Mohaashami & Jaggi
2023
3

---

Published as a conference paper at ICLR 2024
Table Effectiveness of S²-Attn under different context lengths. 'Short' means 1/4 of the target context length, while 'Long' equals to the target context length. Models are fully fine-tuned upon a Llama2 (Touvron et al. 2023b) model with 7B parameters on the RedPajama (Computer) 2023 dataset. Results are tested perplexity on PG19 (Rae et al. 2020) validation split.
Training
Target Context Length
Setting
Position Embedding
Attention
Shift
8192
16384
32768
Full Attn
Long
8.02
8.05
8.04
Short Attn
Pl (Chen al. 2023
Short
x
8.29
8.83
9.47
-
Short
v
8.04
8.03
8.08
efficient approach, but somewhat lossy. It compresses long context inputs into retrieved tokens. Our method saves substantial fine-tuning costs, while preserving the quality of the original attention. Ours maintain full access to the entire input via unmodified attention during inference.
Some literature focuses on the position embedding modification of LLMs for long context extension, including Position Interpolation (Chen et al.) 2023), NTK-aware (ntk 2023), Yarn (Peng et al. 2023 positional Skipping (Zhu et al. (023). and methods based on out-of-distribution analysis (Han et al.
2023). Our method focuses on efficient fine-tumi tinettunig and retaining the original architecture during inference, which is orthogonal to these position embedding methods.
Efficient Fine-tuning. This work based on LoRA (Hu et al. 2022). a classical efficient fine-tuning approach. In addition to LoRA (Hu et al.] 2022). ) there are many other parameter-efficientt fine-tuning methods, including prompt tuning (Lester et al. 202T), prefix tuning (Li Liang [02TT, hidden state tniin (Liu et al., 2022). bias tuning (Zaken et al. (022), and masked weight learning (Sung et al.) 202II. Input-tuning (An et al. 2022) introduces an adapter to tune input embedding. Although the input embedding layers also trainable in ours, this is not enough for long context extension. We make a comprehensive analysis on layer types in experiments, in Table Existing work (Chen et al. 2022
shows sparse masks effectively save training costs and avoid performance drops.
LONGLORA
3.1 BACKGROUND
Transformer: LLMs are typically built with transformers. Taking Llama2 (Touvron et al. 2023b) for example, shown in Figure an LLM model consists of an embedding input layer and number of decoder layers. Each decoder layer comprises a self-attention module. It maps input features into set of queries, keys, and values {q, k v). via linear projection layers with weight matrices (W We. W.). Given {g. k. v), it computes the outputs o as
= softmax(gk"""
The outputs are then projected by linear layer with weight matrix W.. And MLP layers followed. Before and after self-attention modules, layer normalization (Ba et al. 2016) is applied. A final normalization is conducted after all decoder layers.
For long sequences, self-attention struggles with computation cost, which is quadratic to the sequence length. This dramatically slows down the training procedure and increases GPU memory costs.
Low-rank Adaptation. LoRA (Hu et al. models have a low intrinsic rank during adaptation. For a pre-trained weight matrix W E dR it updated with a low-rank decomposition W + AW W + B.A where B E dæ and A E grik The rank r < min(d, k). During training, is frozen with no gradient updates, while A and are trainable. This is the reason why LoRA training is much more efficient than full fine-tuning.
2022) hypothesizes that the weight updates in pre-trained
In the Transformer structure, LoRA only adapts the attention weights (W WR, Wo, W.) and freezes all other layers, including MLP and normalization layers. This manner is simple and parameter- efficient. However, we empirically show that only low-rank adaptation in attention weights does not work for long context extension.
4

---

Published as a conference paper at ICLR 2024
Algorithm 1: Pseudocode of S²-Attn in PyTorch-like style.
batch size; sequence length or number of tokens; G: group size;
# H: number of attention heads; D; dimersion of each attertion head
gkv shape K, 3, D), projected quezies, keys, and values
# Key Line 1: split gkv an H into chunks, and shift 0/2 on N
qkv eat (qxv.chums 12, 3) [0], gkv.chunk 12, 3) [1J.roll 11), 31 G,
stendard self-attention Eunction out = se f_attr (qkv!
out in shape (B, K, H, DI
Key Line 2: split out on H into 2 chucks, and then roll back G/2 on N eat (out, chunk 2) [0], aut.chunt (2, 2) [1].rell (G/2, 1)1, 2)
cat: concatenation; chunk: split into the specified number of chunks; roll roll the tensor along the given dimension.
3.2 SHIFTED SPARSE ATTENTION
Standard self-attention costs O computations, making LLMs on long sequences high memory cost and slow. To avoid this issue during training, we propose Shifted Sparse Attention (S*-Atun), as shown in Figure In the following, we make a pilot study and explain our design step by step.
Pilot Study. Table I we build up a standard baseline that is trained and tested with full attention and fine-tuning, which presents consistently good quality in various context lengths. The first trial is to train with short attention, only pattern in Figure 2 As we know for a long context, the high cost mainly comes from self-attention modules. Thus, in this trial, since the input is long, we split into several groups in self-attention. For example, the model takes 8192 tokens as input in both the training and testing stages, but self-attention conducted in each group with a 2048 size. The group number is 4, as ablated in Section [B.2 in the appendix. This pattern is efficient but still does not work in a very long context, as shown Table T The perplexity becomes larger as the context length increases. The reason behind this is that there is no information exchange between different groups.
To introduce communication between groups, we include a shifted pattern, as shown in Figure 2 We shift the group partition by half group size in half attention heads. Taking the overall 8192 context length for example, pattern 1, the first group conducts self-attention from 1 to 2048th tokens. In Pattern 2, the group partition is shifted by 1024. The first attention group begins from 1025th and ends at " tokens, while the first and the last 1024 tokens belong to the same group. We use patterns 1 and 2 in each half self-attention heads respectively. This manner does not increase additional computation costs but enables the information flow between different groups. We show that it gets close the standard attention baseline in Table 1
Consistency to Full Attention. Existing efficient attention designs can also improve the efficiency of long-context LLMs. However, most of them are not suitable for long-context fine-tuning. Because, these transformers (Qiu et al. 2020; Child et al. 2019), designed for training from scratch, have gaps to the standard full attention, which is used in pre-training. In Table 6 we show that A not only enables efficient fine-tun theett but also supports full attention testing. Although other attentions can also be used in long context fine-tuning, models must be tested with the attention used during fine-tuning. Shifting prevents models from being over-fitted specific attention patterns.
Easy Implementation. A²-Attn is easy to implement. It involves only two steps: (1) shifting tokens in half attention heads, and (2) transposing features from token dimension to batch dimension. Two lines of code are enough. We provide a P PyTocc-sttyle code in Algorithm|
3.3 IMPROVED LORA FOR LONG CONTEXT
LoRA (Hu et al. 2022) is efficient and popular manner for adapting LLMs to other datasets. It saves much trainable parameters and memory cost, compared to full fine-tuning. However, adapting LLMs from short context length to long is not easy. We empirically observe obvious gap between LoRA and full fine-tuning tunint.nng. As shown in Table [ the gap between LoRA and full fine-tuning grows as the target context length becomes larger. And LoRA with larger ranks cannot reduce the gap.
5

---

Published as a conference paper at ICLR 2024
Table Finetuning normalization and embedding layers crucial for low-rank long-context adaptation. Llama2 7B (Touvron et al., 2023b) models with the proposed S²tAttn are trained on the RedPajama [Computer 2023) dataset. The target context length 32768. Normal Embed' means normalization or embedding layers are trainable. Perplexity results are evaluated on PG19 (Rae et al. 2020) validation set. For long context adaptation, there a large performance gap between standard LoRA (Hu et al. 2022) and full fine-tuning. Without trainable normalization or embeddings, larger ranks in LoRA can not close this gap.
LoRA (rank)
LoRA (rank 8)
Method
Full FT
128
Norm
Embed Norm Embed
PPL.
8.08
11.44
11.82
11.92
11.96
11.97
11.98
10.49
8.29
8.12
Table 3: Perplexity evaluation on proof-pile (Rae et al. 2020) test split. A²tAtt:: Shifted Sparse Attention. LoRA#: improved LoRA. We fine-tune Llama2 (Touvron al. 2023b) in 7B and 13B model sizes on the RedPajama (Computer) 2023) dataset under 8k-32k context lengths. We show that method achieves comparable performance the full attention or full FT baselines, with better efficiency. We use the same training setting as the model evaluated on PG19 introduced in Section
(Rae et al. 2020
B.I in the appendix.
Training Context Length
LongLoRd
Evaluation Context Length
Size
- LoRA
2048
4096
8192
16384
32768
3.14
2.85
2.66
8192
3.15
2.86
2.68
3.20
2.91
2.72
7B
3.17
2.87
2.68
2.55
16384
3.17
2.87
2.66
2.51
V
3.20
2.90
2.69
2.54
2.49
32768
/
3.35
3.01
2.78
2.61
2.50
2.96
2.69
2.53
8192
3.01
2.74
2.57
/
3.04
2.77
2.60
13B
2.99
2.72
2.53
2.40
16384
/
3.03
2.74
2.55
2.41
3.04
2.75
2.56
2.42
2.33
32768
3.05
2.76
2.57
2.42
2.32
To bridge this gap, open embedding and normalization layers for training. As shown in Table 2 they occupy limited parameters but make effects for long context adaptation. Especially for normal- ization layers, the parameters are only 0.004% in the whole Llama2 7B. We denote this improved version of LoRA as LoRA+ + experiments.
4 EXPERIMENT
EXPERIMENTAL SETTINGS
Models We extend the pre-trained 7B, 13B, and 70B Llama2 Touvron et al., 2023b) models. The maximum extended context window sizes up 100k for 7B models, 65536 for 13B models, and 32768 for 70B models. The position indices for these models are re-scaled with Position Interpolation (Chen al. 2023).
Training Procedure We follow most training hyper parammeas in Position Interpolation (Chen et al. 2023), except that our batch size is smaller as we use a single 8x A100 GPUs machine in some cases. All models fine-tuned via the next token prediction objective. We use AdamW (Loshchilov Hutter 2019) with 3 = 0.9 and 32 = 0.95. The learning rate set to 2 x 10-5 for 7B and 13B models, and -0-5 for 70B models. We also linear learning rate warmup. The weight decay is

---

Published as a conference paper at ICLR 2024
Table 4: Maximum context length that we can fine-tune for various model sizes on single 8x A A00 machine. We use the same training and evaluation settings as in Table [] We use Flash- Attention2 (Dao (023) and DeepSpeed (Rasley et al. 2020) in stage 3 during fine-tuning. With LongLoRA, the maximum context length for 7B, 13B, and 70B models are 100k, 64k, and 32k respectively. Evaluation on PG19 (Rae al. 2020 Section B. in the appendix.
Training
Evaluation Context Length
Size
Context Length
2048
4096
8192
16384
32768
65536
100,000
7B
100,000
3.36
3.01
2.78
2.60
2.58
2.57
2.52
13B
65536
3.20
2.88
2.66
2.50
2.39
2.38
70B
32768
2.84
2.57
2.39
2.26
2.17
Table 5: Topic retrieval evaluation with LongChat (Li et al. 2023). We compare our model to other open-source long-context LLMs. This task involves retrivingg target topics from very long conversation with around 3k, 6k, 10k, 13k, and 16k context lengths. As some questions in the evaluation set longer than 16k, our model fine-tuned upon Llama2 13B. It achieves comparable performance to the state-of-the-art LongChat-13B (Li et al. 2023) with a lower fine-tuning cost.
Evaluation Context
3k
6k
10k
13k
ChatGLM2-6B (Du el al. 2022
0.88
0.46
0.02
0.02
0.02
MPT-300-chal (Team. 202.a
0.96
1.0
0.76
MPT-7B-storywnii Team 2023b
0.46
0.46
0.28
0.34
0.36
LongChat-133B (Li et al. 2023
1.0
1.0
1.0
0.98
0.9
Ours-13B
1.0
0.98
0.98
0.98
0.94
zero. We set the per-device batch size as and gradient accumulation steps as 8, which means that the global batch size equals 64, using 8 GPUs. We train our models for 1000 steps.
Datasets We use the Redpajama (Computer) 2023 dataset for training. We evaluate the long- sequence language modeling performance of our fine-tuned models on the book corpus dataset PG19 (Rae al., 2020) and the cleaned Arxiv Math proof-pile dataset (Azerbayev et al. (022). We use the test split of PG19 (Rae et al.] (202)). consisting of 100 documents. For the proof-pila dataset, also the test split of it for evaluation. We follow Position Interpolation (Chen et al. 2023) for proof-pul pile data processing. We evaluate perplexity by using a sliding window approach with
S 256, following (Press al. 2022).
4.2 MAIN RESULTS
Long-sequence Language Modeling. In Table we report the perplexity for our models and baseline on proof-pile (Azerbayev et al.) 2022) and PG19 datasets. Under certain training context lengths, our models achieve better perplexity with longer context sizes. This indicates the effectiveness of our efficient fine-tuning method. In Table for the same training and evaluation context length cases, the perplexity decreases as the context size increases. By increasing the context window size from 8192 to 32768, for the Llama2 7B model, we observe that the perplexity gets better from 2.72 to 2.50 by -0.22. For Llama2 13B model, we observe that the perplexity reduces by -0.28.
In Table 4 further examine the maximum context length that A100 machine. We extend Llama2 7B, 13B, and 70B to 100k, 65536, and 32768 context length respectively. LongLoRA achieves promising results on these extremely large settings. In addition, we find some perplexity degradation on small context sizes for the extended models. This is a known limitation of Position Interpolation (Chen et al. 2023)
fine-tune
single 8x
Retrieval-based Evaluation. We conduct experiments retrieval in long contexts. In Table 5 we compare our model with other open LLMs on the topic retrieval task introduced in LongChat (Li et al. 2023). This task is to retrieve the target topic from very long conversation, with lengths varying from 3k, 6k, 10k, 13k, to 16k. As some questions in LongChat (Li al., (023) are longer than 16k, fine-tuned Llama2 13B with context length of 18k. The training cost is similar to that for 16k.
7

---

Published as a conference paper at ICLR 2024
Passkey Retrieval Accuracy
100%
60%
20%
10k
12k
14k
16k
18k
20k
26k
28k
30k
32k
34k
36k
38k
40k
44k
46k
48k
Liuna2
7B
Llaama2 (extended
Ours 7B
Ours (extended
Figure Accuracy comparison passkey retrieval between Llama2 7B and our 7B model fine-tuned on 32768 context length. Our model presents no retrieval accuracy degradation until 33k or 34k, which exceeds the context length. can further enhance its capability of long sequence modeling through a straightforward extension of position embeddings, without additional fine-tuning.
Perplexity
Perplexity along fine-tuning steps
16
15:82
12
10
8.63
8.15
8.14
8.12
8.11
8.1
8.08
8.04
8,02
8.17
8.07
8.06
8.03
7.96
7.95
7.94
100
200
300
400
500
600
700
900
1000
Fine-tuning steps
Full FT
LoRA*
Figure Ablation on tine-tuning steps in both full fine-tuning and LoRA* We fine-tune Llama2 (Tuu
vron et al. 2023b) 7B with the proposed A?ttttn. The target context length is 8192. We use RedPa- jama (Computer] 2023) for training and PG19 (Rae et al., 2020) validation set for perplexity testing. Full fine-tuning converges faster than LoRA* + at the beginning, but the final performance gap is small.
Our model achieves comparable performance LongChat-13B (Li et al.
2023 the state-of-the-ar
model in this task. Unlike LongChat- 13B (Li et al. 2023), which is fully fine-tuned on self-collected long context conversation text, our model is efficiently adapted on RedPajama (Computer (003) via next-token generation. Our model even slightly outperforms LongChat-133 13B in the 16k evaluation.
In Figure we present the passkey retrieval accuracy of our model, following Landmark Atten- tion Mohtashami & Jaggil 2023). This task has also been adopted by other literature (Chen et al.) 2023 Tworkowski et al., 2003). In this task, the models need to find random passkey hidden in long document. We show the document format is in Section A.2 in the appendix. We study Llama2 7B (Touvron et al., 2023b) and our LongLoRA model which fine-tunes Llama2 7B with 32768 context length. We test the passkey retrieval accuracy from 1k to 34k, with interval of roughly Ik (as the sentence length can not precisely controlled). For each document length, we test the model 10 times with different random passkey values. Our model achieves reasonable passkey retrieval accuracy until 33k or 34k. Without further fine-tuning, We modify the max position embeddings to 48k in the position interpolation, which is the Ours 7B (extended PI) in Figure We show that this model can handle longer documents by simply extending the position interpolation. As the dashed orange line Figure 4 the model, fine-tuned ability (60%-90% accuracy) in the range of 33k to 45k. Even with the position interpolation extended, Llama2 7B suffers from a sharp accuracy degradation (dashed blue line) after the 4k context length.
32k context length, presents moderate retrieval
4.3 ABLATION STUDY
In this section, we introduce ablation studies on the number fine-tuning steps and attention patterns. Other experimental results including ablations on group sizes, attention variants, and efficiency analysis are Section B in the appendix.
Ablation on Fine-tuning Steps. We report the relationship between perplexity and fine-tune fine-tuning steps for Llama2 7B model extending to the 8192 context length the PG19 validation set, in
8

---

Published as a conference paper at ICLR 2024
Table 6: Comparisons among A'tt and alternative attention patterns during fine-tuning. We adapt Llama2 7B model to 32768 context length with different attention patterns and improved LoRA at training time. We include four typical efficient attention designs, e,g., shift, dilate DDing et al. 2023), block sparse (Qiu et al. 2020), stride sparse (Child et al. 2019) for comparison. "cro. heads / layers' means to swap different attention settings across attention heads or sequential layers. Taking S²-Attn as an example, 'cro. layers' is to swap between w/ and w/o shift in sequential self-attention layers. 'only P1/P2' means all attention heads use pattern (all no shift) or Pattern 2 (all shift) in Figure We visualize the patterns of different attention in Figure 7 in the appendix. For each attention pattern, we evaluate its performance under two protocols. In the first row, we use sparse attention in both training and testing. In the second row, we use full attention for testing.
Test w/
S
Dilate cro. heads
Block sparse cro. heads
Stride sparse cro. heads
Futl-Attn
cro, heads
cTD. layers
only PI.
only P2.
8.64
8.63
9.17
9.64
8.75
11,49 8.30
32.81 24.03
8.12
9.70
8.39
9.81
11.78
Figure [] We see that without fine-tuning, step 0, the model has a limited long context capability, e.g., 15.82 perplexity. We show that the perplexity drops quickly. Full fine-tuning converges faster than low-rank training. They come closer after 200 steps, without large gap at the end.
Attention Patterns. In Table 6 we show the effects of different attention patterns during fine- tuning. We fine-tune a Llama2 7B (Touvron et al.) 2023b) model to 32768 context length on Redpajama (Computer] (023) datasets and evaluate the perplexity on PG19 (Rae et al., 2020) validation set. We first examine the manner of swapping among various settings. For the shift operation we used in LongLoRA, there are three choices: disabling it, shifting between sequential layers, and shifting among attention heads. We show that shifting between layers is acceptable but not the best. In addition, setting all attention heads pattern 1 or pattern 2 does not work. In addition, we empirically find that shifting left or right has little difference in performance.
We then test other types of efficient attention designs, including dilated attention (Ding et al. 2023 block sparse attention (Qiu et al.) 2020), and stride sparse attention (Child et al.) 2019). For dilated attention (Ding al. 2023). we vary the dilate rate from I to 2 evenly among attention heads. For block sparse attention QQuu et al., 2020), we use n = 4 block-wise masking matrices in attention heads and move the block left to make it causal. Stride sparse attention (Child al. 2019) contains both local and stride patterns. These settings share similar computational costs. We visualize these patterns in Figure in the appendix. These attention patterns are invented in training- training-from- scratch transformers. This experiment is to examine their capability of fine-tuning on pre-trained LLMs [Touvron et al., 2023b), toward long context adaptation. Dilated attention performs well in full fine-tuning but is not well with low-rank adaptation. Fine-tuning with stride sparse attention is harmful. They have a large gap to full attention, which is applied in the pre-training stage.
CONCLUSION
In this work, we propose LongLoRA that can efficiently extend the context length of LLMs to be significantly larger. LongLoRA has less GPU memory cost and training time than standard full fine-tuning, with minimal accuracy compromise. At the architecture level, we propose A to approximate the standard self-attention pattern during training. S2 easy to implement, requiring only two lines of code. Moreover, models trained via - retain the original standard attention architecture during inference, making most pre-existing infrastructure and optimization reusable. At the training level, we bridge the gap between LoRA and full fine-tuning with trainable normalization and embedding. Our method can extend Llama2 7B to 100k context length and 70B model to 32k context length, following dataset, LongAlpaca and conducted supervised fine-tuning with LongLoRA. We believe that LongLoRA is a general method that could be compatible with more types of LLMs and position encodings. We plan to investigate these in future work.
single 8x A100 machine. We also present long instruction-
Acknowledgement We would like to thank Xiuyu Li and Bohao Peng for the helpful discussions.

---

Published as a conference paper at ICLR 2024
REFERENCES
Ntk-aware scaled rope, 2023.
URL.
LLaMA/
comnents/1412775/1 sareaarrr_sceee_eeeeeeeeeeeeeeee scaled rope allows 1lama sasaaa_ooelss____ have/
Byeongjoo Ahn, Michael DeZeeuw, Ioannis Gkioulckas. and Aswin C. Sankaranarayanan. Neural kaleidoscopic space sculpting. In CVPR. pp. 4349-4358, 2023.
Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models, 2023.
Shengnan An, Yifei Li, Zeqi Lin, Qian Liu, Bei Chen, Qiang Fu, Weizhu Chen, Nanning Zheng, and Jian-Guang Lou. Input-tuning: Adapting unfamiliar inputs to frozen pretrained models. CoRR, abs/2203.03133 2022.
Zhangir Azerbayev, Edward Ayers, and Bartosz Piotrowski.
Proof-pile, 2022.
URL https:
//github. com/ whanaaaaaa-a-aaeeeaeee//rrrooooo
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey Hinton. abs/1607.0644 2016.
Layer normalization.
CoRR,
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du,
Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308 14508, 2023.
Iz Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long-document transformer. CoRR, abs/2004.05150, 2020.
Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev. Recurrent memory transformer. In NeurIPS, 2022.
Beidi Chen, Tri Dao, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, and Christopher Ré. Pixelated butterfly: Simple and efficient sparse training for neural network models. In ICLR, 2022.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. CoRR, abs/2306. 15595, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL https
30 300e-300vvvaaa/////
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. CoRR, abs/1904.10509 2019.
Together Computer. Redpajama: An open source recipe to reproduce llama training dataset, 2023. URL
ama- ama--ata
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. CoRR. abs/2307.08691, 2023.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In NeurIPS, 2022.
Jayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. Longnet: Scaling transformers to 000, 000, 000 tokens. CoRR, abs/2307.02486. 2023.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiczhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In ACL, pp. 320-335, 2022.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. REALM: retrieval- augmented language model pre-training. CoRR, abs/2002.088 08909, 2020.
10

---

Published as a conference paper at ICLR 2024
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple un-the-fly length generalization for large language models. CoRR, abs/2308.16137, 2023.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan and Weizhu Chen. Lora: Low-rank adaptation large language models. ICLR, 2022.
Yuanzhi Li, Shean Wang. Lu Wang,
Gautier Izacard, Patrick S. H. Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yo Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. CoRR, abss/200.00329, 2022.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP, pp. 6769-6781, 2020.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. ICLR, 2020.
Brian Lester, Rami AI-Rfou, and Noah Constant, The power of scale for parameter-officient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), EMNLP, pp. 3045-3059, 2021.
Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. How long can open-source 1lms truly promise on context length?,
otg.://///gg//o/////////// June 2023, URL http:::///nnss....rr///////22//2 06-0066- 29 longchat
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), ACL, pp. 4582-4597, 2021.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In NeurlPS, 2022.
Ze Liu, Yutong Lin. Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pp. 9902-10002, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. Peft: State- of-the-art parameter-efficient fine-tuning methods. httts:://gi hwt./ogg.///ogaaeeeceeeeoooo
peft 2022.
Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers. CoRR, abs/2305. 163/2005.16300, 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward 7.. Yang, Zachary Vitito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. NeurlPS, pp. 8024-8035, 2019.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. CoRR, abs/2309.00011 2023.
Ofrr Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In ICLR, 2022.
Xiaojuan Qi, Renjie Liao, Jiaya Jia, Sanja Fidler, and Raquel Urtasun. 3d graph neural networks for RGBD semantic segmentation. In ICCV, pp. 5209-5218, 2017.
Jiezhong Qiu, Hao Ma. Omer Levy, Wen-tau Yih, Sinong Wang, and Jie Tang. Blockwise self- attention for long document understanding. In EMNLP, volume EMNLP 2020 of Findings of ACI, pp. 2555-2565, 2020.

---

Published as a conference paper at ICLR 2024
Jack W. Rae, Anna Potapenko. Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. Compressive transformers for long-range sequence modelling. ICLR, 2020.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In KDD, pp. 3505-3506. ACM, 2020.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. CoRR, abs/2104.09864, 2021.
Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. In NeurIPS, pp. 24193-24205, 2021.
MosaicML NLP Team. Introducing mpt-30b: Raising the bar for open-source foundation models, 2023a. URL www. www. amml.cmll.... com/ htt./bllg///g 300
MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable Ilms, 2023b. URL www. www.:/osallcc cm1 com htt.cclgg/blooot.b/.........
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, Abs/2307.09288, 2023b.
Szymon Tworkowski, Konrad Staniszewski, Mikolaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Milos. Focused transformer: Contrastive training for context scaling. CoRR, abs/2307.031700 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, pp. 5998 08886600,15 2017.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. CoRR, 000.2000060478, 2020.
Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing trans- formers. In ICLR, 2022.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. In NeurlPS, 2020.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficient fine-tuning for transformer-baased masked anguage-modele In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), ACL, pp. 1-9, 2022.
Mian Zhang, Lifeng Jin, Linfeng Song, Haitao Mi, Wenliang Chen, and Dong Yu. Safeconv: Explaining and correcting conversational unsafe behavior. In ACL, pp. 22-35, 2023.
Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. Pose: Efficient context window extension of llms via positional skip-wise training, 2023.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A SETTINGS
A.1 ENVIRONMENTS
All our experiments are conducted on 8x A100 machine. We train all models using Py- Torch (Paszke et al. 2019) with the DeepSpeed (Rasley et al., 2020) and Flash-Attention2 (Dao) 2023). By default, we use DeepSpeed (Rasley et al. 2020) in stage and use stage 3 for the maximum context length experiments. Gradient checkpoint is used by default, which a common technique in the Peft codebase (Mangrulkar et al., 2022). Note that sometimes, 8x A100 GPUs might not be necessary and 3090 Ti GPUs are acceptable, like fine-tuning 7B models to 8192 context size.
A.2 FORMAT OF PASSKEY RETRIEVAL
We follow existing literature (Mohtashami & Jaggi 2023
Tworkowski al. 2023 Chen al. 2023
for the document format of passkey retrieval. The document has the following format:
There is an important info hidden inside lot of irrelevant text. Find it and memorize them. will quiz you about the important information there.
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again. (repeat M times)
The pass key is 12362. Remember it. 12362 is the pass key.
The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again. (repeat N times)
What is the pass key? The pass key is
The document length varies with the value of M and N. 12362 is the passkey number retrieve. It is randomly sampled and varies at each testing time.
B EXPERIMENTS
B. EVALUATION PERPLEXITY ON PG19 TEST SPLIT.
In Table 14 and Table 15 we present the evaluation results the PG19 test split. We use the same settings as the models on proof-pile (Azerbayev et al. 2022 evaluation in the paper. Similarly, for model trained on certain context length, as the evaluation context length increases, our models achieve better perplexity. Note that the perplexity in Table [4] and Table 15 is higher than that in the proof-pile dataset, as PG19 (Rae al. 2020) has very different writing styles.
B.2 ABLATION ON GROUP SIZES.
In Table 7 provide ablation study on the group size of the A2ttttn. We experimented with fine-tuning Llama2 7B to 8192 and 16384 context lengths via LongLoRA. The group size varies from (1/2, 1/4, 1/6, 1/8) of the target context length. For example, the group size is 1024 for 1/8 of the context length 8192. We find that the 1/2 and 1/4 settings have minor gaps to full attention fine-tuning. Group sizes less than 1/4 would be good enough. We the group size as 1/4 of the context length in experiments by default.
Table 7: Ablation on group size. We fine-tune a Llama2 7B model to 8192 and 16384 context lengths via LongLoRA and evaluate on PG19 validation set. We vary the group size of - from 11/2, 1/4, 1/6, 1/8} of the target context length. 'Full' means the standard full attention.
Context Length
Full
1/2
1/4
1/6
1/8
8192
8.02
8.04
8.04
8.10
8.16
16384
7.82
7.84
7.86
7.94
7.98
13

---

Published as a conference paper at ICLR 2024
B.3 ABLATION ON THE VARIANTS OF SA-ATTN.
In Table 8 we ablate some variants of -Attn,,, which are illustrated in Figure Variant 1 is to change the shifting direction from down to up. It shows that the shifting direction has no effect on the perplexity. One concern about Attn is that it moves the last tokens to the front into one group, which might be inconsistent with causal masks. Variant 2 uses individual groups for the shifted tokens, which ablates this concern. Variant 3 swaps the shifted and the original front tokens, which can also ablate the concern. We show that these variants present similar perplexity to ours. We suppose that although there are communications among the front and last tokens, they are originally far away from others while limited the local group. Moreover, S" is only used for fine-tuning, while we standard causal masks and full attention during inference. Variant 2 and 3 also work well but involve additional steps to ours.
Table 8: Ablation on the variants of A2ttn.. These variants are illustrated in Figure Similar to the setting in Table we fine-tune a Llama2 7B to 8192 context and evaluate on PG19 validation set.
Attn
Full
Ours
Variant
Variant 2
Variant
PPL
8.02
8.04
8.04
8.03
8.05
Table Evaluation on LongBench (Bai et al. 2023) benchmark. In each column, we highlight the highest value to be bold and the second highest value with underline.
Single- Doc QA
Multi-
Few-shot Learning
Model
Avg
Summarization
Code
Synthetic
Doc QA
GPI-3.5-Turbe
44.0
39.8
38.7
26.3
67.1
54.1
37.8
Llama2-7B-che
31.0
24.9
22.6
24.7
60.0
48.1
5.9
LongChat-vl.5-1)
34.3
28.7
20.6
26.7
60.0
54.1
15.8
Vicuna-v1.5-7B
31.9
28.0
18.6
26.0
66.2
47.3
5.5
Ours-7B 36.8
28.7
28.1
27.8
63.7
56.0
16.7
Table 10: Evaluation on LEval (An et al. 2023)
open open-ended benchmark. We compare various models
to GPT-3.5-Turbo and judge win rates via GPT-4.
Model
Win-rate
Wins
Ties
LongChat-7B (Li et al. 2023)
33.68
36
56
LongChat-vl.5-777 1l.5h- (Li et
2023
33.59
38
53
Vicuna-v 1.5-BB (Chrang et al.
2023
25.52
22
54
Ours-7B
39.06
45
60
B.4 EVALUATION ON LONG-CONTEXT BENCHMARKS.
We evaluate our method on long-contess benchmarks, LongBench (Bai et al. 2023) in Table and LEval (An et al., 2023) in Table TO We fine-tune Ilama2 7B to 16384 context length, with the supervised tinettning method and data introduced in Section [..] We compare our model with GPT-3.5-Turbo and other Llama2-based long-context models, like Vicuna (Chiang et al., (003) and LongChat (Li et al. 2023) models. It shows that our 7B model presents comparable or even better performance than these Llama2-based long-contexy models, while only takes about 4 hours, about 0.3 billion tokens, on a single 8x A100 machine.
B.5 EFFICIENCY ANALYSIS.
In Table II we break down the FLOPs of Llama2 7B (Touvron et al. 2023b) into various types of layers, including FFN feed-forward layers, Proj projection for queries, values, keys, and attention outputs, Attn self-attention computation, Others other layers like embedding, normalization, LLM head. For full attention, the proportion of Attn sharply increases the context length increases. For
14

---

Published as a conference paper at ICLR 2024
Ours Shift down
Variant Shift
Variant Scparate group
Variant Swap shifted tokens
Figure 6: Illustration the variants of our S²-Attn. Variant changes the shifting direction. Variant 2 splits the shifted tokens into one individual group. Variant swaps the shifted tokens with the original front one.
Table 11: FLOPs profiling on various context lengths. We break down the Llama2 7B model into FFN ffeed-forwate layers), Proj (projection layers for queries, keys, values, and attention outputs), Attn (self-attention kernel), and Others (e. embedding, normalization, LLM head). The ratio of attention in the overall model increases as the context length increases. -2tn reduces the FLOPs by a large margin, especially when the context length is large.
Context Length
FLOPs (T)
A2ln
Attn
Proj
FFN
Others
Total
X
35.2
143.5
8192
35.2
70.9
/
8.8
117.1
140.7
357.2
16384
70.4
141.8
4.3
/
35.2
251.7
x
562.9
996.0
32768
140.7
283.7
8.7
/
140.7
573.8
X
2251.8
3118.0
65536
281.5
567.4
17.3
562.9
1429.1
example, Attn has 24.5% the total FL.OPs the 8192 context length while it increases to 72.2% at the 65536 context length. It decreases to 39.4% when A2-Attn is used.
For the measurement of FLOPs in Table II We profiled the context stage FLOPs Llama2-7B using batch size of and various context lengths using third-party tool, torchprofile The tool traces the computation graph and sums up the FLOPs of each node the graph (e.g. Q/K/V/O projections, multi-head self-attention, fulne-connected layers, and normalization layers).
In Table 12 we compare the training cost among full fine-tuning, plain LoRA (Hu et al. 2022). and LongLoRA o.A. It records details for Figure iin the paper. The major difference between LoRA (Hu et al. 2022) and LongLoRA the A Although there are many FLOPs saving, the peak memory cost has limited difference, because of the highly optimized Flash-Attention2 (Dao 2023). In contrast, the training hour saving is relatively clear. For example, LongLoRA spends 56.6% training hours as that of LoRA in the 65536 context length.
In Table 13 we present the effects of S2-Attn without Plash-Attentions Dao 2023
LoRA- is
included in this ablation. It shows that S2-Attn achieves more speedup than that Table 12 Without the help of Flash-Attention2 (Dao 2023), the full attention baseline encounters OOM at the 16384 context fine-tuning in 8x A100 machine, while A2tAttn is sufficient for this.
B.6 SUPERVISED FINE-TUNING.
We further conducted supervised fine-tuning ours to improve their QA ability. Although the models fine-tuned with Redpajama (Computer) 2023) present good perplexities, their chat ability is limited. We collect some question-answer pairs, relating to the materials like technical papers, science
15

---

Published as a conference paper at ICLR 2024
Table 12: Efficiency comparison on training hours and GPU memory cost. We fine-tune Llama2 Tou- vron et al. 2023b) 7B model for 1000 iterations on 8x A GPUs. We set batch size GPU as and gradient accumulation steps as 8. OOM means out of GPU memory. Flash-Attention2 (Dao 2023) and DeepSpeed (Rasley et al. (020) in stage 2 are included in these experiments. LongLoRA requires significantly lower computational overhead than fine-tuning the full model. also demands fewer training hours compared to LoRA (Hu et al. 2022). Furthermore, the plain LoRA (Hu et al.
2022) fails to maintain the same level accuracy as full fine-tuning when handling longer contexts.
8192
16384
32768
65536
Training setting
Train hours
Memory (GB)
Train hours
Memory (GB)
Train hours
Memory (GB)
Train hours
Memory (GB)
Full FT
7.4
46.3
16.3
57.4
39.8
68.8
00M
LoRA
6.0
25.7
14.0
34.7
36.5
46.5
92.5
71.1
LongLoRA
5.2
25.6
11.3
34.6
24.6
46.4
52.4
69.8
Table 13:
The efficiency effects of S²-Attn without Flash-Ateention2 (Dao 2023) The fine-tuning settings are the same to Table [2] LoRA is used. Without Flash-Attention2 (Dao
2023), -
improves the training speed by 2.1x x and GPU memory cost by 1.8 8192 context length. Without -2-Attn and Flash-Attention2, Llama2 7B can not be extended to 16384 context, due to 0OM.
8192
16384
S2tAtth
Train hours 17.5
Memory (GB)
Train hours
Memory (GB)
55.5
0OM
8.2
30.3
20.8
57.1
fiction, and other books. We have already filter out any potentially harmful or negative content in our training data. The questions we designed include summarization, relationships, and characters. We build the prompt format as the following line:
Below mmaterial_tyyee Memorize the content and answer my question after the paper. imattrial_cconemn Now the material ends. {question]
(material } can "book", "paper", and others. {material caatterialccontntt// content) is the long-context content in the document. {question} is the question we design. These questions can be some commonly used they can specific to the material, like the question that is related to some roles in the book. We named our long-conter instruction following dataset as Alnaalpaca- 12k, which contains 9k long-context QAs and 3k short QAs sampled from the original Alpaca data.
ones, like summarization and limitation.
For SFT, we use the same learning rate, weight decay, and batch sizes as the context extension step. We train the models for epochs. In the following, provide some example questions and the answers from our model, in Figure & and Figure Note that these example questions are not in the training set.
16

---

Published as a conference paper at ICLR 2024
Table 14: Perplexity evaluation on PG19 (Rae 2020) test split. We fine-tune Llama2 Touvron et al.
2023b) in 7B and 13B sizes with 8192, 16384, and 32768 context lengths.
Training
LongLoRA
Evaluation Context Length
Size
Context Length
S -Att LoRA
2048
4096
8192
16384
32768
7.55
7.21
6.98
8192
7.53
7.20
7.01
7.70
7.35
7.14
7B
7.56
7.21
6.97
6.80
16384
/
7.65
7.28
6.86
7.76
7.36
7.09
7.04
7.03
32768
8.29
7.83
7.54
7.35
7.22
6,95
6.60
6,43
8192
6.94
6.63
6.45
7.03
6.73
6.58
13B
6.90
6.58
6.37
6.22
16384
7.05
6.70
6.47
6.31
7.14
6.76
6.52
6.39
6.36
32768
7.14
6.78
6.55
6.38
6.29
Table 15: Perplexity evaluation on PG19 (Rae 2020)
test split with the maximum context length that we can fine-tune on single 8x A100 machine. The Llama2 (Touvron et al. (003b) models are fine-tuned on RedPajama (Computer 2023
Training
Evaluation Context Length
Size
Context Length
2048
4096
8192
16384
32768
65536
100,000
7B
100,000
8.38
7.90
7.57
7.33
7.16
7.06
7.04
13B
65536
7.63
7.21
6.94
6.75
6.62
6.57
70B
32768
5.93
5.63
5.44
5.32
5.27
Block sparse attention
Dilated attention
Stride sparse attention
Figure 7: Illustration on alternative sparse attention patterns discussed in the paper. We modify the original block-wise sparse attention (Qiu et al. 2020) to causal. For block sparse attention, we move its right-up blocks to left down, because of the causal mask. For stride sparse attention, switch the patterns between local attention and stride attention. The local size is and the stride
interval is Vn, following (Child et al., 2019). We make sure that alternative designs have similar amount of computation cost compared with Attt...
17

---

Published as a conference paper at ICLR 2024
Below is content in the book. Memorize the content and answer question after the book. 'n The book begins. (book, content) in Now the material ends. the person Sirius Black
kill?
Based the book. Strius Black did not actually wani kill anyone. The details
1. Sirius Black was wrongly imprisoned for betraying Harry's parents and killing Peter Pettigrew.
Ours-I 3B
2. was actually Pettigeew who berayed the Potters staged own death frame Sirius Black.
3. When Sirius Black escuped from prison, he was inttially believed to be hunting down Harry kill him for revenge. actuilly
revealed Strias betraying the Potters.
Petligrew get revenge for framing and
5. Sirius Black did not actually to hamm Harry. In fact, be Harry's godfather and wanted to take Harry his name was cleared.
summary, Sirius initially wanted Petligrew framing him betmying the Potters, He did not actually want to kill Harry.
In the book, Harry Potter and the Prisoner of Azkabun, who was the person Sirius Black want to kill?
Hello! I'm here to help harmfull assumption. characker Sirux Black not person wants anyone. He character the Hamry Potter series who wrongly accused of crime and seeking to clear his name. people
your questions, and want
that the question you've asked might be based
Llama2-13B (Char)
imporant remember that making assumptions himiful and perpetuntc negative stereolypes. Instead, suggest focasing on the lacts and context of the story to understand characters their mutivations. If you have any other quesiions or concerss, please leel free to ask, and will do my best to assist you in safe and respectfal manner.
limited informarion media portnyyas
Below is some content in the book. Memarize the content and answer my question after the book. (beok content) Now the material ends. Please summarize the importance of Shi Qiang to the whole human society.
Shi Qiang plays an important role in helping Luo Ji find meaning and purpose. Some key points:
Ours-13B 13B
Ji. Zhuang brings joy and innocence It's which was empty and nimles before. She gives him motivation and hope
Shi Qiang Zheang
Qiang's final request to
carry on the Shi family lincage, implies
an important
play. It gives Luo Ji sense responsibillit and mission.
Shi Quang's sacrifice and selfless help for Luo show the good in human nature. This gives Luo Ji faich humanity and oprimism that bumans can defest the aliens.
Shi Qiang's search Zhuang Yan helps prove Lao Ji that there perfect people the contury what Lao Ji originally belioved. This encourages Luo Ji and dispels his sense of hopelessness.
summary, Qiang finds, represents hops and light in Lue It's life. Together they help give Luo Ji newfound sonse of purpose and belief that can carry out and affect the fate of human civilication.
catalyst reawakens
and determination. Zhuang
Qiang
Wallfheer.
Qlang's may seem small, impact on could ripple
In the book. The Dark Forest, please summarize the importance of Shi Qiane the whole buman society.
Hello! a compassionate assistant, I'm here help you your question while ensuring a safe and respectruul conversation.
2-3B (Chat)
In The Forest, Qiang fictional character plays significant recagnize that his actions and motivations are shaped by the novel's themes of survival, power dynamics, and the human condition. Rather than focusing Shi Qiang's individual importance. may ideas explored the novel as a whole. The Dark Forest rasses questions about human nature, the consequences of power imiasaances, and the importance of cooperation the face alvensity.
story. However, important
productive discuss themes and
If you're interested in discussing the novel in a safe and respectful manner. I'd be happy to help facilitate a conversation that prioritizes inclusivity, sensitivity. and the well-being of individunls.
Figure 8: Examples on book-related questions 12 We compare the answers from our 13B model with SFT and the chat version of Llama2 13B. During the pre-training stage of Llama2, some books might be used as training data. To ablate this, we ask the Llama2 13B chat model with the book title and the question. We provide our model with the related book sections. Our model produces more structured and concise answers, referring to detailed context, than those from Llama2 without long documents.
at
Potter shd_tthe_Prieoer__.ee
/hehe///////////////eo/__/ hark Forest
18

---

annotates Published as a conference paper at ICLR 2024
Contribution
Below paper. Memorize the material answer my question after the paper. (puper/j'n in Now the material ends. What are the main contributions and novelties of this work?
paper proposes an
Relation Leaming frumework The Action Units R Relation Transformer (ART) that models the relations between different facial action units (AU) AU AU-annsttic putches and the AU-specific regions. This helps cupture intru intru-face relations that are useful for forgery detection.
parts:
1.
2. The Tampered AU Prediction (TAP) that tampers the AU-related regions and provides Local Tampering Supervision. This helps improve the generalization ability of the model to unseen manipulation methods.
The key contributions are:
1. Modeling the relations between different AUs at the AU-agnostic patches using the ART encoder. This helps capture intru face relations that useful forgery detection.
2. Generating chullenging pseuda samples for the model to leurn by tampering the AU-related regions using the TAP model unseen manipulation methods.
process. helps improve the genenlization ability
3. Achieving state of aheee--f-hhh---- performance cross datuset and musipuunipllaiion evaliations, demonstrating the effectiveness and genendization of the proposed framework.
Providing qualitative visualizations how different regions madified the manipulation process.
tampered regions using predicted patch tokens, helps interpret
process that generntes challenging pseudo samples, which together improve the generalization the deepfake detection model.
summary, contributions the proposed encoder
intra-
Limitation
Below paper. Memorize the material answer my question after the paper. [paper]i in Now the material ends. What are limitations proposed 3DGNN method?
Possible
1. 3DGNN relies on graph construction of this graph construction affect the performance of the model. The graph construction method used may not be optimal for scenes.
points nearest neighbors. accumnyy
3DGNN model uses a number propugation sleps. optimal number propagation steps may vary different scenes and objects. Using a fixed number of steps not be the best approach.
3. The 3DGNN model striggles with objects have similar depths or complex shapes, as discussed in the failure case amalysis. mudel's performance relies on the geometric context, which may not be objects.
for some types
4. The model requires depth information from un RGBD sensr, which limits its applicability to scenes where depth data available. The model camnot be directly applied to 2D RGB images.
5. The model performance may degrade for small objects, shown in the annlysis of performance gain by object size. The model seems betier suited for medium and large objects.
Summarization
Below paper. Memorize the muterial answer my question after the paper. [piper]] in Now the material ends. Please summarize the paper in paragraph.
paper proposes unsafe spans in utterances and provides safe altemative responses to replace unsale responses. The annotated unsafe spans help explain why an uttennne is unsafe and provide guidance for generating sufe responses, The datasel contains unsafe spans, unsafe responses, safe altemnttvve responses compares SAFECONV with other dalasets and shows that SAFECONY is more comprehensive SAFECONV demonstrates identifying unsafe spans can well explain detection of unsafe utterances, rewriting unsafe responses with context mitigate large propertion of unsale behavior in chatbots. The dataset and models released to advance the reseurch of conversational safety.
called SAFECONV
conversatonal safety.
dialogues from social media platforms. The paper
Figure 9: Examples on paper (Ahn al. 2023 et al. 2017 Zhang et al., 2023) and questions related to contributions, limitations, and summarizations.
19


Reviews:
[{"review_id": "yHUD09zCLP", "reviewer": "Reviewer_7K1u", "summary": "The authors propose a new method for adapting pretrained large language models (LLMs) to longer sequence lengths with a focus on efficiency. Prior works are either costly, due to requiring full fine-tuning of the language model or loose performance. The authors show that combining Low Rank Adaptation (LoRA) with sparse local attention provides improves efficiency while preserving performance. For LoRA, the authors note that a simple and cheap modification to LoRA, un-freezing embedding and normalization layer parameters, can prevent LoRA from loosing performance as sequence lengths increases. For sparse local attention, they employ a simple heuristic of splitting attention into independent groups of 2048 tokens. By overlapping groups within each layer at different attention heads, they ensure information flow between groups and are able to preserve performance at a level close to the much costlier full attention.", "questions": "- Have you done experiments / ablation on optimal group size for different target sequence lengths? It seems you have derived that setting the group size to 25% of target sequence length is reasonable for 8192 sequence length, but it is unclear whether this 25% heuristic or a constant group size translates to longer sequence lengths.\n- There are multiple ways to estimate model flops. Please provide the method / formula you used for Table 10.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The authors propose an extremely simple method, that performs well and is applicable to existing pretrained models", "weaknesses": "- The authors only evaluate perplexity and retrieval setting", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LhmjbCk8CO", "reviewer": "Reviewer_FsJg", "summary": "This paper introduces a novel approach to extend the context length of transformer-based language models. The approach consists of two main ideas: 1) split the context into smaller subgroups and conduct attention in each group individually; 2) adapt the model to make use of this new attention approach via parameter-efficient fine-tuning with LoRA.\n\nThe authors conduct experiments with the Llama2 model family using models with 7B, 13B, and 70B parameters and compare their newly proposed approach to several baselines. In terms of perplexity, their proposed approach is able to maintain performance even when extending the context size by a factor of 16. \n\nBeyond language modelling, the authors evaluate their method in a retrieval setup (finding a hidden key in a long sequence of text), demonstrating its improved performance over baselines.", "questions": "**Presentation**\n\n- I had difficulties understanding Figure 3. It would help if you add indices and annotations to the matrices in this plot. Additionally, it could be helpful to draw a (visual) connection between the blue matrices on the left and the attention patterns on the right. \n- Be more consistent about the usage of $S^2$, shift short attention, LoRA+, and LongLoRA. Make it more explicit that LongLoRA = $S^2$ attention + LoRA.\n- Table 7 is a great candidate for a line plot. \n- When pointing to results in the Appendix, make sure to reference a specific section in the Appendix. \n- The \"attention patterns\" ablation feels repetitive. How is it different from the \"consistency to full attention\" discussion in Section 3.2?\n- In the section on retrieval-based evaluation you mention that your model is \"somehow\" able to handle longer context. What does this mean?\n\n**Experiments**\n\n- You mention several times that the original standard self-attention can be retained at inference time. It would be helpful to provide more details on that. Also, Table 2 is mentioned as evidence for that. It would be helpful to elaborate more about the results in this table. \n- Table 3: What about an additional baseline that trains LoRA + embeddings?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed method builds on previous work and shows strong empirical results on long lange language modelling and a retrieval task\n- The proposed approach is conceptually simple and can be implemented in a few lines of code (as demonstrated by the authors)\n- The proposed approach can be combined with existing approaches for context extension such as positional interpolation \n- The authors provide a detailed discussion of related work", "weaknesses": "- The efficiency aspect of the could could be more prominently discussed in the main body of the paper\n- The presentation of the work could be improved. See below for suggestions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ke4BKU68Fr", "reviewer": "Reviewer_Sw6W", "summary": "This paper presents a method to perform LLM context extension with less memory and wall-clock time than existing methods. Their main modifications to improve efficiency are (1) training on local rather than global attention using the shift-short attention pattern, (2) using LoRA, and (3) modifying the norm and embedding layers in addition to the self-attention and feed-forward layers. The resulting method performs similarly to full fine-tuning.", "questions": "(1) While this is somewhat outside the scope of this paper, I would be curious about comparisons to methods that involve training a long-context LM from scratch.\n\n(2) I am a bit confused why regular LoRA and LoRA+ (Table 11) use the same amount of memory. Does S^2-Attn reduce memory usage as well, or only flops?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "(1) The method seems useful and impactful, and the evaluation is thorough with strong results.\n\n(2) The authors perform very thorough ablations and isolate key design decisions (attention shift, modifying the norm & embedding layers) that enable the method to match full fine-tuning.\n\n(3) The paper is well-written.", "weaknesses": "No major weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nxauXgEcUM", "reviewer": "Reviewer_WV76", "summary": "this paper proposes some computationally efficient methods to continue finetuning a pretrained model to support longer context. The paper proposed a modification for localized attention to support longer context by shifting the subgroups during finetuning. The paper also experimented with LoRA for long-context adaptation.", "questions": "1. Have you tried evaluating on any generative tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. the paper is well written and easy to follow. the proposed approach is a simple method that can adapt LLM for longer context without too much compute.\n2. the paper has good ablation to show that LoRA on embedding and normalization is important for long-context adaptation.", "weaknesses": "1. the paper only evaluated on retrieval and perplexity. It would be good to evaluate on other generative tasks that require longer context.\n2. the improvement on perplexity doesn't seem super consistent in Table. 4", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)