With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generation
Authors: Yan Wang
Dongyang Ma
Deng Cai
Venue: COLM (2024)
Keywords: Inference-Time Training; Long Context;
TL;DR: Temp-Lora embeds the infinite-long-context information directly into the model's parameters via a temporary Lora module, thus substantially enhances generation quality for long texts

Abstract: Long text generation, such as novel writing and discourse-level translation with extremely long contexts, presents significant challenges to current language models. Existing methods mainly focus on extending the model's context window through strategies like length extrapolation. However, these approaches demand substantial hardware resources during the training and/or inference phases.

Our proposed method, Temp-Lora, introduces an alternative concept. Instead of relying on the KV cache to store all context information, we embeds this information directly into a temporary Lora module. In the process of long text generation, this module is progressively trained with text generated previously. This approach not only efficiently preserves contextual knowledge but also prevents any permanent alteration to the model's parameters given that the module is discarded post-generation.

Extensive experiments on the PG19 language modeling benchmark and the GuoFeng discourse-level translation benchmark validate the effectiveness of Temp-Lora. Our results show that: 1) Temp-Lora substantially enhances generation quality for long text, as indicated by a 13.2\% decrease in perplexity (PPL) on a subset of PG19, and a 29.3\% decrease in PPL along with a 113.2\% increase in BLEU score on a subset of GuoFeng, 2) Temp-Lora is compatible with and enhances most existing long text generation methods, and 3) Temp-Lora can greatly reduce computational costs by shortening the context window. For example, we can ensure a moderate improvement in generation quality (a decrease of 3.8\% in PPL) while enabling a 51.5\%  memory usage reduction and a 60.0\% decrease in latency for inference.

Published as a conference paper at COLM 2024
With Greater Text Comes Greater Necessity:
Inference-Time Training Helps Long Text Generation
Yan Wang" DM' Deng Cai
Independent Researchers
yanwang. www.ang.bandeen@@mml.commm thisisicykcddggmil.com
Abstract
Long text generation, such as novel writing and discourse-level translation with extremely long contexts, presents significant challenges to current language models. Existing methods mainly focus on extending the model's context window through strategies like length extrapolation. However, these approaches demand substantial hardware resources during the train- ing and/or or inference phases.
Our proposed method, Temp-Lora, introduces an alternative concept. In- stead of relying on the KV cache store all context information, we embeds this information directly into temporary Lora module. In the process of long text generation, this module is progressively trained with text gener- ated previously. This approach not only efficiently preserves contextual knowledge but also prevents any permanent alteration to the model's pa- rameters given that the module is discarded post-generation.
Extensive experiments on the PG19 language modeling benchmark and the GuoFeng discourse-level translation benchmark validate the effectiveness of Temp-Lora. Our results show that: 1) Temp-Lora substantially enhances generation quality for long text, as indicated by a 13.2% decrease in per- plexity (PPL.) on subset of PG19, and 29.3% decrease in PPL along with a 113.2% increase in BLEU score on a subset of GuoFeng, 2) Temp-Lora is compatible with and enhances most existing long text generation methods, and 3) Temp-Lora can greatly reduce computational costs by shortening the context window. For example, we can ensure a moderate improvement in generation quality (a decrease of 3,4% in PPL) while enabling 51.5% memory usage reduction and 60.0% decrease in latency for inference.
1 Introduction
Long text generation has become increasingly important in a variety of real-world applica- tions, ranging from creative writing assistance (Shi et al., 2022), chat-style AI assistant (Ope- nAI, 2023) to generative agents (Park et al., 2023). However, the generation of coherent and contextually relevant long text poses significant challenges to language models (LMs), particularly in terms of understanding and maintaining contexts that are longer than the model's 's pre-defined context window size.
Existing methods, including those based on length extrapolation (Press et al., 2022; Su et al., 2023) and context window extension (Chen et al., 2023b; Han et al., 2023; Dao et al., 2022; Peng et al., 2023; Chen et al., 2023a), aims to store extensive text information within the KV cache, thereby improving the model's long text comprehension. However, they demand significant hardware resources during training and or inference. Consequently, in many applications where LMs are frequently queried for long text processing, users often resort to other strategies such as retrieval summarization to reduce the cost (Park et al., 2023).
In this paper, we propose an alternative method, Temp-Lora, which stores context infor- mation in the model parameters instead of the KV cache. The core idea of our approach is
Equal Contribution

---

Published as a conference paper at COLM 2024
Input
Prefix
Generate
Training
Chunk
Lora,
Generate
Outpu g
Training
Chunk;
Lora
LM
Generate
<QQ
Lora, T
Generate
Chunk
Complete
Delete
Figure The framework of long text generation with Temp-Lora
extremely simple: we store the context information in temporary Lora module (Hu al., 2021) that only exists during long text generation. We update this module in a streaming fashion during the generation process, using the previously-generate data, thereby achieving the goal of storing knowledge in the model parameters. Once inference is complete, this module is discarded to avoid permanently impacting the model's parameters. This approach allows us to efficiently store nearly infinite context information without extending the context window.
content training
We evaluate Temp-Iora on two benchmark datasets, PG19 (Rae et al., 2019) and GuoFeng (Wang al., 2023b). We evaluate Temp-Lora across models with different context window sizes and find that Temp-Lora substantially reduces their PPL on long text with large margin (-13.2% on PG19 and -29.3% on GuoFeng). This trend becomes increasingly pronounced the context length increases, which can be simply concluded "with greater text comes greater necessity". Further analysis also reveals that Tema-Lora, by shortening the maximum input length, can greatly reduce hardware consumption.
2 Temp-Lora
The Temp-Lora framework, depicted in Figure 1, presents straightforward yet innovative approach for long text generation. At the heart of this method lies the progressive training of a temporary Lora module (Hu et al., 2021), which is named Temp-Lora, on previously- generated text an auto-regressive manner. The continuous adaptation and refinement of the Temp-Lora module ensure evolving understanding of both recent and distant contexts. It is important to highlight the framework's adaptability in handling cases where the initial input lengths already substantial, such novels academic papers. In such scenarios, we may proactively pre-train the Temp-Lora module with the contexts, thereby laying a robust foundation for enhanced context comprehension in the generation process.
The algorithm is detailed in Algorithm 1. During the generation process, tokens are gener- ated chunk-by-chunk. At each chunk generation, we use the most recent Lx tokens the input X for generating the subsequent tokens (all preceding tokens beyond the distance Lx are discarded). Once the number of generated tokens hits predefined chunk size A, we initiate the training of the Temp-Lora module using the latest chunk and start the next chunk generation afterwards. It is important note that L + A S W, where W is the model's ss context window size.
For the training of the Temp-Lora module, is crucial to recognize that learning to generate the new chunk without any condition may not constitute meaningful training objective and potentially lead to significant over-fitting. address this concern, we incorporate the preceding LT tokens of each chunk into our training process, using them the input and the chunk as the output.
Cache Reuse For more efficient inference, we propose strategy called cache reuse. In the standard framework, after each update of the Temp-Lora module, need to re-compute
2

---

Published as a conference paper at COLM 2024
Algorithm 1: Long Text Generation with Temp-Lora
Input: Input 2 Input Length Lx; Training Input Xr; Training Input Length LT; Chunk size A; Learning rate a; Epoch number n; LM ④
Output: Long sequence y
1 if len(X) Lx then
Pre-train O on X' to obtain LM with an initial Temp-Lora module Oo;
3 end
4 Total token number: < len(X);
5 Temp-Lora ID: k t- 0;
6 Token number in this chunk: m <- 0;
7 Output sequence ) = X;
8 X' X[-(minnLxx len (X)) :];
9 while In Generation do
if < A then
11
111 = xx i
(X + 1 m :));
end
14
else
Chunky x = 3/1 1-mmm :); X7 = V1= + m) mm]; k++;
17
Train O₁: In Ny Out Chunky (n epochs, learning rate a);
x= - :); m 0;
end end
Destroy Temp-Lora module: O₂ < O;
the KV states of the latest Lx tokens with the updated parameters. Alternatively, we can also reuse the existing cached KV states while employing the updated model for subsequent text generation. We empirically find that cache reuse can accelerate the generation speed without compromising the generation quality.
Attention Sinks The recently proposed Attention Sink (Xiao al., 2024) method allows us to further enhance inference efficiency. By permanently storing four initial tokens as attention sinks in the KV cache, the model, after generating up to the maximum length, no longer needs to recompute the KV states. Therefore, if we simultaneously employ cache reuse and attention sink strategies, during long-text inference, we only need to continuously update the model (Temp-Lora module) without re-computing the KV states.
3 Experiments
We evaluate the proposed Temp-Lora framework using the Llama2 (Touvron et al., 2023) families and Yi-Chat-6B considering their wide adoption and popularity. Its effectiveness is evaluated in two different long text generation tasks: 1) Novel Generation; and 2) Discourse- Level Literary Translation.?
Dataset: The first dataset we adopt is a subset of the long text language modeling bench- mark, PG19 (Rae al., 2019). It is well-established benchmark that consists of more than 28K books which were published before 1919. Since we are especially concerned with the effectiveness of the Temp-Lora framework in extremely long text scenarios, and considering that some of the PG-19 data might already be included in our models' pre-training corpora,
Pleesse note that simply adding attention sink without implementing cache reuse is ineffective because, without cache reuse, each update of Temp-Lora necessitates re-computing KV states 2Codes and Data available at: https: //gi www..tt....oo.eeeeooooooo
3

---

Published as a conference paper at COLM 2024
select 100 books with the highest PPL from those whose lengths range between 200K and 600K tokens. From these, we randomly sample 40 books as our test set.
We also evaluate the effectiveness of Temp-Lora on downstream task, Discourse-Level Literary Translation, with a randomly sampled subset of GuoFeng dataset from WMT 2023 (Wang et al., 2023b;a). This subset contains 20 web novels, originally written in Chinese by various novelists and subsequently translated into English by professional translators. Their lengths (en zh) range from minimum of 84K to maximum of 370K tokens.
We split novels in the GuoFeng dataset into segments based on sentence boundaries. Each segment has approximately 512 tokens (en + zh) in terms of length. The model's input includes the current source segment along with its historic source and target text, and it is required to output the translation of current source segment.
Baselines:
We apply the Temp-Lora framework to three Llama2 variants and Yi-Chat-6B:
Llama2-7B-4K: the standard Llama2-7B model.
Llama2-13B-4K: the standard Llama2-13B model."
To-Llama2-13B-33 This a very strong baseline model that extends the context window of Llama2-7B-4K to 32K. The developers from TogetherAl followed Meta's linear interpolation (Chen et al., 2023b) and continue pre-trained with 1.5B tokens.
We believe it is much stronger than those training-free methods. 5
Mistral-7B: the standard Mistral-7B-vO.1 model with 8K context window size."
phi-2: the standard phi-2 model with 2K context window size."
qwen-7B: the standard qwen-7B model with a 8K context window size.
Yi-Chat-6B: A chat-style model trained on both Chinese and English corpus. We
will use this model in the Discourse-Level Literary Translation task. 9.
Please note that the length of most samples in our test set significantly surpass the models' context window sizes. In such instances, when the model hits the token generation limit, we slide the context window and recompute the KV states from the recent W 1024 tokens.
The primary metric is perplexity (PPL), standard measure language modeling to assess the model's prediction capability. We employ sliding window approach for PPL measurement as suggested by (Press et al., 2022). For the translation task, in addition to PPL, we also employ two common evaluation metrics used in machine translation, BLEU (Papineni al., 2002) and COMET (Rei al., 2020), for comprehensive evaluation.
Evaluation Metric:
Setup: We set the chunk size 1024 for all models. For the Llama2-7B-32K, we additionally set two wider chunk size A 2048 and 4096 to investigate the effects of chunk size on generation quality and computational cost.
The input length Lx for all models is set to be W A for maximizing the use of the models' context window size. For the Llama2-7B-32K, we notice that its PPL increases when the number of tokens within its context window approaches 32K. Therefore, we set its context window size W to 24K, which, based on our preliminary experiments, is the optimal context window size. Note that the number of context tokens may be smaller than Lx- In such cases, we simply take all context as the input. In the generation process, by default, once the token number in the context window reaches the context window size (4K and 24K for different
http://ggggnnfffeccccccccaaalaaa 2-7b
"https://hugginggaaee
htt.:/aaaccccmc.a.a co/meta hat.eetaalaammmmaaaaa222112

---

Published as a conference paper at COLM 2024
models) we re-compute the KV states taking the Lx recent tokens as input X' (i.e., no cache reuse). When we update the Temp-Lora module, the length Ly is set to 1024.
In the experiments on GuoFeng, each segment is treated the model translates a complete segment, this segment is then used to update the Temp-Lora module. We decode the translations with greedy search (repetition penalty 1.12). 10
chunk. In other words, after
All experiments done with single NVIDIA A800 GPU. We set the learning rate for all models. We use linear learning rate warmup for the first 2 chunks. We set Temp-Lora & = 64, rank = 64, dropout = 0.05. Its training is in bfloat16 format, using Deepspeed ZeRO Stage 2, Flash Attention V2 (Dao, 2023), and gradient checkpointing.
10-5 and the number of epochs
Model
0-100K
100-300K
300-500K
500K+
Avg.
Llama2-7B-IK
10.14
9.76
8.88
4.57
9.81
+TL
1024
9.79 (3.4%)
9.07 (7.0%)
8.07(9.1%)
3.96 (13.2%)
9.23 (5.9%)
Llama2-13B-4K
9.21
8.88
8.12
4.19
8.93
+TI.
1024
8.91 (3.1%)
8.31 (6.3%)
7.44 (8.4%)
3.70 (11.6%)
8.45 (5.3%)
Mistral-7B
10.58
10.16
8.83
4.32
10.20
+TL
1024
10.34 (2.2%)
9,57 (5.7%)
8.19 (7.1%)
3,74 (13.3%)
9.73 (4.5%)
Qwen-7B
18.39
17.51
14.08
17.70
+TL
1024
18.02 (2.0%)
16.44 (6.1%)
13:05 (7.3%)
16,90 (4.5%)
To-Llama2-7B-32K
10.21
9.74
8.70
4.01
9.81
+TL
1024
9.92 (2.8%)
9.15 (6.0%)
7.96 (8.4%)
3.66 (8.8%)
9.32 (5.0%)
+TL CR
1024
9.94 (2.7%)
9.16 (6.0%)
7,97 (8.3%)
3.68 (8.3%)
9.33 (4.9%)
+TL
2048
9.95 (2.6%)
9.17 (5.8%)
7.99 (8.1%)
3.71 (7.6%)
9.34 (4.8%)
+TL CR +TI.
2048
9.97 (2.4%)
9.18 (5.7%)
8.01 (7.9%)
3.75 (6.6%)
9.36 (4.7%)
4096
9.99 (2.1%)
9.21 (5.4%)
8.04 (7.6%)
3.77 (6.1%)
9.38 (4.4%)
+TL + CR
4096
10.03 (1.8%)
9.23 (5.2%)
8.06 (7.3%)
3.81 (5.0%)
9.41 (4.1%)
Io-Llama-7BB-3KK+ AS
10.35
9.91
8.88
4.20
9.98
+TL + CR
1024
10.02 (3.1%)
9.26 (6.6%)
8:08 (9.0%)
3,78 (9.8%)
9.42 (5.5%)
+TI. CR
2048
10.05 (2.8%)
9.28 (6.3%)
8.12 (8.6%)
3.85 (8.3%)
9.45 (5.2%)
+TL CR
4096
10.12 (2.2%)
9.34 (5.8%)
8.18 (7.9%)
3.93 (6.4%)
9.51 (4.6%)
Table 1: PPL of different moedls. All tokens in PG19 divided into four segments based on their position in the novel, the actual context length of the token. For instance, if token is the 160K-th token in a novel, it would be categorized into the 100-300K segment. In this table, we report the PPL of various models under different settings. The percentages in indicate the relative reduction of PPL of the model compared the base model. TL, CR, and AS are short for Temp-Lora, cache reuse, and attention sink, respectively.
3.1 Main Results
PG19: Table presents the PPL comparisons across various models with and without the Temp-Lora module on PG19. We measure the PPL. scores at different positions separately. Specifically, for a sequence x, we compute the PPLs of the following subsequences: [x]: 100K),x[1006, 300K], x/00K 500K), 500K :]]. Intuitively, In the initial segments such as 0-100K, where there is less context information, the improvement from Temp-Lora should be relatively modest. In contrast, the segments grow longer and more information falls outside the model's context window, Temp-Lora's effects should become more pronounced.
The experimental results in Table 1 confirm our hypothesis. Firstly, the augmentation of Temp-Lora leads to a significant PPL reduction for all models, where we observe an average decrease of 5.9% on Llama2-7B-44 different text segments reveals it is more notable on long text. For example, Temp-Lora augmented Llama-7B-4K achieves direct PPL reduction of 13.2% in the 500K+ segment.
An in-depth examination of Temp-Lora's impact across
For the base model Yi-Chat-6B, we have included the most recent three segments as examples in the input to better maintain consistency in translation; for the Temp-Lora augmented model, there is no need to retain examples in the input
5

---

Published as a conference paper at COLM 2024
zhZen
en2zh
Model
PPL
BLEUT
COMET
PPL.
BLEUT
COMET
Yi-6B-Chat
4.36
12.4
72.5
6.99
11.5
77.2
+TL
3.32 (-23.8%)
19.0 (53.2%)
78.6 (8.4%)
4.95 (-29.1%)
24.7
(113.1%) 82.8 (7.2%)
Table 2: PPL, BLEU, and COMET of Yi-6B-Chat with and without Temp-Lora.
Surprisingly, on segments whose context length is greater than 300K, Temp-Lora helps Llama2-7B achieve a lower PPL than the 13B model. In contrast, its effect is relatively less pronounced in the 0-100K segment, where it only reduces the PPL. by 3.6%. We may simply conclude the results as: with greater text comes greater necessity for Temp-Lora.
Experiments with the strong baseline model, Io-Llama2-7B-32K, which was fine-tuned on 1.5 billion tokens of long-con Long-context language data, demonstrate that Temp-Lora operates orthogonally to existing long text generation techniques. Even though this model has already employed the most advanced Context Window Extension and Length Extrapolation techniques, and continue pre-trained with 1.5B tokens, within the Temp-Lora augmentation, it achieves a reduction in Perplexity (PPL.) comparable to that observed in those 4K and 8K context models.
Furthermore, adjusting the chunk size from 1024 to 2048 and 4096 resulted in slight increase PPL. It not surprising, as the Temp-Lora module was trained on the data from the previous chunks. An increase in chunk size means that the information stored in the module is more distant from the current token. Indeed, the choice chunk size is critical trade-off between generation quality and computational efficiency, which will receive detailed analysis in Section 3.2.
We also discover that cache reuse almost does not cause any loss in performance. This is a very encouraging observation, as this technique can significantly reduce computational cost. Moreover, another set of experiments verifies the compatibility of Temp-Lora with the attention sink. Encouragingly, the experimental results confirmed that these two can be seamlessly integrated. With the augmentation of attention sink is even greater than its
Cache reuse and attention sink:
Temp-Lora, the reduction in PPL of Llama2-7B-32K standard version (5.5% vs 4.9 4,9%, 5.2% vs 4.7%, and 4.6% vs 4. 4.1%).
GuoFeng: Table 2 presents the remarkable impact of the Temp-Lora on the task of discourse-level literary translation. Compared to the base model Yi-Chat-6B, there are significant improvements across all metrics and all language pairs: decrease in PPL by -2388% an increase in BLEU score by +,, and a rise in COMET score by +8.4% in Chinese-to-Englishh translation; decrease in PPL by -29.1%, an increase in BLEU score by +113.1% and a rise in COMET score by +7.2% in Eigglihh-o--hhnnes experiment illustrates that the Temp-Lora module is also effective in downstream tasks.
translation. This
One might wonder why the effects are more pronounced in translation. This is attributed to the differences between tasks: open-domain novel completion is a task without standard answers, making it challenging for even an active human reader of specific novel to predict the content of the next chapter. In such cases, even if the Temp-Lora module stores substantial context information, the model can only slightly reduce the PPL of ground- truth text. In contrast, in discourse-level translation, most word translations are unique for consistency. By effectively storing these translation mappings in the Temp-Lora module, substantial improvements can be readily achieved.
3.2 Further Analysis
Contrast to Dynamic-NTK One might be curious about how the existing Length Interpo- lation methods perform in these super-long context scenarios. Table 3 presents the outcomes on the PG19 dataset after extending the Llama2-7B-4K model using Dynamic-NTK (NTK, 2023; Peng et al., 2023). Unfortunally, Dynamic-NTK are not suitable for this scenario. One may easily find that once the context window extends to more than four times its training window, PPL. will collapse directly.

---

Published as a conference paper at COLM 2024
0-100K
100-300K
300-500K
500K+
Avg.
8K
11.2
10.9
9.7
5,3
10.9
16K
22.5
20.9
10.5
24K
65.8
87.0
78.3
24.8
77.7
32K
106.3
161.0
147.1
63.3
137.1
40K
212.4
420.6
385.7
145.1
324.4
Table 3: PPL of Llama2-7B-4K with Dynamic-NTK
Context Window Size: In Section 3.1, we demonstrate the Temp-Lora framework's ability to enhance long text generation. A key observation is that certain context information is simultaneously stored within both the model's parameters (Temp-Lora) and its KV cache, leading to an overlap. For further understanding of the framework's efficiency, we gradually shorten the context window size W during inference After shortening W, we can reduce the computational cost for both models. However, this reduction results in less context information being stored in the KV cache, compelling the model to rely primarily We measure Llama-7B-32K's PPL, memory usage, and latency across these varied context window sizes W in Table 4. This experiment aims to explore the balance between generation quality and computational efficiency in different scenarios.
eliminate this overlap.
the Temp-Lora module for accessing contextual information.
We observe that models augmented with Temp-Lora consistently surpass the base model's performance, regardless of the context window size. Notably, even when we reduce the window size 1/12 of its maximum (2048), reduction in PPL from 9.81 to 9.65 also observed. Please note that as the context window diminishes, there is still a gradual increase in the model's PPL. As a whole, this trend highlights that Temp-Lora and the KV cache are orthogonal, jointly enhancing the overall performance of the model when used together.
Metric
A
Base
24K
16K
8K
4K
2K
PPI.
1024
9.81
9.32 (-5.0%)
9.33 (-4.%%
9.39 (-1.3%
9.48 (-3.4%
9.65 (-1.6%)
PPL.
2048
9.81
9.34 (-4.8%)
9.36 (-46%%
9.43 (-4.0%
9.55 (-2.7%)
PPL
4096
9.38 (-4.%%)
9.41 (4.1%%
9.50 (+3.1%)
Mem. (GB)
38.5
38.8 (+0.3%)
30.5 (-20.6%)
22.6 (-11.2%)
18.6 (5)5%%
16.7 (-56.5%)
(st.(s)
60.1
60.2 (+0.1%)
44.4(-26.0%)
28.8 (-52.0%)
24.0 (-60.0%)
23.9 (-60.1%)
Table 4: PPL. Memory Usage, and latency (1K tokens) across various context window sizes W for To-Llama2-7B-32K with Temp-Lora and cache reuse. The base model's context window size is 24K. Tokens are decoded with greedy search. Note that W should be larger or equal the sum of the chunk size and training input length, A + LT- not, the context window will be in-sufficient to include all tokens in Temp-Lora updating. The percentages in () indicate the relative PPL change of the model compared to the base model's best.
Metric
4096
2048
512
Lat (s)
1.67
2.38
3.83
1024 Mem. (GB)
18.0
17.4
17.3
Lat (s)
2.44
4.28
5.35
7.40
2048
Mem. (GB)
19.0
18.0
17.5
17.3
Lat (s)
3.59
4.97
5.88
8.46
14.02
4096
Mem. (GB) 21.0
19.0
18.0
17.5
17.3
Table 5: Cost of Temp-Lora training for To-Llama2-7B-32K The numbers in the first row represent the batch size during training. For example, if we set the chunk size 2048, the batch size as 1024, and training epochs n = 2, then updating the Temp-Lora module requires * (2048/ /0044) training steps.
Efficiency: Next, let's explore the efficiency of the Temp-Lore-augmenter model in prac- tice. When deploying the model in real-world applications, two distinct deployment strate-
7

---

Published as a conference paper at COLM 2024
gies can be considered based the availability of computational resources: 1) Cascaded: As outlined in Algorithm 1, this strategy temporarily pauses inference after generating a chunk of data. This pause allows for the Temp-Lora module to be updated with the newly generated chunk before resuming the generation process. 2) Parallelized: When abundant computational resources are available, the Temp-Lora module can be updated in parallel with the token generation process during inference. In this approach, after generating chunk, we do not pause inference. Instead, we update the Temp-Lora module using this chunk in another process. Once the Temp-Lora module is updated, we immediately replace the old one with the updated version.
With the parallelized deployment strategy, the model's memory usage and inference latency during inference, as shown in Table are nearly identical to those of the base model with the same context window size. We were surprised to find that in the most "latency-sensidee scenario, we may set 1K and W 4K, the model not only reduces PPL by 3.4% but also saves 51.5% of Memory usage and 60.0% of latency. Conversely, if we disregard computational costs entirely, then in the most "luxurious" configuration, A = 1K and W 24K, we achieve 5.0% reduction in PPL with minimal additional memory consumption (0.3 Gb) and latency (0.1 ms).
On the other hand, in scenarios of limited resources, the cascaded deployment strategy becomes preferable. In such cases, the total time required to generate a chunk equates to the sum of the model's inference latency and the time spent updating the Temp-Lora. As detailed in Table 5, we demonstrate the relationship between training latency and memory usage across various training batch sizes. Notably, training Temp-Lora is significantly faster than the inference process itself. For instance, in the "latency-sensitive" configuration, with
1K and W 4K, the average inference latency for every 1K tokens is 24 seconds, whereas Temp-Lora training takes merely 1.67 seconds. Furthermore, regarding memory usage, reassuring that the memory required for Temp-Lora training rarely surpasses that during inference. Thus, repurposing the memory allocated for inference to train the Temp-Lora module is viable, eliminating the risk of an 'Out of Memory' issue.
Lisma-78
Llmma-78
Llama-78
0.-
Llama-13
Uaama-188
53
Uama-138
Lisma-78.322
Lioma-78-322
Uama-78-32X
0.4
0.4
0.4
8.5
Num. of Epochs
Lora Rank
Learning Rate
(a)
(b)
(c)
Figure The relationship between model PPL and hyper-parameters, depicted from left to right, are (a) Epochs, (b) Lora Rank, and (c) Learning Rate. Note that the PPL at X-axis value of in each graph corresponds to the performance of the base model.
3.3 Discussion
One might find it surprising that the training cost for Temp-Lora is significantly lower than for inference, despite the widespread belief that model training generally requires more time than inference. This efficiency stems from the model's ability to leverage parallel processing techniques during the training phase, allowing it to handle the entire chunk simultaneously. In contrast, the generation phase is constrained by the autoregressive nature of the language model, necessitating a token-by-token generation approach. This fundamental difference significantly accelerates the training of chunk compared to its generation.
8

---

Published as a conference paper at COLM 2024
After summarizing the experimental results, we got some suggestions for applying Temp- Lora in real-world scenarios: 1) For applications demanding the highest level of long text generation, integrating Temp-Lora into existing models without altering any parameters can significantly enhance performance at a relatively modest cost. 2) For applications where minimal latency or memory usage is paramount, computational costs can be signifi- cantly lowered by reducing input lengths and storing context information within Temp-Lora. Under this setup, we can process text of almost infinite length (500K+ in our experiments) using fixed short window size. 3) Note that in scenarios without extensive text, for example, less than the model's window size in training, Temp-Lora is useless.
3.4 Parameter Sensitivity Analysis
One may wonder how sensitive Temp-Lora is to the training hyper-parameters. In Figure 2, we investigate the sensitivity of our model to three key hyper-parameters, Epochs, Lora Rank, and Learning Rate, on subset of PG19 that contains 3 books.
training each chunk for just one epoch consistently leads to a significant PPL reduction, where we observe average decrease of about 0.2. Expanding the number of epochs to two further reduces PPL, but excessively high epoch counts risk catastrophic forgetting, and negating the benefits by losing previously learned information.
Epochs:
Lora Rank: lora rank determines the number of parameters in Temp-Lora module. From Figure 2 (b) observe that the model is robust to this hyper-parameter. When the lora rank increases from 16 to 256, the model's PPL remains essentially stable.
Learning Rate: from Figure 2 (c) observe that the learning rate is relatively important parameter that has a significant impact on the model's performance. Intuitively, too-small learning rate may lead to less knowledge being stored in the Temp-Lora module, whereas too-large learning rate might cause the model to overfit to the current chunk.
Related Work
In recent years, numerous efforts have been made to enable language models to understand and generate longer texts (Pawar al., 2024; Zhao et al., 2023). The primary focus of these studies has been to store more information within the context window. They can be broadly categorized into three types: 1) Length Extrapolation, 2) Context Window Extension, 3) External Memory. It is worth noting that there are some overlaps between categories 1) and 2). Following Chen et al. (2023b), we categorize training-free methods under Length extrapolation, while those fine-tune-based methods are categorized under context window extension. Our Temp-Lora framework pioneers a new direction in long-te generation area, which is orthogonal to these three types and can be seamlessly integrated.
Length Extrapolation aims to find ways to process long contexts with short context windows. This "Train Short, Test Long" paradigm was first introduced in Press et al. (2022), which proposed the ALiBi position embedding method that leverages linear-decaying attention biases achieve the extrapolation of position encoding. On the top of another position embedding method, RoPE (Su et al., 2023), Methods like Position Interpolation (Chen et al., 2023b), NTK-Aware scaling (NTK, 2023), and ReRoPE (Su, 2023) make some further progresses to enable the LLM to handle unseen positions at the inference time. A recent method, SelfExtend (Jin al., 2024), attempt to extend the context window of LLMs by constructing bi-level attention information. Based on some empirical experience, although these methods can theoretically be scaled to an infinite context window length, in practical applications, extending the context window length to more than four times the size of the context window will lead to a serious degradation problem.
Context Window Extension centers expanding the LLMs' context window, enabling the processing of more tokens in one forward pass. The principle behind "Extending" might sound straightforward: train LLMs on longer texts. However, the most significant challenge lies in the training efficiency problem. Consequently, researchers are exploring ways to reduce the training costs. Their solutions range from focueed optimizations like

---

Published as a conference paper at COLM 2024
FlashAttention (Dao et al., 2022; Dao, 2023) that accelerates attention computation and reduces memory usage, to approximate attention methods (Zaheer et al., 2021; Beltagy et al., 2020; Wang et al., 2020; Kitaev et al., 2020) that trade model quality for efficiency. Additionally, Most Length Extrapolation methods (Chen et al., 2023b; Peng et al., 2023; Zhu et al., 2024) may also benefit from continual fine-tuning, where the extended context can be better utilized. Beyond algorithm-level optimization, ProLong (Chen et al., 2024a) try to tackle this long-context modeling issue from data-centric perspective, which only filters a small amount of long dependency data for extension training. Recently, researchers also leveraged LoRA for cost-effective training (Chen et al., 2024b). However, all these techniques only extend LLMs' context window to limited extent, which falls short of our paper's primary concern of handling infinite inputs.
External Memory tackles the long-contes understanding problem from different perspec- tive: It stores all necessary knowledge into a pre-computed index and only retrieves useful data as the working context (Li et al., 2022). The retrieved data can be either the supervised data (Cai et al., 2021) or chunked text (Lan et al., 2023; Xu et al., 2024). Although the perfor- Generation (RAG) continues to improve (Mohtashami & Jaggi, 2023; Tworkowski et al., 2023), its biggest challenges lie in 1) the accuracy of retrieval is still far from satisfactory, and 2) incorrect retrieval can lead to error accumulation, resulting in a decline in the quality of the generated content.
mance of Retrieval-Augmenner
Concurrent Work Our research coincides with the work of Sun et al. (2024), which proposed to add Test-Time Training (TTT) layer to the model to store the long context. The main difference between our work and theirs is that we focus on the Transformer architecture, while they primarily concentrate on the RNN architecture.
Conclusion
We proposed Temp-Lora, an alternative way for efficient long text generation. Its essence lies in training during the inference process using the generated output. It enables the storage of nearly infinite context information directly within the model's parameters, marking distinct difference from existing attention weights-based techniques.
Our experimental results across various applications, including language modeling and discourse-leve literary translation, demonstrated the profound impact of Temp-Lora. We showed that Temp-Lora not only greatly enhances the quality of long text generation but also significantly reduces computational costs. In particular, Temp-Lora led to remarkable improvements in perplexity (a reduction of 13.2% on a subset of PG19 and 29.3% of GuoFeng), as well as increases in BLEU (by 113.2%). The effectiveness of Temp-Lora becomes increasingly apparent as text length grows. When considering the implementation of Temp-Lora in your applications bear in mind this guiding principle: With Greater Text Comes Greater Necessity for Temp-Lora rule that becomes increasingly relevant in the context of extremely long text.
subset
References
Ntk-aware scaled rope 2023, URL www..r/dd....///llllallll//.oaoo (taz7//5ntkkwarr_scaledd. poppceleddroee___eeeeeeeeee rope_alppllllo.a rope_allows._seeoeoeaaoooooooooe www_s___lam_mmmess____ sttlls_ttt_________
Iz Beltagy, Matthew F. Peters, and Arman Cohan. Longformer: The long-decument trans- former, 2020.
Deng Cai, Yan Wang, Huayang Li, Wai Lam, and Lemao Liu. Neural machine translation with monolingual translation memory. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7307-7318, Online, August 2021. Association for Computational Linguistics. doi: 10.18653 (0018653/112221.ac--oo 22021..al--oong.56/ 567. URL https: http:///aaaaattotgae org org/2021 2021. acl acl-long.56t 567.
10

---

Published as a conference paper at COLM 2024
Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. Clex: Contin- uous length extrapolation for large language models, 2023a.
Longze Chen, Ziqiang Liu, Wanwei He, Yunshui Li, Run Luo, and Min Yang. Long context is not long at all: A prospector of long-dependency data for large language models. Xiv preprint ar Xiv:2400 17915, 2024a.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023b.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models, 2024b.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré, Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022.
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models, 2023.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yual Chang, Huiyuan Chen, and Xia Hu. Llm maybe longlm: Self-extend Ilm context window without tuning, 2024.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer, 2020.
Tian Lan, Deng Cai, Yan Wang, Heyan Huang, and Xian-Ling Mao. Copy is all you need. In The Eleventh International Conference on Learning Representations, 2023. URL
Huayang Li, Yixuan Su, Deng Cai, Yan Wang, and Lemao Liu. A survey on retrieval- augmented text generation, 2022.
Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers, 2023.
OpenAL. Gpt-4 technical report, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: (0.3115/1073083.107330 URL https: httpss///caaathlogo/oo org/P02-1044
Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior, 2023,
Saurav Pawar, S. M Towhidul Islam Tonmoy, S M Mehedi Zaman, Vinija Jain, Aman Chadha, and Amitava Das. The what, why, and how of context length extension techniques in large language models a detailed survey, 2024.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models, 2023.
Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022.
11

---

Published as a conference paper at COLM 2024
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arxio preprint, 2019. URL https://aaxxv.oorg/aa// 1911 111991.00 1005577.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. Online, November 2020, Association for Computational Linguistics. doi: 10.18653 / /v/ /1/22000emmllmmaal 213. URL https: /att:///hllaatooogyo..o org/2020. http/2200.eemmmmmmaamm emnlp-main. 213.
Shuming Shi, Enbo Zhao, Duyu Tang, Yan Wang, Piji Li, Wei Bi, Haiyun Jiang, Guoping Huang, Leyang Cui, Xinting Huang, Cong Zhou, Yong Dai, and Dongyang Ma. Effidit: Your ai writing assistant, 2022.
Jianlin Su. Rectified rotary position embeddings, 2023. 9708.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023.
Yu Sun, Xinhao Li, Karan Dalal, Chloe Hsu, Sanmi Koyejo, Carlos Guestrin, Xiaolong Wang, Tatsunori Hashimoto, and Xinlei Chen. Learning to (learn at test time), 2024. URL https://aaxxv.org//bbb/ 2300b/
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Szymon Tworkowski, Konrad Staniszewski, Mikotaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Milos. Focused transformer: Contrastive training for context scaling, 2023.
Longyue Wang, Zefeng Du, DongHuai Liu, Deng Cai, Dian Yu, Haiyun Jiang, Yan Wang, Shuming Shi, and Zhaopeng Tu. Guofeng: A discourse-aware evaluation benchmark for language understanding, translation and generation, 2023a. URL https: /oppnre/ip.. net/forummlddxIIInnhheeee
Longyue Wang, Zhaopeng Tu, Yan Gu, Siyou Liu, Dian Yu, Qingsong Ma, Chenyang Lyu, Liting Zhou, Chao-Hong Liu, Yufeng Ma, Weiyu Chen, Yvette Graham, Bonnie Webber, Philipp Koehn, Andy Way, Yulin Yuan, and Shuming Shi. Findings of the WMT 2023 shared task on discourse-level literary translation: A fresh orb in the cosmos of LLMs. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz (eds.), Proceedings of the Eighth Conference on Machine Translation, pp. 55-67, Singapore, December 2023b. Association for Computational Linguistics. doi: 10.18653 v 2tt./2222.mmm11... wmt-1.3. URL https://aclanthooogg) 20232222..rg 1..-1.3.
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self- attention with linear complexity, 2020.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient stream- ing language models with attention sinks. In The Tweifth International Conference on Learning Representations, 2024. URL https::///ppeeeeieieeeeee///
GuangxuanXial
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models, 2024.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences, 2021.
12

---

Published as a conference paper at COLM 2024
Liang Zhao, Xiaocheng Feng, Xiachong Feng, Bing Qin, and Ting Liu. Length extrapolation of transformers: A survey from the perspective of position encoding, 2023.
Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. PoSE: Efficient context window extension of LLMs via positional skip-wise training. In The Twelfth International Conference on Learning Representations, 2024. URL htttps://opereevier net/forum2id=3211 gxuuuuu
13


Reviews:
[{"review_id": "YqXTBHhUiM", "reviewer": "Reviewer_3Zws", "summary": "In this paper, the authos proposed Temp-Lora, an alternative way for efficient long text generation. Its essence lies in training during the inference process using the generated output. It enables the storage of nearly infinite context information directly within the model’s parameters, marking a distinct difference from existing attention weights-based techniques.\nThe experimental results across various applications demonstrated the profound impact of Temp-Lora.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Concern 1**: Comparision with existing methods: \n\nThanks for your valuable suggestions! Firstly we should clarify that we already compare with a **very strong baseline \"Together LLaMA2-7B-32k\"** in our paper. This is the introduction of this model: https://www.together.ai/blog/llama-2-7b-32k. The\ndevelopers followed Meta’s linear interpolation and continue pre-trained it with 1.5B tokens. We believe it is much stronger than those training-free methods. Additionally, in Table 4, we also made a comparison with the StreamingLLM method (+AS), and verified the orthogonality of Temp-Lora with it.\n\nIn fact, most existing methods like Dynamic-NTK are not suitable for super-long context scenarios. This Table shows the results on PG19 after extending Llama2-7B-4K using Dynamic-NTK:  \n|Model|0-100K |100-300K |300-500K |500K+ |Avg.|\n| :-:| :-:|:-:|:-:|:-:|:-:|\n| *8K | 11.2 |10.9 |9.7 |5.3 |10.9 |\n| *16K|  22.5 |24.1 |20.9 |10.5 |23.2 |\n| *24K| 65.8 |87.0 |78.3 |24.8 |77.7|\n| *32K| 106.3 |161.0 |147.1 |63.3 |137.1 |\n| *40K| 212.4 |420.6 |385.7 |145.1 |324.4 |\n\nOne may find that once the context window extends to more than four times its training window, PPL will collapse directly. \n\nIn addition, you may also interested in the evaluation of the proposed method on other LLMs. Please refer to **Concern 1 from Reviewer Sdmf** for details. \n\n**Concern 2**: using established benchmarks.\n\nFrom the introduction of LongBench: the average length of most tasks ranging from 5k to 15k.\n\nThe problem is that they are **too short** to validate the effectiveness of Temp-Lora. Just as we concluded in the last sentence of section 3.3: \"Note that in scenarios without extensive text, for example, less than the model’s window size in training, Temp-Lora is useless.\" \n\tThe selected test data in this paper, PG19, is also a famous benchmark for long-text generation. \n\n**Concern 3**: a detailed analysis of batch decoding.\n\tThis is the detailed memory usage and latency of Llama2-7B with batch sizes of 1, 2, and 5. The '#' symbol indicates an 'out of memory' issue. \n|Metric|Batch Size|base|24k|16K|8K|4K|2K|\n| :-:| :-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Mem. (GB)|2|64.3 |65.2 |48.3 |31.3 |22.8 |18.5 |\n|Lat.(s)|2|115.1 |117.1 |80.0 |45.9 |30.1 |29.3 |\n|Mem. (GB)|5|# |# |# |56.3 |35.2 |24.7 |\n|Lat.(s)|5| #| #| #|68.5 |49.6 |31.6 |\n\n**Concern 4**: why the PPL decrease?\n\nThis is due to the natural distribution of the dataset. Due to the length limit, we will provide detailed statistics in discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method is interesting and has the potential to significantly enhance both effectiveness and efficiency in the domain of long-context generation.\n\n2. This paper is well-written, with a clear and comprehensible description of the method.", "reasons_to_reject": "1. The experimental results presented are insufficient. The authors must first conduct a comparative analysis with existing methods for long context generation. Additionally, it is imperative to validate the proposed method using established long-text benchmarks to ensure its robustness and applicability.\n[1] Dynamic-NTK: https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases/\n[2] LongBench: https://github.com/THUDM/LongBench\n\n2. The primary advantage of the proposed method is its ability to operate without the need for saving the KV Cache. The authors should provide a comprehensive demonstration of the improvements in overall efficiency across various settings and scenarios to substantiate this claim, e.g., throughput in batch decoding.", "questions_to_authors": "1. I am interested in understanding the performance of the proposed method within the context of batch decoding. Could the authors provide a detailed analysis of its efficiency and effectiveness in such scenarios?\n\n2. In Table 1, why does the perplexity (PPL) decrease as the range extends further back? Additionally, how is the data distributed across the different intervals? A thorough explanation of these observations would be beneficial.", "extra_scores": {}}, {"review_id": "aKboxZMlqI", "reviewer": "Reviewer_Sdmf", "summary": "This paper proposes that a Low Rank Adaptation module (Temp-Lora) that is temporarily invoked and discarded during the generation of long text. The results of applying this model to text generation tasks appear to be substantial, with improvements in perplexity accompanied by reductions in memory usage and inference latency.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Concern 1**: Does the efficacy of the approach depend on the model?\n\tThank you for your suggestion! Temp-Lora significantly improves almost all other Open Source models, but we only selected the Llama family as their representative in the paper. Upon your suggestion, we present the effects of Temp-Lora on five models in the table below (Due to differences in the tokenizer, some models do not have the interval of 500K+).\n\n|Model|0-100K |100-300K |300-500K |500K+ |Avg.|\n| :--| :--|:--|:--|:--|:--|\n|mistral01-7B-8K|10.58 |10.16 |8.83 |4.32 |10.20 |\n|+TL|10.34 (2.2%) |9.57 (5.7%) |8.19 (7.1%) |3.74 (13.3%) |9.73 (4.5%) |\n|phi-1-2k|195.31 |183.33 |149.02 | |186.08 |\n|+TL|94.04 (51.8%) |67.52 (63.1%) |52.33 (64.8%) | |76.11(59.0%) |\n|phi-2-2k|22.12 |23.12 |22.76 | |22.78 |\n|+TL|22.53 (-1.8%)|21.29 (7.8%) |19.37 (14.8%) | |21.68 (4.8%) |\n|qwen-7B-8k|18.39 |17.51 |14.08 | |17.70 |\n|+TL|18.02 (2.0%) |16.44 (6.1%) |13.05 (7.3%) | |16.90 (4.5%) |\n|qwen-1.8B-8k|27.03 |25.30 |21.96 | |25.84 |\n|+TL|25.33 (6.2%) |22.23 (12.1) |18.32 (16.5%) | |23.26 (9.9%) |\n\nWe will supplement these experimental results in the next version of the manuscript\n\t\t\n**Concern 2**: the paper does not attempt to compare performance with one of more of the approaches noted in the second paragraph of the introduction. \n\nFirstly we should clarify that we already compare with a **very strong baseline \"Together LLaMA2-7B-32k\" (refer as Llama2-7B-4k)** in our paper. Please refer to our response to Reviewer 3Zws, Conern 1, for more details of this strong baseline, and the comparision between Temp-Lora and other existing long-text generation methods.\n\t\n**Concern 3**: Overreliance on a single dataset (PG19) may be an issue.\n\nYes, you are correct, this is an issue that also troubles us. One reason for this issue can be found in the **response to Concern 2 for Reviewer 3Zws**: Currently, all benchmarks, such as LongBench and Arxiv, are **too short** to validate the effectiveness of Temp-Lora. Therefore, to the best of our knowledge, we have found two datasets that are more suitable for super-long text generation scenarios: PG19 and GuoFeng. Since most of the novels in PG19 are still too short, we had to re-organize it to better validate the performance of Temp-Lora.\n\t\n**Concern 4**: Acronyms should be spelled out.\n\nThank you very much for your kind suggestions! We will carefully re-organize the introduction to make this paper more easy to follow!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tTechnological impact: The paper proposes replacing the widely used KV cache with a mechanism that stores context information in model parameters during training and generation and with some apparent savings of computational cost. This reviewer, at least, found the approach interesting, and potentially useful.\n\nThe Temp-Lora module only exists during long-text generation, and therefore requires a mechanism to determine when then long-text generation is being performed. At some leve, this proposal is a technically-sophisticated hack—an interesting one, but nevertheless one whose functionality might eventually be more organically folded into the model itself. In its current form, therefore, the proposal is probably not a long-term solution, but one that allows the fields to move ahead while defining a performance space that a longer-term, more-organic solution would need to cover. The definition of this achievable performance space is probably the primary reason that this paper should be accepted.  \n\n2.\tEvaluation:  The paper presents extensive ablation studies showing consistently improved results at least on the metrics and datasets employed. \n \n3.\tClarity:  The paper should be largely clear to someone who is well accomplished in language modelling.  For the most part it is readable and interesting, with good documentation of results and valuable discussion.", "reasons_to_reject": "Evaluation:  Although the evaluation is extensive, it has limitations.  The only comparisons made in Table 1 are with models in the llama family.  The paper would be more convincing if they had shown benefits using other Open Source models, e.g., Phi-2 or Mistral, as well. (Does the efficacy of the approach depend on the model?)\n\nit is a pity, perhaps, that the paper does not attempt to compare performance with one of more of the approaches noted in the second paragraph of the introduction. The perplexity and BLEU improvements seem large, but the reader does not get to see how the approach compares with some of the alternative approaches available.   \n\nOverreliance on a single dataset (PG19) may be an issue. The authors used a random sample of 40 books with high perplexity from within this data set. Though the reasons for this manner of selection are stated in the paper, given that the models may have seen this dataset during training, the high starting perplexity is likely to bias the results. It may be easier to obtain strong percentage-wise reductions from a higher starting point than a lower one.", "questions_to_authors": "Acronyms should be spelled out when they are introduced in the abstract and introduction. The manner in which the paper introduces the terms KV cache and Lora imposes unnecessary presuppositional burdens on the reader.  The KV cache is not really an organic part of a language model, but a heuristic designed to make inference more efficient. Similarly Lora. These issues should be easily remediable before the final version is ready.", "extra_scores": {}}, {"review_id": "2TGGM7diWU", "reviewer": "Reviewer_1qNy", "summary": "This paper propose a method called Temp-Lora, which embeds the context information into a Lora module for each chunk to enable long text generation. The proposed method is implemented based on Llama2 and evaluated on two benchmark datasets. The performance on PPL, memory usage and latency get improved compared with Llama2.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Concern 1**: The proposed method has only been evaluated on Llama2 & lacking some strong baselines\n\nThanks for your valuable suggestions! Firstly we should clarify that we already compare with a **very strong baseline \"Together LLaMA2-7B-32k\"** in our paper. Please refer to our response to **Conern 1 from Reviewer 3Zws**, for more details of this strong baseline, and the comparision between Temp-Lora and  other existing baselines.\n\tFurthermore, please refer the response to **Concern 1  from Reviewer Sdmf** for details about the evaluation of the proposed method on other LLMs.\n\t\n**Concern 2**: Need some cases.\n\nWe apologize, but due to the difficulty in demonstrating the effects of long-text generation, we followed the majority of work in this field and used PPL as the evaluation metric. Upon your suggestion, we conducted an additional experiment to allow you to intuitively understand Temp-Lora's effects in long-context tasks.\n\nWe selected a book from PG19, BIOGRAPHIA LITERARIA, written by Samuel Taylor Coleridge, and used the content of this book as the context, taknig Llama2-7B-Chat as the base model to compare the effect of Temp-Lora. \nAfter we input the instruction:**'Could you please introduce four main characters that appear frequently in the book BIOGRAPHIA LITERARIA by Samuel Taylor Coleridge**, the model with Temp-Lora output: \n'The book is divided into three parts: the first part consists of biographical sketches of various writers, including William Shakespeare, John Milton, William Wordsworth, and Alexander Pope; the second part contains critical essays on Shakespeare's plays and Milton's poetry; and the third part...'\n\nAfter string matching BIOGRAPHIA LITERARIA, we found that William Shakespeare, John Milton, William Wordsworth, and Alexander Pope **appeared 55, 50, 102, and 12 times respectively, indeed being the main characters in this book**.\n\nWhen we input the last 4K tokens of this book and the same instruction to the vanilla Llama2-7B-4K, the model's output is as follows (due to the length limit of the rebuttal, we omitted some irrelevant content): \n'Here is a brief introduction to the main characters in Samuel Taylor Coleridge's Biographia Literaria:\\n\\n1. William Wordsworth ....\\n2. John Wilson - ...\\n3. ....\\n4. Thomas De Quincey ....'\nUnfortunately, **only William Wordsworth among these four actually appears in the book**. The comparison of these two cases fully illustrates the effectiveness of Temp-Lora in long-context tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method is simple and it shows improved generation performance and decreased inference time compared with Llama2.", "reasons_to_reject": "1. The proposed method has only been evaluated on Llama2, leading to limited generalization.\n\n2. Temp-Lora performs better than Llama2. This is good. But the paper only shows the performance of Llama2, lacking some strong baselines such as flashattention, pagedattention. Therefore, we can't know how good Temp-lora is compared with some strong models.", "questions_to_authors": "Can the author show some cases generated by the proposed method? Only providing the PPL can't intuitively show the generation performance of the proposed method.", "extra_scores": {}}, {"review_id": "FGFEjMPx2T", "reviewer": "Reviewer_535h", "summary": "This paper introduce a simple and effective framework Temp-Lora to extend the context window for LLMs for long text generation.\nThe author argues that traditional methods (length extrapolation & context window extension) demand significant hardware resources\nduring training / inference,  so Temp-lora is proposed to store context information in model parameters instead of KC cache.\nThe parameters of Temp-Lora is training on the fly using previously-generated tokens, and discarded when the inference is completed.\nExperiment on PG19 data-set shows a significant improvement in PPL/BLEU, and a large amount of memory reduction and latency decrease.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Concern 1 & 2**:  \nThanks for your valuable suggestions! Firstly we should clarify that we already compare with a\n**very strong baseline \"Together LLaMA2-7B-32k\"**\nin our paper. Please refer to our response to **Conern 1 from Reviewer 3Zws**, for more details of this strong baseline, and the comparision with other existing baselines.\n\nFurthermore, please refer the response to **Concern 1 from Reviewer Sdmf** for details about the evaluation of the proposed method on other LLMs.\n\nFinally, we also verify the compatibility of Temp-Lora with the StreamingLLM method (Efficient Streaming Language Models with Attention Sinks\"). The model \"7B-32K+AS\" in Table 1 represents this method. This is our analysis in Section 3.1:\n\nMoreover, another set of experiments verifies the compatibility of Temp-Lora with the recent proposed StreamingLLM method (attention sink). Encouragingly, the experimental results confirmed that these two can be seamlessly integrated. With the augmentation of Temp-Lora, the reduction in perplexity (PPL) of Llama2-7B-32K + attention sink is even greater than its standard version (5.5% vs 4.9%, 5.2% vs 4.7%, and 4.6% vs 4.1%).\n\n**Concern 3**: maybe the author could add more detail on the computation cost, and list out computation overhead for a better presentation of this paper\n\nIn fact, we have already conducted an analysis in this regard in Section 3.2, and we will elaborate it more clearly in the next version.\n\nIn Table 4, we detailed the computational cost of on-the-fly training of the Temp-Lora module under different chunk sizes and batch size settings. By comparing Table 3 and Table 4, we can conclude that: Under similar memory usage conditions, the time consumed to update the Temp-Lora module with one chunk is only 1/15 of the time used to infer this chunk (1.67s vs 24s). The analysis in our paper is as follows:\n\nNotably, training Temp-Lora is significantly faster than the inference process itself. For instance, in the 'latency-sensitive' configuration, with ∆ = 1K and W = 4K, the average inference latency for every 1K tokens is 24 seconds, whereas Temp-Lora training takes merely 1.67 seconds. Furthermore, regarding memory usage, it is reassuring that the memory required for Temp-Lora training rarely surpasses that during inference. Thus, repurposing the memory allocated for inference to train the Temp-Lora module is viable, eliminating the risk of an 'Out of Memory' issue.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The motivation of this paper is solid, focusing resolving the context limitation for long text generation. \nThe proposed method is simple and effective, the paper is easy to understand.\nAnd the experimental results seems promising, achieving PPL reduction, memory reduction & latency reduction at the same time.", "reasons_to_reject": "1. I think the related work comparison is weak. In the intro, the author strongly claims that \"Temp-Lora is compatible with and enhances most existing long text generation methods.\"  But I don't see enough evidences in the experiment part to support this claim.  So maybe the author can elaborate on this point.\n2. It's exciting to see good experimental results  (achieving PPL reduction, memory reduction & latency reduction at the same time),\nbut on the other side of the coin, I think it's also partially because the baseline is relatively weak, maybe adding more comparison system with the length extrapolation and context window extension methods will make the results more convincing.\n3. One of the main advantage of this method is preserving the hardware resources, I assume it also includes computation cost, \nand the on-the-fly training of Temp-lora requires additional GPU during inference.  So maybe the author could add more detail on this direction, and list out computation overhead for a better presentation of this paper.", "questions_to_authors": "see weakness part.\nAlso in Figure 1, there is a missing arrow to point to Chunk1 to \"Generate\"", "extra_scores": {}}]

Author rebuttal:

Meta-review: Some concerns were addressed by the authors during the response period. 
Some reviewers are enthusiastic about the paper. 
Towards Accept.
Decision: Accept