With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generation

Authors: yan  wang
deng  cai
training  generate
…  lora
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 11:31:39

Abstract: Long text generation, such as novel writing and discourse-level translation with extremely long contexts, presents significant challenges to current language models. Existing methods mainly focus on extending the model's context window through strategies like length extrapolation. However, these approaches demand substantial hardware resources during the training and/or inference phases.

Our proposed method, Temp-Lora, introduces an alternative concept. Instead of relying on the KV cache to store all context information, we embeds this information directly into a temporary Lora module. In the process of long text generation, this module is progressively trained with text generated previously. This approach not only efficiently preserves contextual knowledge but also prevents any permanent alteration to the model's parameters given that the module is discarded post-generation.

Extensive experiments on the PG19 language modeling benchmark and the GuoFeng discourse-level translation benchmark validate the effectiveness of Temp-Lora. Our results show that: 1) Temp-Lora substantially enhances generation quality for long text, as indicated by a 13.2\% decrease in perplexity (PPL) on a subset of PG19, and a 29.3\% decrease in PPL along with a 113.2\% increase in BLEU score on a subset of GuoFeng, 2) Temp-Lora is compatible with and enhances most existing long text generation methods, and 3) Temp-Lora can greatly reduce computational costs by shortening the context window. For example, we can ensure a moderate improvement in generation quality (a decrease of 3.8\% in PPL) while enabling a 51.5\%  memory usage reduction and a 60.0\% decrease in latency for inference.

Main paper: 
 introduction: 
 Long text generation has become increasingly important in a variety of real-world applications, ranging from creative writing assistance (Shi et al., 2022) , chat-style AI assistant (Ope-nAI, 2023) to generative agents (Park et al., 2023) . However, the generation of coherent and contextually relevant long text poses significant challenges to language models (LMs), particularly in terms of understanding and maintaining contexts that are longer than the model's pre-defined context window size.Existing methods, including those based on length extrapolation (Press et al., 2022; Su et al., 2023) and context window extension (Chen et al., 2023b; Han et al., 2023; Dao et al., 2022; Peng et al., 2023; Chen et al., 2023a) , aims to store extensive text information within the KV cache, thereby improving the model's long text comprehension. However, they demand significant hardware resources during training and/or inference. Consequently, in many applications where LMs are frequently queried for long text processing, users often resort to other strategies such as retrieval or summarization to reduce the cost (Park et al., 2023) .In this paper, we propose an alternative method, Temp-Lora, which stores context information in the model parameters instead of the KV cache. The core idea of our approach is Figure 1 : The framework of long text generation with Temp-Lora extremely simple: we store the context information in a temporary Lora module (Hu et al., 2021) that only exists during long text generation. We update this module in a streaming fashion during the generation process, using the previously-generated content as training data, thereby achieving the goal of storing knowledge in the model parameters. Once inference is complete, this module is discarded to avoid permanently impacting the model's parameters. This approach allows us to efficiently store nearly infinite context information without extending the context window.We evaluate Temp-Lora on two benchmark datasets, PG19 (Rae et al., 2019) and GuoFeng (Wang et al., 2023b) . We evaluate Temp-Lora across models with different context window sizes and find that Temp-Lora substantially reduces their PPL on long text with a large margin . This trend becomes increasingly pronounced as the context length increases, which can be simply concluded as "with greater text comes greater necessity". Further analysis also reveals that Temp-Lora, by shortening the maximum input length, can greatly reduce hardware consumption. 
temp-lora: 
 The Temp-Lora framework, depicted in Figure 1 , presents a straightforward yet innovative approach for long text generation. At the heart of this method lies the progressive training of a temporary Lora module (Hu et al., 2021) , which is named Temp-Lora, on previouslygenerated text in an auto-regressive manner. The continuous adaptation and refinement of the Temp-Lora module ensure an evolving understanding of both recent and distant contexts. It is important to highlight the framework's adaptability in handling cases where the initial input lengths are already substantial, such as novels or academic papers. In such scenarios, we may proactively pre-train the Temp-Lora module with the contexts, thereby laying a robust foundation for enhanced context comprehension in the generation process.The algorithm is detailed in Algorithm 1. During the generation process, tokens are generated chunk-by-chunk. At each chunk generation, we use the most recent L X tokens as the input X for generating the subsequent tokens (all preceding tokens beyond the distance L X are discarded). Once the number of generated tokens hits a predefined chunk size ∆, we initiate the training of the Temp-Lora module using the latest chunk and start the next chunk generation afterwards. It is important to note that L X + ∆ ≤ W, where W is the model's context window size.For the training of the Temp-Lora module, it is crucial to recognize that learning to generate the new chunk without any condition may not constitute a meaningful training objective and potentially lead to significant over-fitting. To address this concern, we incorporate the preceding L T tokens of each chunk into our training process, using them as the input and the chunk as the output.Cache Reuse For more efficient inference, we propose a strategy called cache reuse. In the standard framework, after each update of the Temp-Lora module, we need to re-compute Pre-train Θ on X to obtain LM with an initial Temp-Lora module Θ 0 ; 3 end 4 Total token number: i ← len(X ); 5 Temp-Lora ID: k ← 0; 6 Token number in this chunk:m ← 0; 7 Output sequence Y = X ; 8 X = X [-(min(L X , len(X )) :]; 9 while In Generation do if m < ∆ then Y [i] = Θ k (X + Y [-m :]); i + +; m + +; end else Chunk k = Y [-m :]; X T = Y [-(L T + m) : -m]; k + +; Train Θ k : In X T → Out Chunk k (n epochs, learning rate α); X = Y [-L X :]; m = 0; end end Destroy Temp-Lora module: Θ k ← Θ;the KV states of the latest L X tokens with the updated parameters. Alternatively, we can also reuse the existing cached KV states while employing the updated model for subsequent text generation. We empirically find that cache reuse can accelerate the generation speed without compromising the generation quality.The recently proposed Attention Sink (Xiao et al., 2024) method allows us to further enhance inference efficiency. By permanently storing four initial tokens as attention sinks in the KV cache, the model, after generating up to the maximum length, no longer needs to recompute the KV states. Therefore, if we simultaneously employ cache reuse and attention sink strategies, during long-text inference, we only need to continuously update the model (Temp-Lora module) without re-computing the KV states.1  
experiments: 
 We evaluate the proposed Temp-Lora framework using the Llama2 (Touvron et al., 2023) families and Yi-Chat-6B considering their wide adoption and popularity. Its effectiveness is evaluated in two different long text generation tasks: 1) Novel Generation; and 2) Discourse-Level Literary Translation.2 The first dataset we adopt is a subset of the long text language modeling benchmark, PG19 (Rae et al., 2019) . It is a well-established benchmark that consists of more than 28K books which were published before 1919. Since we are especially concerned with the effectiveness of the Temp-Lora framework in extremely long text scenarios, and considering that some of the PG-19 data might already be included in our models' pre-training corpora, we select 100 books with the highest PPL from those whose lengths range between 200K and 600K tokens. From these, we randomly sample 40 books as our test set.We also evaluate the effectiveness of Temp-Lora on a downstream task, Discourse-Level Literary Translation, with a randomly sampled subset of GuoFeng dataset from WMT 2023 (Wang et al., 2023b; a) . This subset contains 20 web novels, originally written in Chinese by various novelists and subsequently translated into English by professional translators. Their lengths (en + zh) range from a minimum of 84K to a maximum of 370K tokens.We split novels in the GuoFeng dataset into segments based on sentence boundaries. Each segment has approximately 512 tokens (en + zh) in terms of length. The model's input includes the current source segment along with its historic source and target text, and it is required to output the translation of current source segment.We apply the Temp-Lora framework to three Llama2 variants and Yi-Chat-6B:• Llama2-7B-4K: the standard Llama2-7B model.3 • Llama2-13B-4K: the standard Llama2-13B model.4 • To-Llama2-13B-32K: This is a very strong baseline model that extends the context window of Llama2-7B-4K to 32K. The developers from TogetherAI followed Meta's linear interpolation (Chen et al., 2023b) and continue pre-trained it with 1.5B tokens. We believe it is much stronger than those training-free methods.5 • Mistral-7B: the standard Mistral-7B-v0.1 model with a 8K context window size.6 • phi-2: the standard phi-2 model with a 2K context window size.7 • qwen-7B: the standard qwen-7B model with a 8K context window size.8 • Yi-Chat-6B: A chat-style model trained on both Chinese and English corpus. We will use this model in the Discourse-Level Literary Translation task.9 .Please note that the length of most samples in our test set significantly surpass the models' context window sizes. In such instances, when the model hits the token generation limit, we slide the context window and recompute the KV states from the recent W -1024 tokens.Evaluation Metric: The primary metric is perplexity (PPL), a standard measure in language modeling to assess the model's prediction capability. We employ a sliding window approach for PPL measurement as suggested by (Press et al., 2022) . For the translation task, in addition to PPL, we also employ two common evaluation metrics used in machine translation, BLEU (Papineni et al., 2002) and COMET (Rei et al., 2020) , for comprehensive evaluation.We set the chunk size ∆ = 1024 for all models. For the Llama2-7B-32K, we additionally set two wider chunk size ∆ = 2048 and 4096 to investigate the effects of chunk size on generation quality and computational cost.The input length L X for all models is set to be W -∆ for maximizing the use of the models' context window size. For the Llama2-7B-32K, we notice that its PPL increases when the number of tokens within its context window approaches 32K. Therefore, we set its context window size W to 24K, which, based on our preliminary experiments, is the optimal context window size. Note that the number of context tokens may be smaller than L X . In such cases, we simply take all context as the input. In the generation process, by default, once the token number in the context window reaches the context window size (4K and 24K for different models) we re-compute the KV states taking the L X recent tokens as input X (i.e., no cache reuse). When we update the Temp-Lora module, the length L T is set to 1024.In the experiments on GuoFeng, each segment is treated as a chunk. In other words, after the model translates a complete segment, this segment is then used to update the Temp-Lora module. We decode the translations with greedy search (repetition penalty = 1.12).10 All experiments are done with a single NVIDIA A800 GPU. We set the learning rate α = 5 × 10 -5 and the number of epochs n = 2 for all models. We use a linear learning rate warmup for the first 2 chunks. We set Temp-Lora α = 64, rank = 64, dropout = 0.05. Its training is in bfloat16 format, using Deepspeed ZeRO Stage 2, Flash Attention V2 (Dao, 2023) , and gradient checkpointing. PG19: Table 1 presents the PPL comparisons across various models with and without the Temp-Lora module on PG19. We measure the PPL scores at different positions separately. Specifically, for a sequence x, we compute the PPLs of the following subsequences:[x[: 100K], x[100K : 300K], x[300K : 500K], x[500K :]].Intuitively, In the initial segments such as 0-100K, where there is less context information, the improvement from Temp-Lora should be relatively modest. In contrast, as the segments grow longer and more information falls outside the model's context window, Temp-Lora's effects should become more pronounced.The experimental results in Table 1 confirm our hypothesis. Firstly, the augmentation of Temp-Lora leads to a significant PPL reduction for all models, where we observe an average decrease of 5.9% on Llama2-7B-4K. An in-depth examination of Temp-Lora's impact across different text segments reveals it is more notable on long text. For example, Temp-Lora augmented Llama-7B-4K achieves a direct PPL reduction of 13.2% in the 500K+ segment. Surprisingly, on segments whose context length is greater than 300K, Temp-Lora helps Llama2-7B achieve a lower PPL than the 13B model. In contrast, its effect is relatively less pronounced in the 0-100K segment, where it only reduces the PPL by 3.6%. We may simply conclude the results as: with greater text comes greater necessity for Temp-Lora.Experiments with the strong baseline model, To-Llama2-7B-32K, which was fine-tuned on 1. Furthermore, adjusting the chunk size from 1024 to 2048 and 4096 resulted in a slight increase in PPL. It is not surprising, as the Temp-Lora module was trained on the data from the previous chunks. An increase in chunk size means that the information stored in the module is more distant from the current token. Indeed, the choice of chunk size is a critical trade-off between generation quality and computational efficiency, which will receive detailed analysis in Section 3.2.We also discover that cache reuse almost does not cause any loss in performance. This is a very encouraging observation, as this technique can significantly reduce computational cost. Moreover, another set of experiments verifies the compatibility of Temp-Lora with the attention sink. Encouragingly, the experimental results confirmed that these two can be seamlessly integrated. With the augmentation of Temp-Lora, the reduction in PPL of Llama2-7B-32K + attention sink is even greater than its standard version (5.5% vs 4.9%, 5.2% vs 4.7%, and 4.6% vs 4.1%).GuoFeng: Table 2 presents the remarkable impact of the Temp-Lora on the task of discourse-level literary translation. Compared to the base model Yi-Chat-6B, there are significant improvements across all metrics and all language pairs: a decrease in PPL by -23.8%, an increase in BLEU score by +53.2%, and a rise in COMET score by +8.4% in Chinese-to-English translation; a decrease in PPL by -29.1%, an increase in BLEU score by +113.1%, and a rise in COMET score by +7.2% in English-to-Chinese translation. This experiment illustrates that the Temp-Lora module is also effective in downstream tasks.One might wonder why the effects are more pronounced in translation. This is attributed to the differences between tasks: open-domain novel completion is a task without standard answers, making it challenging for even an active human reader of a specific novel to predict the content of the next chapter. In such cases, even if the Temp-Lora module stores substantial context information, the model can only slightly reduce the PPL of groundtruth text. In contrast, in discourse-level translation, most word translations are unique for consistency. By effectively storing these translation mappings in the Temp-Lora module, substantial improvements can be readily achieved.Contrast to Dynamic-NTK One might be curious about how the existing Length Interpolation methods perform in these super-long context scenarios. Table 3 presents the outcomes on the PG19 dataset after extending the Llama2-7B-4K model using Dynamic-NTK (NTK, 2023; Peng et al., 2023) . Unfortunally, Dynamic-NTK are not suitable for this scenario. One may easily find that once the context window extends to more than four times its training window, PPL will collapse directly. After shortening W, we can reduce the computational cost for both models. However, this reduction results in less context information being stored in the KV cache, compelling the model to rely primarily on the Temp-Lora module for accessing contextual information.We measure Llama-7B-32K's PPL, memory usage, and latency across these varied context window sizes W in Table 4 . This experiment aims to explore the balance between generation quality and computational efficiency in different scenarios.We observe that models augmented with Temp-Lora consistently surpass the base model's performance, regardless of the context window size. Notably, even when we reduce the window size to 1/12 of its maximum ( 2048), a reduction in PPL from 9.81 to 9.65 is also observed. Please note that as the context window diminishes, there is still a gradual increase in the model's PPL. As a whole, this trend highlights that Temp-Lora and the KV cache are orthogonal, jointly enhancing the overall performance of the model when used together. Efficiency: Next, let's explore the efficiency of the Temp-Lora-augmented model in practice. When deploying the model in real-world applications, two distinct deployment strate-gies can be considered based on the availability of computational resources: 1) Cascaded: As outlined in Algorithm 1, this strategy temporarily pauses inference after generating a chunk of data. This pause allows for the Temp-Lora module to be updated with the newly generated chunk before resuming the generation process. 2) Parallelized: When abundant computational resources are available, the Temp-Lora module can be updated in parallel with the token generation process during inference. In this approach, after generating a chunk, we do not pause inference. Instead, we update the Temp-Lora module using this chunk in another process. Once the Temp-Lora module is updated, we immediately replace the old one with the updated version.With the parallelized deployment strategy, the model's memory usage and inference latency during inference, as shown in Table 4 are nearly identical to those of the base model with the same context window size. We were surprised to find that in the most "latency-sensitive" scenario, we may set ∆ = 1K and W = 4K, the model not only reduces PPL by 3.4% but also saves 51.5% of Memory usage and 60.0% of latency. Conversely, if we disregard computational costs entirely, then in the most "luxurious" configuration, ∆ = 1K and W = 24K, we can achieve a 5.0% reduction in PPL with minimal additional memory consumption (0.3 Gb) and latency (0.1 ms).On the other hand, in scenarios of limited resources, the cascaded deployment strategy becomes preferable. In such cases, the total time required to generate a chunk equates to the sum of the model's inference latency and the time spent updating the Temp-Lora. As detailed in Table 5 , we demonstrate the relationship between training latency and memory usage across various training batch sizes. Notably, training Temp-Lora is significantly faster than the inference process itself. For instance, in the "latency-sensitive" configuration, with ∆ = 1K and W = 4K, the average inference latency for every 1K tokens is 24 seconds, whereas Temp-Lora training takes merely 1.67 seconds. Furthermore, regarding memory usage, it is reassuring that the memory required for Temp-Lora training rarely surpasses that during inference. Thus, repurposing the memory allocated for inference to train the Temp-Lora module is viable, eliminating the risk of an 'Out of Memory' issue.0 1 2 3 4 5Num. of Epochs Figure 2 : The relationship between model PPL and hyper-parameters, depicted from left to right, are (a) Epochs, (b) Lora Rank, and (c) Learning Rate. Note that the PPL at a X-axis value of 0 in each graph corresponds to the performance of the base model.One might find it surprising that the training cost for Temp-Lora is significantly lower than for inference, despite the widespread belief that model training generally requires more time than inference. This efficiency stems from the model's ability to leverage parallel processing techniques during the training phase, allowing it to handle the entire chunk simultaneously. In contrast, the generation phase is constrained by the autoregressive nature of the language model, necessitating a token-by-token generation approach. This fundamental difference significantly accelerates the training of a chunk compared to its generation.After summarizing the experimental results, we got some suggestions for applying Temp-Lora in real-world scenarios: 1) For applications demanding the highest level of long text generation, integrating Temp-Lora into existing models -without altering any parameters -can significantly enhance performance at a relatively modest cost. 2) For applications where minimal latency or memory usage is paramount, computational costs can be significantly lowered by reducing input lengths and storing context information within Temp-Lora.Under this setup, we can process text of almost infinite length (500K+ in our experiments) using a fixed short window size. 3) Note that in scenarios without extensive text, for example, less than the model's window size in training, Temp-Lora is useless.One may wonder how sensitive Temp-Lora is to the training hyper-parameters. In Figure 2 , we investigate the sensitivity of our model to three key hyper-parameters, Epochs, Lora Rank, and Learning Rate, on a subset of PG19 that contains 3 books.Epochs: training each chunk for just one epoch consistently leads to a significant PPL reduction, where we observe an average decrease of about 0.2. Expanding the number of epochs to two further reduces PPL, but excessively high epoch counts risk catastrophic forgetting, and negating the benefits by losing previously learned information.Lora Rank: lora rank determines the number of parameters in Temp-Lora module. From Figure 2 (b) we observe that the model is robust to this hyper-parameter. When the lora rank increases from 16 to 256, the model's PPL remains essentially stable.Learning Rate: from Figure 2 (c) we observe that the learning rate is a relatively important parameter that has a significant impact on the model's performance. Intuitively, a too-small learning rate may lead to less knowledge being stored in the Temp-Lora module, whereas a too-large learning rate might cause the model to overfit to the current chunk. 
related work: 
 In recent years, numerous efforts have been made to enable language models to understand and generate longer texts (Pawar et al., 2024; Zhao et al., 2023) . The primary focus of these studies has been to store more information within the context window. They can be broadly categorized into three types: 1) Length Extrapolation, 2) Context Window Extension, 3) External Memory. It is worth noting that there are some overlaps between categories 1) and 2). Following Chen et al. (2023b) , we categorize training-free methods under Length extrapolation, while those fine-tune-based methods are categorized under context window extension. Our Temp-Lora framework pioneers a new direction in long-text generation area, which is orthogonal to these three types and can be seamlessly integrated.Length Extrapolation aims to find ways to process long contexts with short context windows. This "Train Short, Test Long" paradigm was first introduced in Press et al. ( 2022), which proposed the ALiBi position embedding method that leverages linear-decaying attention biases to achieve the extrapolation of position encoding. On the top of another position embedding method, RoPE (Su et al., 2023) , Methods like Position Interpolation (Chen et al., 2023b) , NTK-Aware scaling (NTK, 2023), and ReRoPE (Su, 2023 ) make some further progresses to enable the LLM to handle unseen positions at the inference time. A recent method, SelfExtend (Jin et al., 2024) , attempt to extend the context window of LLMs by constructing bi-level attention information. Based on some empirical experience, although these methods can theoretically be scaled to an infinite context window length, in practical applications, extending the context window length to more than four times the size of the context window will lead to a serious degradation problem.Context Window Extension centers on expanding the LLMs' context window, enabling the processing of more tokens in one forward pass. The principle behind "Extending" might sound straightforward: train LLMs on longer texts. However, the most significant challenge lies in the training efficiency problem. Consequently, researchers are exploring ways to reduce the training costs. Their solutions range from system-focused optimizations like FlashAttention (Dao et al., 2022; Dao, 2023) that accelerates attention computation and reduces memory usage, to approximate attention methods (Zaheer et al., 2021; Beltagy et al., 2020; Wang et al., 2020; Kitaev et al., 2020) that trade model quality for efficiency. Additionally, Most Length Extrapolation methods (Chen et al., 2023b; Peng et al., 2023; Zhu et al., 2024) may also benefit from continual fine-tuning, where the extended context can be better utilized. Beyond algorithm-level optimization, ProLong (Chen et al., 2024a) try to tackle this long-context modeling issue from a data-centric perspective, which only filters a small amount of long dependency data for extension training. Recently, researchers also leveraged LoRA for cost-effective training (Chen et al., 2024b) . However, all these techniques only extend LLMs' context window to a limited extent, which falls short of our paper's primary concern of handling infinite inputs.External Memory tackles the long-context understanding problem from a different perspective: It stores all necessary knowledge into a pre-computed index and only retrieves useful data as the working context (Li et al., 2022) . The retrieved data can be either the supervised data (Cai et al., 2021) or chunked text (Lan et al., 2023; Xu et al., 2024) . Although the performance of Retrieval-Augmented Generation (RAG) continues to improve (Mohtashami & Jaggi, 2023; Tworkowski et al., 2023) , its biggest challenges lie in 1) the accuracy of retrieval is still far from satisfactory, and 2) incorrect retrieval can lead to error accumulation, resulting in a decline in the quality of the generated content.Our research coincides with the work of Sun et al. (2024) , which proposed to add a Test-Time Training (TTT) layer to the model to store the long context. The main difference between our work and theirs is that we focus on the Transformer architecture, while they primarily concentrate on the RNN architecture. 
conclusion: 
 We proposed Temp-Lora, an alternative way for efficient long text generation. Its essence lies in training during the inference process using the generated output. It enables the storage of nearly infinite context information directly within the model's parameters, marking a distinct difference from existing attention weights-based techniques.Our experimental results across various applications, including language modeling and discourse-level literary translation, demonstrated the profound impact of Temp-Lora. We showed that Temp-Lora not only greatly enhances the quality of long text generation but also significantly reduces computational costs. In particular, Temp-Lora led to remarkable improvements in perplexity (a reduction of 13.2% on a subset of PG19 and 29.3% on a subset of GuoFeng), as well as increases in BLEU (by 113.2%). The effectiveness of Temp-Lora becomes increasingly apparent as text length grows.  
 
 Appendix: Please note that simply adding attention sink without implementing cache reuse is ineffective because, without cache reuse, each update of Temp-Lora necessitates re-computing KV statesCodes and Data are available at: https://github.com/TemporaryLoRA/Temp-LoRA/tree/mainhttps://huggingface.co/meta-llama/Llama-2-7bhttps://huggingface.co/meta-llama/Llama-2-13bhttps://huggingface.co/togethercomputer/LLaMA-2-7B-32Khttps://huggingface.co/mistralai/Mistral-7B-v0.1/discussions/4https://huggingface.co/microsoft/phi-2https://huggingface.co/Qwen/Qwen-7Bhttps://huggingface.co/01-ai/Yi-6B-ChatFor the base model Yi-Chat-6B, we have included the most recent three segments as examples in the input to better maintain consistency in translation; for the Temp-Lora augmented model, there is no need to retain examples in the input 


Decision: Some concerns were addressed by the authors during the response period. 
Some reviewers are enthusiastic about the paper. 
Towards Accept.

Reviews: review: paper_summary: paper_summary: In this paper, the authos proposed Temp-Lora, an alternative way for efficient long text generation. Its essence lies in training during the inference process using the generated output. It enables the storage of nearly infinite context information directly within the model’s parameters, marking a distinct difference from existing attention weights-based techniques.
The experimental results across various applications demonstrated the profound impact of Temp-Lora.
strength_weakness: strength_weakness: 1. The experimental results presented are insufficient. The authors must first conduct a comparative analysis with existing methods for long context generation. Additionally, it is imperative to validate the proposed method using established long-text benchmarks to ensure its robustness and applicability.
[1] Dynamic-NTK: https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases/
[2] LongBench: https://github.com/THUDM/LongBench

2. The primary advantage of the proposed method is its ability to operate without the need for saving the KV Cache. The authors should provide a comprehensive demonstration of the improvements in overall efficiency across various settings and scenarios to substantiate this claim, e.g., throughput in batch decoding.
questions: questions: 1. I am interested in understanding the performance of the proposed method within the context of batch decoding. Could the authors provide a detailed analysis of its efficiency and effectiveness in such scenarios?

2. In Table 1, why does the perplexity (PPL) decrease as the range extends further back? Additionally, how is the data distributed across the different intervals? A thorough explanation of these observations would be beneficial.
paper_summary: paper_summary: This paper proposes that a Low Rank Adaptation module (Temp-Lora) that is temporarily invoked and discarded during the generation of long text. The results of applying this model to text generation tasks appear to be substantial, with improvements in perplexity accompanied by reductions in memory usage and inference latency.
strength_weakness: strength_weakness: Evaluation:  Although the evaluation is extensive, it has limitations.  The only comparisons made in Table 1 are with models in the llama family.  The paper would be more convincing if they had shown benefits using other Open Source models, e.g., Phi-2 or Mistral, as well. (Does the efficacy of the approach depend on the model?)

it is a pity, perhaps, that the paper does not attempt to compare performance with one of more of the approaches noted in the second paragraph of the introduction. The perplexity and BLEU improvements seem large, but the reader does not get to see how the approach compares with some of the alternative approaches available.   

Overreliance on a single dataset (PG19) may be an issue. The authors used a random sample of 40 books with high perplexity from within this data set. Though the reasons for this manner of selection are stated in the paper, given that the models may have seen this dataset during training, the high starting perplexity is likely to bias the results. It may be easier to obtain strong percentage-wise reductions from a higher starting point than a lower one.
questions: questions: Acronyms should be spelled out when they are introduced in the abstract and introduction. The manner in which the paper introduces the terms KV cache and Lora imposes unnecessary presuppositional burdens on the reader.  The KV cache is not really an organic part of a language model, but a heuristic designed to make inference more efficient. Similarly Lora. These issues should be easily remediable before the final version is ready.
paper_summary: paper_summary: This paper propose a method called Temp-Lora, which embeds the context information into a Lora module for each chunk to enable long text generation. The proposed method is implemented based on Llama2 and evaluated on two benchmark datasets. The performance on PPL, memory usage and latency get improved compared with Llama2.
strength_weakness: strength_weakness: 1. The proposed method has only been evaluated on Llama2, leading to limited generalization.

2. Temp-Lora performs better than Llama2. This is good. But the paper only shows the performance of Llama2, lacking some strong baselines such as flashattention, pagedattention. Therefore, we can't know how good Temp-lora is compared with some strong models.
questions: questions: Can the author show some cases generated by the proposed method? Only providing the PPL can't intuitively show the generation performance of the proposed method.
paper_summary: paper_summary: This paper introduce a simple and effective framework Temp-Lora to extend the context window for LLMs for long text generation.
The author argues that traditional methods (length extrapolation & context window extension) demand significant hardware resources
during training / inference,  so Temp-lora is proposed to store context information in model parameters instead of KC cache.
The parameters of Temp-Lora is training on the fly using previously-generated tokens, and discarded when the inference is completed.
Experiment on PG19 data-set shows a significant improvement in PPL/BLEU, and a large amount of memory reduction and latency decrease.
strength_weakness: strength_weakness: 1. I think the related work comparison is weak. In the intro, the author strongly claims that "Temp-Lora is compatible with and enhances most existing long text generation methods."  But I don't see enough evidences in the experiment part to support this claim.  So maybe the author can elaborate on this point.
2. It's exciting to see good experimental results  (achieving PPL reduction, memory reduction & latency reduction at the same time),
but on the other side of the coin, I think it's also partially because the baseline is relatively weak, maybe adding more comparison system with the length extrapolation and context window extension methods will make the results more convincing.
3. One of the main advantage of this method is preserving the hardware resources, I assume it also includes computation cost, 
and the on-the-fly training of Temp-lora requires additional GPU during inference.  So maybe the author could add more detail on this direction, and list out computation overhead for a better presentation of this paper.
questions: questions: see weakness part.
Also in Figure 1, there is a missing arrow to point to Chunk1 to "Generate"

Comments: title: Thanks
Thanks
Reply to authors
A Discussion about some common questions
comment: Thank you for your comments!
I have read the authors' response.  I think this is a decent paper. If some of the issues can be fixed, it may go over quite well.
Thank you for your recognition of our work! 

We sincerely apologize for directing one reviewer to read the responses addressed to another reviewer. In fact, if there were no word limit for the rebuttal, we would certainly not have done so. However, due to the word limit of the COLM rebuttal (2500 characters), we were unable to fully include responses to all concerns in your rebuttal. In fact, the rebuttal provided to you has already reached 2495 characters. We resorted to this measure out of necessity, and we apologize once again for disrupting your review experience.
Thanks for the authors' response! I think most of my concerns have been addressed. Therefore, I increase my score to 6.

By the way, I feel it's not a good idea to direct one reviewer to read the responses addressed to another reviewer.
Dear reviewers! Thank you for your hard work. 

During the rebuttal phase, we conducted many experiments and addressed most of your questions in detail: 1) Comparison of Temp-Lora with other existing long-text generation methods like Dynamic-NTK (Please refer to **Response to concern 1 of Reviewer 3Zws**); 2) The effect of Temp-Lora on other open-source models (Please refer to **Response to concern 1 of Reviewer Sdmf**); 3) Why we don't use benchmarks like LongBench (Please refer to **Response to concern 2 of Reviewer 3Zws**).

We are confident that after reading the rebuttal, you will have a new and more comprehensive understanding of the effectiveness of Temp-Lora. In fact, we have already applied Temp-Lora technology in many real-world applications, such as: 1) In our chat-style assistant, users can upload infinitely long documents as background information, and we will use these documents to initialize a Temp-Lora module and continue training it with subsequent user conversations. 2) We have developed some lifelong game NPCs that need to accompany users for a very long time and gradually have "unique memories" between them. These memories are implemented with Temp-Lora. 3) We also have a translation tool specifically for professional translators. They can update the Temp-Lora module with their translation history when translating long documents (such as novels) to maintain better consistency in subsequent translations (this scenario is very similar to our second experiment).

We believe that Temp-Lora represents a new paradigm for storing long-context information: in addition to being stored in the context window and external databases, model parameters can also effectively store historical knowledge. This is also consistent with human nature: people do not re-attend all context information every time they make a decision, most of the time we rely on the memory in our brains (model parameters) to make decisions, only when we need detailed information and can't remember it, will we review past records (KV cache and RAG) to help make decisions.

If you have any other questions, please send them before the end of the discussion period, and we will answer them as soon as possible.