Online Adaptation of Language Models with a Memory of Amortized Contexts
Authors: Jihoon Tack
Jaehyung Kim
Eric Mitchell
Jinwoo Shin
Yee Whye Teh
Jonathan Richard Schwarz
Venue: NeurIPS 2024 poster (2024)
Keywords: Language Model
Online Learning
Continual Learning
Meta-Learning
Primary area: online_learning

Abstract: Due to the rapid generation and dissemination of information, large language models (LLMs) quickly run out of date despite enormous development costs. To address the crucial need to keep models updated, online learning has emerged as a critical tool when utilizing LLMs for real-world applications. However, given the ever-expanding corpus of unseen documents and the large parameter space of modern LLMs, efficient adaptation is essential. To address these challenges, we propose Memory of Amortized Contexts (MAC), an efficient and effective online adaptation framework for LLMs with strong knowledge retention. We propose a feature extraction and memory-augmentation approach to compress and extract information from new documents into compact modulations stored in a memory bank. When answering questions, our model attends to and extracts relevant knowledge from this memory bank. To learn informative modulations in an efficient manner, we utilize amortization-based meta-learning, which substitutes an otherwise required optimization process with a single forward pass of the encoder. Subsequently, we learn to choose from and aggregate selected documents into a single modulation by conditioning on the question, allowing us to adapt a frozen language model during test time without requiring further gradient updates. Our experiment demonstrates the superiority of MAC in multiple aspects, including online adaptation performance, time, and memory efficiency. In addition, we show how MAC can be combined with and improve the performance of popular alternatives such as retrieval augmented generations (RAGs). Code is available at: https://github.com/jihoontack/MAC.

Online Adaptation of Language Models with a Memory of Amortized Contexts
Jihoon Tack', JJahhung Kim Eric Mitchell Jinwoo Shin', Yee Whye Teh Jonathan Richard Schwarz² KAIST Yonsei University SStanford University
University of Oxford Harvard University Thomson Reuters jihoontackkkaist.aac ac.st.....k..
Abstract
Due to the rapid generation and dissemination of information, large language models (LLMs) quickly run out date despite enormous development costs. To address the crucial need to keep models updated, online learning has emerged a critical tool when utilizing LLMs for real-world upplications. However, given the ever-expanding corpus of unseen documents and the large parameter space modern LLMs, efficient adaptation is essential. To address these challenges, we propose Memory of Amortized Contexts (MAC), efficient and effective online adaptation framework for LLMs with strong knowledge retention. We approach to compress
propose feature extraction and eeemory-augmentinn and extract information from new documents into compact modulations stored in memory bank. When answering questions, our model attends and extracts relevant knowledge from this memory bank. To learn informative modulations in an efficient manner, we utilize amortization-basee based meta-learning, which substitutes an otherwise required optimization process with a single forward pass of the encoder. Subsequently, we learn choose from and aggregate selected documents into a single modulation by conditioning on the question, allowing us to adapt a frozen language model during test time without requiring further gradient updates. Our experiment demonstrates the superiority of MAC in multiple aspects, including online adaptation performance, time, and memory efficiency. addition, we show how MAC can be combined with and improve the performance of popular alternatives such as retrieval augmented generations (RAGs). Code is available at: https: / gi thub. com/ /hoontack/MAC..
1 Introduction
Language models (LMs) [7, 79] have significantly accelerated progress in natural language processing (NLP) and thus become
technology in various real-world applications, such coding assistants [10], search engines [90], and personal Al assistants [16]. However, LMs are typically static artifacts, and the world changes, the knowledge encoded in their parameters becomes outdated. This becomes especially problematic for large language models (LLMs), as multiple applications (e.g.. Chatbots [34, 55]) require the model be up-to-date, yet retraining LLMs with new documents from scratch requires high computational demands [31].
To tackle this issue, multiple studies suggested online and continual learning frameworks for LMs, i.c., adapting the LM on stream of new documents. One line of work proposes to use retrieval- -uggmeeedd models by saving the stream of documents and selecting the most relevant document based on the input [9. 33). However, even large models often fail to update their learned knowledge when the retrieved document consists of counterfactual information [48, 44, 75] and it may not be suited for
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Context Documents
Stream of Documents
Save Amortized Contexts
Amartization Network
Aggregation Network
Answer
y
d
M
d
Pamort
Φ1
v
basa
Damort
ΦK
Adapted LM
MAC: Mamory Amortized Contexts
Question
x
Ó Prrpur.
Question
Learning to Amortize and Aggregate
Online Adaptation
An overview of MAC: we amortize each context document into PEFT modulation ø and learn to aggregate modulations into single target modulation ø based on the given question input x to adapt the frozen LM Baase. During online adaptation, we store the amortized contexts into a memory bank M, then adapt the LM via aggregating the memory bank based on the given question.
Figure 1:
edge computing as large number of documents poses expensive computation for model inference [26]. Due to these limitations, another line of recent works suggests finetuning the model on stream of documents to directly update the knowledge inside the LM (i.e., online finetuning [42, 32). While effective, online finetuning schemes also face limitations such as a large computation for gradient calculation, the sensitivity of the online optimization hyper-pate parameter [26], and the aforementioned catastrophic forgetting problem [50, 39]. In this paper, we instead ask: Can we tackle the limitations of retrieval augmented models and online finetuning by assimilating and retaining knowledge from incoming documents without the need for gradient-bute based learning at test time?
To this end, we suggest bridging this gap through complementary learning systems approach [41] by introducing an end-to-end differentiable auxiliary retrieval augmentation system that can be run alongside (frozen) target LM. This system extracts knowledge from incoming documents, builds memory bank, and learns to automatically select relevant information from this memory bank, which is subsequently passed as additional input to the target model. Once learned, this system can be effectively employed purely through forward passes.
Contribution. We propose Memory of Amortized Contexts (MAC). an efficient and effective online learning framework for LMs (see the overview in Figure 1). The core idea of MAC is to freeze LM parameters (thus reducing undesirable side effects common for online finetuning) and instead incorporate new information through additional learned input tokens (an established Parameter-Effiche Fine- Tin-TTnnig technique [47]), utilizing amortization-be based meta-learning [19, 65]. Specifically, instead of optimizing individual PEFT tokens (which necessitates labels and gradient computations), we instead learn to directly predict these tokens based on query and memory bank alone, without the need for labels at test time, thus proposing amortized optimization 49].
To ensure the scalability of MAC, we propose two memory-efficient techniques for training and inference: (I) We find that the process of training our complementary retrieval and aggregation operation for LLMs, necessitates a sufficiently large batch size, which introduces significant memory constraints. To address this issue, we backpropagate on only a random subset of documents, signifi- cantly saving memory while still providing an unbiased approximation of the full gradients [6]. (2) Large memory banks can further increase GPU memory usage when aggregating information relevant to query during inference. To address this, we propose a divide-and-congue the large set of modulations into smaller, manageable groups and repeating this procedure with the predicted modulations until the final modulation parameters are determined.
approach, sub-grouping
We verify the efficacy of MAC through evaluations on multiple datasets and architectures. Overall, our experimental results demonstrate the strong results of MAC. For instance, when measured with the Fl score (%), MAC improves performance from 18.97 - 21.79 over prior work on StreamingQA [45], and 18.66 > 21. 14 on SQuAD-See [26]. Furthermore, we demonstrate that MAC shows significant effectiveness retaining learned knowledge when compared to other online finetuning baselines, justifying the neemory-augmentee retrieval augmented generation (RAG) and in effect, further increases the selection quality of retrieved documents, resulting in an improvement of 71.83 + 74.89 over BM25 alone [66] on ArchivalQA Finally, we highlight the efficiency of MAC in multiple aspects, measuring adaptation time, training, and inference memory usage, again demonstrating strong improvements over baselines.
approach. In addition. MAC can be readily combined with

---

2 Related Work
Amortization-based meta-learning. Amortization- based meta-learning which encodes the given context to directly predict the task model, has gained much attention due to its computational efficiency as it only requires a single encoder forward pass when adapting the model [69, 51, 19, 18]. These approaches, especially when combined with modulation techniques, have achieved notable success in various applications, such as few-shot visual recognition [65, 6, 11] and 3D reconstructions [20, 35]. Recently, this idea has been extended to language domains where prior works facilitate hypernetworks to adapt LMs with given feo-shot prompts [58, 28]. In this paper, we extend the use of amortizzttion-aase based meta-learming to extract the knowledge of a given document into compact yet informative modulation for online adaptation.
Online learning. Online learning, also referred to as continual or lifelong learning, is task of adapting models to new data or task distributions [77]. Such ideas are becoming increasingly relevant in the era of deep learning generally and with the advent of extremely large models [78. 17, 71] specifically, In the language domain, there have been various attempts to tackle online learning [40, 92, 63] where recent studies focus more on online learning of LLMs, e.g., finetuning on stream of documents [42], architectural constraints [32], and the use of replay buffers [14]. Among them, Hu et al. [26] found that online finetuning can be effective when an LM focuses on important tokens during the adaptation and proposed a muddient-based meta-learning approach to automatically learn token importance weighting model. However, such gradient-based meta-learning schemes require a compute-cxpensive second-order gradient calculation [15, 64]. Morcover, online finetuning schemes can face multiple challenges, including (i) inevitable forgetting of the learned knowledge, (ii) gradient computation of LLMs during adaptation, and (iii) high sensitivity to the online optimization hyperparameter (e.g., learning rate [26)). MAC does not suffer from such issues as amortization strategy is efficient without introducing any hyperparameters while effectively preserving knowledge.
Retrieval augmentation for LMs. Retrieval augmentation of LMs with relevant information from external knowledge sources has served as an effective way to improve the performance of LMs on various NLP tasks [21, 43, 30, 70, 80] by reducing hallucination and leveraging external knowledge which is not seen during pre-traiming. However, retrieval augmentation drastically increases computational cost [88] as documents often consist of thousands of words. In addition, its effectiveness is sensitive to the configuration of retrieved information [46], and even negatively affects the performance of LMs when the retrieved information is counterfactual [75]. MAC is more efficient than retrieval augmentation as it amortizes the external knowledge to modulate LMs rather than directly incorporating it. Furthermore, we believe MAC and retrieval augmentation has similarities as both methods store the knowledge and utilize them base on the user query, while the main difference is that MAC attend to multiple documents simultaneously using the aggregation network, allowing the LLM to capture shared information across documents. We thus believe that the joint usage benefits retrieval augmentation, MAC can guide retrieval augmentation to capture missing information not retrieved by the retriever (see Section 4.1 for the supporting experiment).
Memory augmented LMs. Recently, memory augmentation has also shown great promise for LMs where it significantly improves the performance and efficiency in various directions [84, 56, 94, 54, 24]. e.g., extending context length with memory retrieval [87, 83], personalization [2], and model editing [53]. Unlike these methods, which store the raw text or use the memory bank to train new LMs, MAC stores compact modulation parameters (in the shape of learned tokens) and adapts the frozen target LM, thereby utilizing large models without the heavy computation of training LMs.
3 MAC: Online Adaptation with a Memory of Amortized Contexts
In this section, we first briefly describe our problem setup (Section 3.1), then core components, namely amortization and aggregation framework (Section 3.2) and finally, efficient training and inference schemes for MAC (Section 3.3). Algorithm and 2 in Appendix B provide detailed training and online adaptation processes for our framework.
3.1 Problem setup: Online adaptation
We consider the online adaptation scenario proposed in Hu et al. [26] where static LM parameterized by Ossse is adapted to an online stream of documents Ctest = (di, .g ). After incorporating
3

---

the final document, we then evaluate the adapted model's performance with set of queries {x,} and a corresponding labels [y.]. where the hth query and label are drawn from a conditional distribution of a document d, i.e., X+) ~ p(x, y yd,). Here, note that the query x; is not accessible during online adaptation; hence, retaining the learned information from d, critical for achieving good results. While the query input and label pair (x.y) y) can be in any format or task, we mainly focus on question and answering (QA) tasks by following Hu et al. [26]. i.c., x; is a question and y; is the corresponding answer based on the given information in d;, as it is straightforward to evaluate the LM's updated knowledge. Nevertheless, we also consider an additional non-QA setup in Section 4.3.
3.2 MAC: Memory of amortized contexts
The stated goal of MAC is (i) the efficient adaptation of a given LM to unseen information (ii) while retaining previously learned knowledge, both from its original training stage as well as updates from prior examples in stream of novel data. To this end, we propose to utilize amortization- based system. Amortization-bant bbase meta-learning with modulations [27, 65, 4] learns to predict task-specific modulation (i.e., compact representation of a task) through amortizing the given context set sampled from the task distribution. This enables efficient adaptation using the learned amortization network, as it only requires single forward pass to adapt a model, foregoing the cost of gradient computation. is worth noting that this is also beneficial as the LM does not have access to the input and label pair (x. y) during the online adaptation, where we can design the amortization to find the modulation only with the given document d. Furthermore, meta-learned modulations have been found to preserve the task information well (c.g.. showing great potential for generating or classifying distributions of tasks [72, 73). They can hence be expected to effectively extract document information. Based on this insight, suggest meta-learning the amortization network to directly predict a compact modulation for a new document.
meta-learning [18. 19] of memory-augment
Learning to amortize contexts. For given context document da sampled from the training docu- ment set Crrain, we learn an amortization network parameterized by Oanort. to predict a modulation parameter (of the same shape as embedded tokens) Φb as: Φx == 90 (dx).. Here, we use a hyper- network [22] for a we modify the T5 architecture [60] by having learnable tokens as the input of the decoder to have a consistent number of output tokens by following [58]. One can design the
modulation with any type of PEFT scheme (c.g., LoRA [25] or FILM [57]), among which we use P- P-Tuning v2 [47] (i.e.. predictions of the key-value of each attention layer).
Modulating LMs via aggregating amortized contexts. Given a memory bank of compressed documents in the form of modulations {Φx } k= we now learn to choose relevant information in the form of a modulation of for given input X,. While one design choice is to select/retrieve single modulation, this has two drawbacks: (i) risk of selecting the wrong modulation and (ii) limited utilization of learned knowledge across different modulations. Moreover, it is worth noting that recent studies empirically show that linear interpolation (or advanced merging) between the modulations trained from the same pre-trained LM can even perform better than individual modulation (coined "model soup" [86, 93]). In this regard, we thus aggregate the memory bank into a single modulation based on the given input. Formally, we learn a set aggregation network hyg that satisfies permutation invariance (i.e., invariance the order of modulations in the memory bank) by utilizing cross-attention blocks [81, 36, 89] to select o;:
o = hep m00 (x,), (cx (o
(I)
where 0imput is the input encoder, and we use the same architectural design as the amortization
network fanort. albeit resorting to reduced number of parameters for efficiency reasons. Note that [K is often referred to as a context set in the meta-learning literature, hence inspiring the
name of our method. We provide more architecture design details of Oanort and i in Appendix A.
End-to-end training objective. To learn aggregation and amortization networks, we optimize both networks in an end-to-end fashion as follows:
1 N
Σ (LMM (x);))).
min Oxsers Bengues
where is is the loss function, i.e., negative log-likelihood of the given label y, and N is the batch size of training query inputs and labels. Here, it is important to state that we make no updates to the static LM 0 frese, which would carry the risk of catastrophic forgetting by overwriting important parameters.

---

Online adaptation stage. After training amortization and aggregation networks based on given training set, we now consider the online adaptation scenario. Here. we consider a stream of K Kteat documents di .... Kut given to the LM in sequential manner, where the task input x x is not accessible during adaptation. To this end, we propose to store the compact modulations into a memory bank M == {g (d ] 1 and later predict the modulation using the aggregation network to adapt the LM, i.e., LM (xtest, ;0) where o² == hap (g0 (x M)..
3.3 Memory efficient training and inference for MAC
Due to aforementioned challenges, the training of MAC can quickly become prohibitive. The following sections cover techniques drastically reduce memory requirements.
Backpropagation dropout. During the online adaptation stage, the aggregation network is required to predict the modulation based the memory bank, which may consist of large numbers of modulations (examples extracted from thousands of novel documents in our experimental setup). To handle large batch inference, it is crucial to present similar examples during training to avoid distribution shift between training and online adaptation stage and ensure that memory selection robust. To this end, propose memory-effiiiett way to increase the training context size K by computing gradients using only a subset of randomly chosen examples (ensuring unbiased gradient computation), thus allowing training with significantly larger memory sizes. More concretely, with probability p. we perform amortization at training time with stop- -opgdreddent operation, i.c., important
stopgrad Me (d:)) where p is a syper-parameter. thus reminiscent of dropout. to note that this random sub-sampling yields unbiased approximation of the full gradient under amortization-bast meta-learning schemes [6], hence, does not hurt the overall performance.
Hierarchical modulation aggregation. In addition, we propose an efficient inference technique to deal with the accumulated memory bank. Let T be the number of output tokens for each context and K the number of amortized contexts, respectively. Then, the memory usage made by a single cross-attention layer becomes O(KT) (note that the input is also mapped into T tokens). This indicates the aggregation process requires memory cost that linearly scales with the size of the memory bank.
To alleviate memory consumption, we propose hierarchical modulation aggregation that uses a prride-and-conquue strategy (see Algorithm 3). Specifically, for given memory bank size of K with T tokens, we subgroup the total KT tokens into M tokens each, thereby having KTT groups ([-] is the ceil function, i.e., the smallest integer which is greater than or equal the given input). Then, we aggregate the modulations of individual subgroups into a single output to obtain [KT modulations. We repeat this procedure until it outputs a single modulation. Assuming no parallelization, one can compute this process by only utilizing the memory complexity of ((T)) where M is a hyperparameter (more details of the complexity calculation in Appendix A.2).
4 Experiments
In this section, provide empirical evaluation of MAC, systematically verifying claims made throughout the manuscript and thus supporting the suitability of its constituent components. Specifi- cally, we investigate the following questions:
How does MAC perform compare to other online learning techniques for LMs? (Table & Table 2)
MAC more efficient compared to online finetuning schemes? (Figure 2)
Does MAC show effective knowledge retention compared to other finetuning methods? (Figure 3)
Does proposed efficient training and inference schemes save memory usage? (Figure 4 & Figure 5)
Before answering each question, we outline the experimental protocol (more details in Appendix A).
Datasets. For the experiment, we utilize three StreamingQA [45], SQuAD [62], and ArchivalQA [82], by following the prior work [26]. Here, unlike the original use of SQuAD and ArchivalQA (i.e., used for evaluating static LMs), we use these datasets for online adaptation (i.e., adapting on stream of documents), hence, denote with an additional "-Seq" notation throughout the section.
(QA) datasets including
5

---

Table I: Comparison of the online adaptation performance between MAC and online finetuning baselines. We report the exact match (EM) and F1 score by adapting the LM on a stream of documents and then performing QA based on the learned data. denotes the adaptation results of CaMeLS using a proxy token weighting LM (i.e., a smaller LM than the base LM) due to memory consumption, and OOM denotes unavailable results due the running out-of-memory on a single NVIDIA A100 80GB GPU (even with a batch size of 1). The bold indicates the best result within the group.
StreamingQA
SQuAD-Seq
ArchivalQA-Sen
Model (# params)
Method
EM (t)
FI (t)
EM (t)
FI (T)
EM (1)
F1 (f)
Uniform
1.62
3.76
1.24
2.54
4.86
4.08
DistilGPT2 (82M)
Salient Spans
1.44
4.67
1.03
2.47
4.52
3.76
CaMeLS
1.62
5.79
1.47
3.08
4.62
6.19
MAC (ours)
5.59
10.18
2.01
6.85
7.55
10.58
Uniform
4.74
7.00
3.64
4.97
7.66
8.71
GargeLarge
Salient Spans
4.86
8.54
4.03
6.48
9.75
11.19
(774M)
CaMeLS
5.35
10.60
4.97
8.63
9.92
12.41
MAC (ours)
7.25
13.31
6,43
11.42
11.84
15.26
Uniform
5.11
7.48
6,10
6.78
8.61
10.78
GPT2-XI.
Salient Spans
5.40
9.42
4.55
6.74
11.81
14.11
(1.5B)
CaMeL
6.55
11.67
6.70
10.15
13.87
15.74
MAC (ours)
8.99
15.38
7.10
12.55
14.01
17.12
Uniform
12.43
13.54
13.25
17.01
18.53
21.35
LLaMA-2 (7B)
Salient Spans
13.33
18.97
13.74
18.66
18.97
22.75
CaMeLS
OOM
MAC (ours)
14.29
21.79
15.07
21.14
20.12
23.90
Online adaptation setup. After training MAC (i.e., learning fanort, Iinutt. and sb parameters) on training dataset that consists of document and QA pairs, evaluate the online adaptation performance on the stream of documents. Here, we use 1,665 documents to adapt the LM and then perform the evaluation after the adaptation, where QA pairs are sampled from the learned documents. Each document can consist of tokens up to 512 when using the Byte Pair Encoding [74].
Baselines. We mainly consider the online finetuming baselines introduced in [26], including Uniform, Salient Spans and CaMeLS. Here, all basclines are first pre-trained on QA-paired training set (without the documentation) and then utilize auto-regressive finetuning to adapt the stream of documents. Specifically, Uniform uses uniform token weighting, Salient Spans assigns uniform weight tokens in salient spans [21] and no weights other tokens, and CaMeLS utilizes the output of the token weighting LM (which is meta-learned to predict the important token so that the performance of the adapted LM maximized). Furthermore, also consider the joint usage of MAC with the retrieval augmentation scheme, including BM25 [66], Contriever [29], and DPR [33].
4.1 Online adaptation with MAC
We first present the main result by comparing the online adaptation performance with other baselines. Here, mainly compare with online finetuning schemes and additionally show that MAC can be jointly used with retrieval augmentation method to further improve the performance.
Comparison with online finetuning methods. In Table 1, we show the online adaptation perfor- mance of MAC and the online finetuning basclines. Overall, MAC significantly outperforms all the prior online finetuning methods by large margin, leading to a better exact match (EM) and Fl score. We also found that CaMeLS [26] suffers from the memory shortage on LLaMA-2 even when using the memory efficient techniques (e.g., 4bit quantization [13] and ZeRO [61]), as it requires -econd-order gradient computation for meta- leeaa-learnin. Consequently, it requires a proxy model (a small-sized LM compared to the base LM) that uses the same tokenization (e.g., we use DistilGPT2 for GPT family as suggested in [26]).
Furthermore, it is worth mentioning that MAC is significantly efficient in both memory and adaptation time compared to other online finetuning methods; we remark that MAC does not require any gradient computation to update the model, while online finetuning needs the gradient update the model. For

---

Untomm
Sallett
GaMeLS
lours)
O Unitorm
Salent
CaMiLS
(ours)
F1 soore retertion rate (%)
100
Pask memory allocation (GB)
Adaptalion time (min)
30-
40.0%
20-
10.21%
leex ca
80-
10-
10
200
400
800
800
1000
1200
1400
1600
Number of adapted documents
(a) Memory
(b) Time
Figure Comparison of the adaptation mem- ory and time efficiency between MAC and online finetuning baselines. We report the peak GPU memory allocation (GB) for adapting one docu- ment and the time (min) for adapting a stream of 1,665 documents under the same memory usage. We use GPT2-XL on StreamingQA.
Figure 3: Catastrophic forgetting analysis under GPT2-XL trained on StreamingQA dataset. We report the Fl score retention rate (%) through mea- surement of relative Fl score decline in the ini- tially adapted 200 documents during subsequent adaptation to new stream of documents (up to additional 1,400 documents).
Table 2: Online adaptation performance of MAC jointly using the retrieval augmentation under ArchivalQA-See dataset. We consider BM25, Contriever, and DPR as retrieval augmentation methods. We report the exact match (EM) and F1 score by adapting the LLaMA2-7B on stream of documents and then performing QA based on the learned data while retrieval augmentation retrieves documents. The bold indicates the best results within the group.
Top-1
Top-3
Top-5
EM
F1
EM
Fl
EM
BM25
48.53
54.17
56.18
63.74
64.74
71.83
BM25 + MAC (ours)
52.81
56,55
60.22
66.82
68.85
74.89
Contriever
44.78
51.55
52.56
61.28
60.10
67.83
Contriever + MAC (ours)
47.99
53.23
53.92
63.75
61.28
70.01
DPR
48.98
55.01
57.02
64.27
65.07
72.24
DPR + MAC (ours)
49.57
55.98
60.19
67.05
68.52
75.00
instance, compared to CaMeLS, MAC reduces 68.0% memory usage for single document adaptation and can adapt 128 times larger number of documents when using the same memory. Moreover, the adaptation time reduces from 28.58 to 2.5 minutes under the same memory usage (i.e., 90.31 % drop). We emphasize that both types of efficiency are crucial for online learning LMs as i) the document corpus is expanding rapidly, and ii) it enables the user to use a larger model for better generalization.
Knowledge Retention of MAC. We now address one of our primary motivations for this study: comparison of knowledge retention by analyzing the catastrophic forgetting of each method. To this end, we evaluate the F1 score retention ratio, which is determined by the decline in the Fl score of the initially adapted 200 documents during the optimization on a subsequent stream of documents. As shown in Figure 3. MAC shows strong knowledge retention compared other online finetuning methods: when adapting additional 1,400 documents, MAC retains the initial performance by 96.2% while CaMeLS retains 70.8%. These results indeed highlight i) the benefit of using a memory bank as a tool for preserving knowledge and ii) our aggregation mechanism well predicts the modulation even when the memory bank's 's cardinality increases throughout the adaptation process. It is also worth noting that online finetuning schemes somewhat suffer from preserving the newly learned knowledge, especially when the number of adapted documents increases, thus may limit the practical usage for real-world applications.
Improving MAC with retrieval augmentation. In addition, we show that MAC can be further improved by using retrieval augmentations. Here, we note that the user requires more inference costs to use retrieval augmentations as prepending the retrieved document front of the question quadratically increases the inference computation based on the document length due to the Attention mechanism [81]. For the experimental setup, we compare it with LMs that are pre-trained on QA training set with an appended top-1, top-3, and top-5 retrieved document for each question, i.c.,
7

---

000000
& 0.25 4 0.50
C 0.755 + mn
Memory (GB)
score
Peek memory allocation (GB)
48
Pesk memoryy allocation (GB)
16
42-
12
65.6%
12
36-
GO
30
24
2
18
32
16
32
64
Full
Cardinaliy of the training amortized context set (K!tar)
Subgroup cardinally (M)
Figure 4: Memory efficiency of the backpropa- gation dropout. We report the peak GPU mem-
Figure 5: Memory efficiency of the hierarchi- cal modulation aggregation. We report the peak GPU memory allocation (GB) and F1 score under GPT2-XL trained on ArchivalQA-See dataset by varying the subgroup cardinality M. The "Full" indicates the use of the full context (i.e., no hierarchical aggregation).
ory allocation (GB) when training GPT2-XL on StreamingQA dataset under varying sizes of amor- tized contexts set size (Ktrrain). p indicates the dropout ratio and 'min' denotes the full dropout except for the single document.
LM (d x: () where and é indicate concatenation and the modulation, respectively. Here, we consider three types of popular retrieval augmentation methods, including BM25 [66], Contriever [29], and DPR [33]. As shown Table using BM25 with MAC significantly improves the performance by large margin in all cases, e.g., F1 score of 71.83% + 74.89% for LLaMA-2 (7B) when using top-5 documents. We conjecture that the aggregation process of MAC enables the utilization of the shared information across the documents, thus improving the performance over the single document retrieval. We believe further extending MAC for the joint usage with retrieval augmentation schemes will be an interesting future direction to explore where one can extend the amortization and input network to enhance the aggregation modulations but also learn to well retrieve documents.
4.2 Efficiency backpropagation dropout and hierarchical modulation aggregation
We verify the proposed memory efficient techniques, namely the backpropagation dropout and the hierarchical modulation aggregation for training and inference, respectively. Here, we report the peak GPU utilization when using the proposed techniques to show the memory efficiency. Furthermore, we re-emphasize that such techniques are important for (i) scaling LMs to larger models and (ii) handling a large number of documents during online adaptation, which are both necessary for scaling.
Training memory efficiency. To show the memory effi- ciency of the backpropagation dropout, we increase the number of amortized contexts K train during training time and vary the dropout ratio p. As shown Figure 4, increasing the dropout ratio can significantly handle more contexts under the same memory constraint. As result, we found that simply using p 0.75 is an effective choice when using large models parameters > IB) as the training context size is small in such cases. For instance, when training LLaMA-2 (7B) model on Streaming dataset without this technique, one can only compute the loss with single document (under 32 GB GPU), thus the aggregation network cannot learn the similarity between the modulations. As result, using backpropagation dropout improves the performance of LLMs (in Table 3).
Table 3: Effect of backpropagation dropout (backprop.) on I.aMMA2-BB under Stream- ingQA dataset. K indicates the batch size.
Method
Memory (GB)
No backprop.
33.86
12.43
MAC
34.01
21.79
Inference memory efficiency. Here, we show that the hierarchical modulation aggregation can significantly reduce memory usage while effectively preserving the performance for the inference. To this end, we vary the cardinality of the subgroup M and report the peak GPU memory usage and Fl score where we only measure the used memory by the modulation aggregation (i.e., excluding the LM cost). As shown in Figure 5, using the subgroup size of M = 16 can reduce the memory by 65.6% while still preserving 93.2 93.2% of the original accuracy. We remark that this technique can be applied even without additional training trick or regularization, demonstrating similar observations from the prior works that uses hierarchical aggregation (or merging) the context of Transformers [5, 76], yet MAC is the first to aggregate the modulations.

---

Question
Quantion
TokenT
Takons
Tokena
Tokens
Tokena
Tokans
Takana
Tokana
Tokera
Tokena
Tokena
Tokena
(a) BM25 retrieved documents
(b) Random documents
Figure 6: Visualization of per-token final layer cross-attention. The aggregation network is provided with the gold document (containing the answer) with five additional documents, which are either (a) retrieved using BM25 or (b) randomly sampled. Each question and document are encoded into K 12 tokens, where K is hyperparameter. Red denotes the high similarity with the question.
4.3 Additional analysis
In this section, provide more analysis of MAC. Here, mainly consider baselines that show effectiveness in the main experiment (c.g., CaMeLS in Table 1) and consider GPT2 family trained with StreamingQA dataset.
Cross-attention analysis. We analyze whether the learned cross-attention is attending to the correct information. To this end, we visualize the final cross-attention layer of the aggregation network trained on StreamingQA with Garg-Large. where we provide the gold document (containing the answer the question) and an additional five documents. Here, we consider providing the retrieved documents using BM25 or random documents, where we average the cross-attention over 25 questions (as considering more number of questions over-smooth the visualization). As shown in Figure 6, the model selectively attends to the gold document when provided with irrelevant random documents, effectively ignoring them, while appropriately attending to relevant documents retrieved using BM25, indicating a well-trained attention mechanism capable of discerning useful information.
Memory bank size constraint. One possible concern of MAC the growing size of the memory bank as the number of adapted documents increases. To this end, we have conducted an additional experiment using a fixed memory bank size for MAC. Specifically, we reduce the number of amortized con- texts when reaches the memory constraint of 1,250 (where the total number of contexts is 1665). Here, consider three simple yet effective schemes: i) random pruning, ii) randomly averaging two modulations onew = [ Φ2),, and iii) av- eraging two nearest-neighbor (NN) modulations based on the cosine distance. As shown in Figure 7, tested LLaMA-2 7B on StreamingQA by reducing the memory bank size where averaging NN modulations shows quite effective preservation. We believe it would be an interesting future direction to further explore MAC under memory bank size constraints where great variety techniques can be developed in this direction, for instance, using neural compression techniques reduce the memory bank size [3, 73].
Random Avg. Full Memory
Random Avg.
F1 score
20
19
Figure 7: Comparison of various memory bank reduction methods on LLaMA2-7B.
Using other types of PEFT. Here, we show that other types of PEFT modulation can also be used for our framework. To this end, we considered LoRA [25] as an alternative to P- tuning v2 [47]. As shown in Table 4, LoRA also performs well compared to other online fine-tuming methods, but overall, P- tuning v2 outperformed LoRA when training GPT2-XL. the StreamingQA dataset. This result aligns with the finding from previous work [58], where they also observed that P-tuning v2 outperforms LoRA when using amortization. Additionally, we believe P-tuning is also easy implement, as it allows efficient batch computation, enabling single forward pass of the LLM with different modulations. In contrast, LoRA requires separate forward passes for each modulation, which increases the training time.
Table 4: Online adaptation perfor- mance different types of PEFT, in- cluding LoRA and P-tuning-v2. We train GPT2-XL StreamingQA.
PEFT type
EM
LoRA 8.67
15.15
P-tmming v2
8.99
15.38

---

Adaptation on out-of-distribution (OOD) datasets. We additionally analyze the online adaptation performance of MAC on the OOD dataset from the training distribution, To this end, we compare the performance with CaMeLI [26] on GPT2-XI, as other online finetuning methods do not involve a training stage (i.e., no training distribution). Here, we use StreamingQA as a training set (i.e., rela- tively large dataset) and other datasets as OOD. As shown in Table 5, MAC outperforms CaMeLS in F1 score. is worth noting that the meta-learning performance scales as the training distribution is more diverse [91], hence, we believe training MAC larger datasets will further improve the OOD generalization.
Table 5: Online adaptation performance on OOD datasets: report the Fl score of GPT2-XI. trained on StreamingQA, adapting to SQuAD and ArchivalQA.
StreamQA +
SQuAD
ArchivalQA
CaMeLS
8.63
13.43
MAC (ours)
10.47
13.73
Language modeling with MAC. While the conventional evaluation protocol for online learning LMs uses QA [32, 31, 26], we additionally conducted language modeling task (i.e., predicting the next token). Specifically, we adapted the LLM on a stream of documents, then gave the initial 10% of the document input to the input network (this is equivalent to a question in the QA task). Here, measured the perplexity of the remaining 90% of the documents on two cases: (i) the documents used for LLM adaptation to measure knowledge preservation and (ii) unseen documents to measure generalization. As shown in Table 6, MAC outperforms other online finetuning baselines in both cases.
Table 6: Perplexity on adapted and un- seen documents. We use GPT2-Large auto-regressively trained on Stream- ingQA documents.
Adapted
Unseen
Uniform Salient Spans
11,43
13.89
27.87
29.69
CaMeL MAC (ours)
11.31
14.77
10.91
12.71
Design choice for the amortization network. Here, we consider different types of design choice for the amortiza- tion network. To this end, we evaluated three architectural configurations: decoder-only, encoder-only, and encoder- decoder language models. Specifically, we experimented with (i) the GPT2 model and (ii) the T5 encoder with learn- able tokens, where input context is compacted into these tokens. As shown in Table 7, the encoder-decoder model demonstrated superior performance over other configura- tions, using GPT2-XL as the base LLM on the StreamingQA dataset.
Table 7: Online adaptation performance across design choices for the amorti- zation network, evaluated by training GPT2-XL on the StreamingQA dataset.
EM
F1
Encoder only (T5-encoder)
8.53
15.01
Decoder only (GP12)
8.01
14.87
Sincoder-Deoodee (15)
8.99
15.38
5 Discussion and Conclusion
We propose MAC, an efficient and effective online adaptation framework for static LMs with strong knowledge retention. MAC compresses the context document into parameter-efficiert finetuning modulations, predicted by a meta-learned amortization network. These contexts are stored in a memory bank for strong knowledge retention and aggregated into single output when question is input. MAC excels in performance, adaptation time, and memory efficiency, and shows superior knowledge retention for newly learned documents when handling stream of documents.
Future works and limitations. We believe will be an interesting future work extending MAC to multiple applications that require online learning in an efficient manner, e.g., federated learning for LMs [8] and model editing [52, 53, 23). Moreover, one possible limitation of MAC is the increasing size of the memory bank during online adaptation. In this paper, we found that the memory bank can be effectively reduced by averaging nearest neighbor modulation (in Section 4.3), where we believe further investigating a metter- technique will be an interesting future direction to explore.
Societal impact. This paper presents a method that enhances the online adaptation performance of LMs through the use of amortization- based meta-learning and the memory bank. Similar to other works, using memory banks for LMs in real-world applications comes with benefits and pitfalls (e.g., privacy concerns when saving documents from users), requiring the responsible use of the technology. We believe further extending the amortization network in the perspective of privacy will be an interesting future direction to explore. For instance, rather than saving the raw text other retrieval augmentations techniques or memory- augmented LMs, one can learn to amortize the context documents to prevent the document's privacy leakage.
10

---

Acknowledgements
We thank Nathan Hu and Minseon Kim for providing helpful feedback and suggestions in prepar- ing an earlier version of the manuscript. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (KS-2019-1119007 Artificial Intelligence Graduate School Program (KAIST), No.RS-2021. 12122668, Artificial Intelligence Innovation Hub, 10.2022-0-00711 real-world problems, and No. RSS2024-005092279 Global AI Frontier Lab) and the NIPA(National IT Industry Promotion Agency), through the Ministry of Science and ICT (Hyperscale Al flagship project).
Meta-learning applicable to
References
[1] B. Amos al. Tutorial amortized optimization. Foundations and Trends@ in Machine Learning, 2023.
[2] J. Back, N. Chandrasckaran, S. Cucerzan, S. K. Jauhar, et al. Knowledge-augmennce large lan- guage models for personalized contextual query suggestion. arXiv preprint arXiv:2311.06318. 2023.
[3] Ballé, D. Minnen, S. Singh, J. Hwang, and N. Johnston. Variational image compression with scale hyperprior. In International Conference on Learning Representations, 2018.
[4] P. Bateni, R. Goyal, V. Masrani, F. Wood, and L. Sigal. Improved few-shot visual classification. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[5] D. Bolya, C.-Y. Fu, Dai, Zhang, Feichtenhofer, and Hoffman. Token merging: Your vit but faster. In International Conference on Learning Representations, 2023.
[6] J. Bronskill, D. Massiceti, M. Patacchiola, K. Hofmann, S. Nowozin, and R. Turner. Memory efficient meta-learning with large images. In Advances in Neural Information Processing Systems, 2021
[7] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
[8] T. Che, J. Liu, Y. Zhou, J. Ren, J. Zhou, V. Sheng, H. Dai, and D. Dou. Federated learning of large language models with parameter-efficient prompt tuning and adaptive optimization. In Conference on Empirical Methods in Natural Language Processing, 2023.
[9] D. Chen, A. Fisch, J. Weston, and A. Bordes. Reading wikipedia to answer open-domain questions. In Annual Conference of the Association for Computational Linguistics, 2017.
[10] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et Evaluating large language models trained on code. arXiv preprint arXiv:2107.033777 2021.
[11] S. Chen, J. Tack, Y. Yang. Y. W. Tch, J. R. Schwarz, and Y. Wei. Unleashing the power of meta-tuning for few-shot generalization through sparse interpolated experts. arXiv preprint arXiv:2403.08477 2024.
[12] A. Chevalier, A. Wettig, A. Ajith, and D. Chen. Adapting language models to compress contexts. In Conference on Empirical Methods in Natural Language Processing, 2023.
[13] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. Qlora: Efficient finetuning of quantized llms. In Advances in Neural Information Processing Systems, 2023.
[14] B. Dhingra, J. R. Cole, J. M. Eisenschlos, D. Gillick, J. Eisenstein, and W. W. Cohen. Time- aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10, 2022.
[15] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, 2017.
11

---

[16] D. Gao, Ji, L. Zhou, K. Q. Lin, J. Chen, Z. Fan, and M. Z. Shou. Assistgpt: A general multi- modal assistant that can plan, execute, inspect, and learn. arXiv preprint arXiv:2306.0888 08640. 2023.
[17] S. Garg, M. Farajtabar, H. Pouransari, R. Vemulapalli, S. Mehta, O. Tuzel, V. Shankar, and Faghri. Tic-clip: Continual training of clip models. arXiv preprint arXiv:2310.16226, 2023.
[18] M. Garnelo, D. Rosenbaum, C. Maddison, T. Ramalho, D. Saxton, M. Shanahan, Y. W. Teh,
D. Rezende, and S. A. Eslami. Conditional neural processes. In International Conference on Machine Learning, 2018.
[19] M. Garnelo, J. Schwarz, D. Rosenbaum, F. Viola, D. J. Rezende, S. Eslami, and Y. W. Teh. Neural processes. arXiv preprint arXiv: 1007001870 2018.
[20] Z. Guo, C. Lan, 7. Zhang, Y. Lu, and Chen. Versatile neural processes for learning implicit neural representations. In International Conference on Learning Representations, 2023.
[21] K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang. Retrieval augmented language model pre-training. In International Conference on Machine Learning. 2020.
[22] D. Ha, A. M. Dai, and Q. V. Le. Hyperneworrss. In International Conference on Learning Representations, 2017.
[23] Hartvigsen, S. Sankaranarayanan, Palangi, Kim, and M. Ghassemi, Aging with grace: Lifelong model editing with discrete key-value adaptors. In Advances in Neural Information Processing Systems, 2023.
[24] Z. Hc, L. Karlinsky, D. Kim. J. McAuley, D. Krotov, and R. Feris. Camelot: Towards large language models with training-fiee ffree consolidated associative memory. arXiv preprint arXiv:2402. 13449, 2024.
[25] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022.
[26] N. Hu, E. Mitchell, Manning, and C. Finn. Meta-lcarning online adaptation of language models. In Conference on Empirical Methods Natural Language Processing, 2023.
[27] Humplik, A. Galashov, Hasenclever, A. Ortega, Y. W. Teh, and N. Heess. Meta reinforcement learning as task inference. arXiv preprint arXiv: arXiv:/905.0m 19905.06224, 2019.
[28] H. Ivison, A. Bhagia, Y. Wang, H. Hajishirzi, and M. Peters. Hint: Hypernetwork instruction tuning for efficient zero-shot generalisation. In Annual Conference of the Association for Computational Linguistics, 2023.
[29] G. Izacard, M. Caron, L. Hosseini, S. Riedel, P. Bojanowski, A. Joulin, and E. Grave. Unsu- pervised dense information retrieval with contrastive learning. In Transactions on Machine Learning Research, 2022.
[30] G. Izacard, P. Lewis, M. Lomeli, L. Hosseini, Petroni, T. Schick, Dwivedi- Yu, Joulin, S. Riedel, and E. Grave. Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 2023.
[31] J. Jang, S. Ye, C. Lee, S. Yang, J. Shin, J. Han, G. Kim, and M. Seo. Temporalwiki: A lifelong benchmark for training and evaluating ever-evolving language models. In Conference on Empirical Methods in Natural Language Processing, 2022.
[32] J. Jang, S. Ye. S. Yang, J. Shin, J. Han, G. Kim, J. Choi, and M. Sco. Towards continual knowl- edge learning of language models. International Conference on Learning Representations, 2022.
[33] V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih. Dense passage retrieval for open-domain question answering. In Conference on Empirical Methods in Natural Language Processing, 2020.
12

---

[34] B. Kim, H. Kim, S.-W. Lee, Lee, D. Kwak, D. Hyeon, S. Park, S. Kim, S. Kim, D. Seo, al. What changes can large-scale language models bring? intensive study on hyperclova: Billions-scale korean generative pretrained transformers. In Conference Empirical Methods Natural Language Processing, 2021.
[35] C. Kim, D. Lee, S. Kim, M. Cho, and W.-S. Han. Generalizable implicit neural representa- tions via instance pattern composers. In IEEE Conference on Computer Vision and Pullern Recognition, 2023.
[36] H. Kim, A. Mnih, J. Schwarz, M. Garnelo, A. Eslami, D. Rosenbaum, O. Vinyals, and Y. W. Teh. Attentive neural processes. In International Conference Learning Representations, 2019.
[37] J.-H. Kim, J. Yeom, S. Yun, and H. O. Song. Compressed context memory for online language model interaction. In International Conference on Learning Representations, 2024.
[38] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[39] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins. A. A. Rusu, K. Milan, al. Overcoming catastrophic forgetting in
J. Quan, T. Ramalho, A. Grabska-Barwinsker neural networks. Proceedings of the National Academy of Sciences, 2017.
[40] R. Kuhn. Speech recognition and the frequency of recently used words: A modified Markov model for natural language. In Coling Budapest 1988 Volume International Conference on Compulational Linguistics, 1988.
[41] D. Kumaran, D. Hassabis, and J. L. McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in cognitive sciences, 2016.
[42] A. Lazaridou, A. Kuncoro, E. Gribovskaya, D. Agrawal, A. Liska, T. Terzi, M. Gimenez, C. de Masson d' Autume, T. Kocisky, S. Ruder, et al. Mind the gap: Assessing temporal generalization in neural language models. In Advances in Neural Information Processing Systems, 2021.
[43] A. Lazaridou, Gribovskaya, W. Stokowiec, and N. Grigorev. Internet-augmented language models through few-shot prompting for open-domain question answering, arXiv preprint a5Xiv:2203.05115, 2022.
[44] W. Li, W. Wu, M. Chen, J. Liu, X. Xiao, and H. Wu. Faithfulness in natural language generation: A systematic survey of analysis, evaluation and optimization methods. arXiv preprint arXiv:2203.05227, 2022.
[45] A. Liška, T. Kočiský, E. Gribovskaya, T. Terzi, E. Sezener, Agrawal, C. d. M. d' Autuuee T. Scholtes, M. Zaheer, S. Young, et al. Streamingqa: benchmark for adaptation to new knowledge over time in question answering models. In International Conference on Machine Learning, 2022.
[46] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03177 2023.
[47] X. Liu, K. Ji, Fu, W. L. Tam, Z. Du, Z. Yang, and J. Tang. P-tuning v2: Prompt tuning can be comparable fine-tuning universally across scales and tasks. In Annual Conference of the Association for Computational Linguistics, 2022.
[48] S. Longpre, K. Perisetla, A. Chen, N. Ramesh, C. DuBois, and S. Singh. Entity-based knowledge conflicts in question answering. arXiv preprint arXiv:2109.05055 2021.
[49] Lorraine, Xie, Zeng, C.-H. Lin, Takikawa, N. Sharp. T.-Y. Lin, M.-Y. Liu, S. Fidler, and J. Lucas. Att3d: Amortized text-to-3d object synthesis. In IEEE International Conference on Computer Vision, 2023.
[50] M. McCloskey and N. J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. The Psychology of Learning and Motivation, 1989.
13

---

[51] N. Mishra, M. Rohaninejad. X. Chen, and P. Abbeel. A simple neural attentive meta-learner. In International Conference on Learning Representations, 2018.
[52] E. Mitchell, C. Lin, A. Bosselut, C. Finn, and C. D. Manning. Fast model editing at scale, In International Conference on Learning Representations, 2022.
[53] E. Mitchell, C. Lin, A. Bosselut, C. Finn, and C. D. Manning. Memory-based model editing at scale. In International Conference on Machine Learning, 2022.
[54] A. Modarressi, A. Köksal, A. Imani, M. Fayyaz, and H. Schütze. Memllm: Finetuning llms to an explicit read-write memory. arXiv preprint arXiv:2444.11672, 2024.
[55] OpenAl. Introducing chatgpt. https: // openai. com/ bl cttg//thhtpt.... 2022.
[56] S. Park and J. Bak. Memoria: Resolving fateful forgetting problem through human-inspired memory architecture. International Conference on Machine Learning, 2024.
[57] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville. Film: general conditioning layer. In AAAI Conference on Artificial Intelligence, 2018.
Visual reasoning with
[58] J. Phang, Y. Mao, P. He, and W. Chen. Hypertuning: Toward adapting large language models without back-propagation. In International Conference on Machine Learning. 2023.
[59] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training, In preprint, 2018
[60] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 2020.
[61] S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimizations toward training trillion parameter models. In International Conference for High Performance Computing, Networking, Storage and Analysis, 2020.
[62] Rajpurkar, Zhang, K. Lopyrev, and P. Liang. Squad: 100,000+ questions for machine comprehension of text. In Conference Empirical Methods in Natural Language Processing, 2016.
[63] M. Rei. Online representation learning in recurrent neural language models. In Conference on Empirical Methods in Natural Language Processing, 2015.
[64] M. Ren, W. Zeng, B. Yang, and R. Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning, 2018
[65] J. Requeima, J. Gordon, J. Bronskill, S. Nowozin, and R. E. Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. Advances in Neural Information Processing Systems, 2019.
[66] S. Robertson, H. Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends@ in Information Retrieval, 2009.
[67] R. Sandhaus. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 2008.
[68] V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019.
[69] A. Santoro. S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-learning with memory-augmented neural networks. In International Conference on Machine Learning, 2016.
[70] P. Sarthi, S. Abdullah, A. Tuli, S. Khanna, A. Goldie, and C. D. Manning, Raptor: Recursive abstractive processing for tree-organized retrieval. In International Conference on Learning Representations, 2024.
14

---

[71] J. Schwarz, W. Czamnekki, J. Luketina, A.
Y. W. Teh, R. Pascanu, and R. Hadsell. Progress & compress: A scalable framework for continual learning. In International Conference Machine Learning. 2018.
[72] J. R. Schwarz and Y. W. Teh. Meta-learning sparse compression networks. Transactions on Machine Learning Research, 2022.
[73] R. Schwarz, J. Tack, Y. W. Teh, J. Lee, and J. Shin. Modality-agnosk variational compression of implicit neural representations. arXiv preprint arXiv:2301.094 2023.
[74] R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. In Annual Conference of the Association for Computational Linguistics, 2015.
[75] C. Si, Z. Gan, Z. Yang, S. Wang, Wang, J. reliable. In International Conference on Learning Representations, 20233.
and L. Wang. Prompting gpt-3 to be
[76] W. Song, S. Oh, S. Mo, J. Kim, S. Yun, J.-W. Ha, and J. Shin. Hierarchical context merging: Better long context understanding for pre-trained LLMs. In International Conference on Learning Representations, 2024.
[77] S. Thrun and T. M. Mitchell. Lifelong robot learning. Robotics and Autonomous Systems, 1995.
[78] M. K. Titsias. J. Schwarz. A. G. d. G. Matthews, R. Pascanu, and Y. W. Teh. Functional regularisation for continual learning with gaussian processes. In International Conference on Learning Representations, 2020.
[79] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, Bhargava, S. Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09228,, 2023.
[80] H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal. Interleaving retrieval with chain- of of-thought reasoning for coowledge-intessiie multi-step questions. In Annual Conference of the Association for Computational Linguistics, 2023.
[81] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t.. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
[82] J. Wang, A. Jatowt, and M. Yoshikawa. Archivalqa: A large-scale benchmark dataset for open-domain question answering over historical news collections. International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022.
[83] W. Wang, L. Dong, H. Cheng, X. Liu, X. Yan, J. Gao, and F. Wei. Augmenting language models with long-term memory. In Advances in Neural Information Processing Systems, 2023.
[84] Y. Wang, X. Chen, J. Shang, and McAuley. Memoryllm: Towards self self-updatable large language models. In International Conference on Machine Learning. 2024.
[85] D. Wingate, M. Shoeybi, and T. Sorensen. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. In Conference on Empirical Methods in Natural Language Processing, 2022.
[86] M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Interna- tional Conference on Machine Learning, 2022.
[87] Y. Wu, M. N. Rabe, D. Hutchins, and C. Szegedy. Memorizing transformers. In International Conference Leurning Representations, 2022.
[88] F. Xu, W. Shi, and E. Choi. Recomp: Improving retrieval-augmeette Ims with compression and selective augmentation. arXiv preprint arXiv:2310.044088 2023.
[89] Xu, J.-F. Ton, H. Kim, A. R. Kosiorek, and Y. W. Teh. Metafun: Meta-learning with iterative functional updates. In International Conference Machine Learning, 2020.
15

---

[90] D. Xuan
Xuan-Quy, Quy, L. Ngoc- Bich. P. Xuan-Dung N. Bac-Bien, and V. The-Duy. Evaluation of chatgpt and microsoft bing chat performances on physics exams vietnamese national high school graduation examination. arXiv preprini arXiv:2306005338 2023.
[91] M. Yin, G. Tucker, M. Zhou, S. Levine, and C. Finn. Meta-learning without memorization. In International Conference on Learning Representations, 2020.
[92] D. Yogatama, C. Wang, B. R. Routledge, N. A. Smith, and E. P. Xing. Dynamic language models for streaming text. Transactions of the Association for Computational Linguistics, 2014.
[93] Zadouri, A. Üstün, A. Ahmadian, B. Ermiş, Locatelli, and Hooker. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning, arXiv preprint arXiv:2309.05444, 2023.
[94] W. Zhong, I.. Guo, Q Gao, H. Ye, and Y. Wang. Memorybank: Enhancing large language models with long-term memory. In AAAI Conference on Artificial Intelligence, 2024.
16

---

A Experimental Details
A.1 Experimental details
Training details. We mainly follow the training configuration suggested by [26]. For all datasets, we train 50 epochs by using Adam [38] optimizer, where we warm up the learning rate for the first epoch (except for training DistilGPT2: 68) and then use a constant value throughout the training. Here, we use a learning rate of le - 5 for all models except for DistilGPT2 where it uses le 4. The output token number of the amortized network T is 12 for DistilGPT2 and 24 for the rest. We apply backpropagation dropout for large models with more than 1 billion parameters, using a ratio of p 0.75. Additionally, we use 4bit quantization [13] and ZcRO [61] when training GPT2-XL [59], and 2 [79] where we also (4-bit) quantize the T5 encoder [60]. It is important to note that the quantization should be applied to pre-trained networks, not the networks learned from the random initialization (e.g., amortization and aggregation network). We use a batch size of 64 for DistilGPT2 and 32 for the rest by using the gradient accumulation.
Evaluation details. We follow the same evaluation protocol from [26]. For the online adaptation, adapt the model on stream of 1,665 documents and then perform QA evaluation. For the online finetuning baselines, we follow Hu et al. [26] to find the best learning rate hyperparameter, where we observed that the performance is somewhat quite sensitive the choice. We mainly used 6.5e- 6 6 for all online finetuning methods except for CaMeLS, which uses 2.5e 5 in most cases. For the catastrophic forgetting analysis in Figure 3, we fixed the learning rate to 6.5e online finetuning methods as we found that forgetting occurs more on larger learning rates, It is worth remarking that MAC does not require any additional hyperparameter during online fine-tuning.
for all
Base LM details. We mainly consider GPT2 family [59] as the static base LM Bbase by following the prior work [26], where we additionally conduct the experiment on LLaMA-2 [79] to verify the scalability of MAC. For the amortization network, we consider the T5 model family [60] that are relatively smaller than the base LM. It is important to note that the output number of tokens T of the amortization and aggregation networks is a parammeerr,,eer. where we use 24 for all architectures except for Distil-GPT2 which uses 12. Then, we map these T tokens into each layer's modulation through a linear layer where we use P-tuming v2 [47] as the modulation design.
Amortization network details. For the model details, mainly describe the design choice of our amortization anort- Note that input encoder Iappt uses the same architectural design as f while using a smaller sized network. For the amortization network, we follow the design choice from [58] and use the T5 encoder-decodee model [60] as the base architecture. Specifically, we learn trainable tokens that are used for decoder input so that the output number of tokens T is consistent. Then, we have an individual two-layered MLP for each output token. For the network size, we use T5-small as the amortization O amert network for Distil-GPT2, T5-base for GPT2-Large, and L5-Large for both GPT2-XL and L (7B) where the input network 0 Innput uses a smaller model (T5-small for Distil-GPT2 and T5-base for the rest).
Aggregation network details. The aggregation network uses four cross-attention blocks, each consisting of one cross-attention layer and one feed-forward network. Here, the set of parameter efficient finetuning (PEFT) modulations (in the memory bank) is the key and value of each cross- attention layer, and the encoded question (go. (x); soft prompt tokens) is the initial query of the cross attention layer (i.e., later layers use the previous block's 's output as the query input). Thereby, the output of the aggregation network is soft prompts that have the same dimension the encoded question.
Dataset details. Here, we describe the dataset detail in the following.
StreamingQA [45] The StreamingQA is composed of questions that are either created by annotators produced using large-scale language model. These questions can be answered using a dynamic knowledge database of English WMT news articles, which have been timestamped and were published from 2007 to 2020, and these articles are also included in the dataset. Following the setups in [26], we use 21k training questions, 1.7k validation questions, and Sk test questions, respectively. Also, the same number of documents with the questions is used for each split, during the experiments. For the baselines that require QA pre-training (see Section 4), we use 40k training questions and 4k validation questions, respectively.
17

---

SQuAD [62]: The Stanford Question Answering Dataset (SQuAD) is composed of questions created by crowdworkers based on collection of Wikipedia articles, where the answer to each question is a span contained in the corresponding article. Following the setups in [26]. we use 39.9k training questions, 5.6k validation questions, and 10.6k test questions, respectively. Next, we use 8.6k training documents, 1.2k validation documents, and 2. 1k test documents, respectively. For the baselines that require QA pre-training (see Section 4), we use 40k training questions and 2.1k validation questions, respectively.
ArchivalQA [82]: The ArchivalQA dataset is constructed with synthetically generated questions from the sophisticatedly designed pipelines with language models. Specifically, questions are generated from articles in the New York Times Annotated Corpus [67]. Also, the answer to each question is span contained in an article. Following the setups in [26], we use 21.7k training questions, 5.3k validation questions, and 8.7k test questions, respectively. Next, we use 12.8k training documents, 3.0k validation documents, and 5.0k test documents, respectively. For the baselines that require QA pre-training (see Section 4), we use 12.4k training questions and 3k validation questions, respectively.
A.2 Memory complexity of hierarchical modulation aggregation
The calculated memory complexity is based on the Attention map size, which equal to the dimension after multiplying the Query and Key of the Cross-Attention layer. Here, the Query dimension is fixed T tokens, and the Key dimension dependent on the size of the memory bank. In this regard, K documents are encoded into KT tokens, thus showing (KT?) for the entire set aggregation. For hierarchical aggregation, we subgroup KT tokens into M tokens for each memory, thus reducing the complexity into (MT)). Here, it is important to note that we do not assume parallelization for the hierarchical aggregation when computing each subgroup, hence, the memory complexity is OMM)).
B Algorithm
B.1 Algorithm of MAC
Algorithm 1 Meta-training of MAC
Algorithm 2 Online learning of MAC
Input:
Oanort, Onnput. Opaser t/, Ctrain, learning Input: Stream of document Ctest, test QA set
rate 3
{x, y: Vi)/- aaortt Iapuut. Orase, 10
1: while not converge do
1: Initialize new memory bank M: 0
2: Sample documents {di,... dx} from
Extract amortized contexts from the stream of documents
Ctrain
3:
Sample QA pairs (x), ~ p(x, v/da).
3: for k = to Kteat do
4: for to K do
-Smmarizz context
5:
Summarize context
5:
Φn - - (d₂) Save da into M
6:
Øk 90 (dl)
7:
end for
7: end for
8:
Aggregate modul ati oos Te (90 (xx), {sh k=
Adapt the LM based on the input and evaluate
9:
9: for = to do
10:
Compute loss
10:
Aggregate modul atnions
11:
Crotel = [ [C (xki of), vw),
11:
o hes (g0.npuc (xx), {di) Kiim y
12:
Optimize
12: y = L.M (x;07
13:
Oanort f- Oxnort- BV Pesees Crotal
13: end for
14:
Oinpuc x Oinpur - V Leotal
Output: Accuracy (f(y,, prod
15:
e 1b VV Protal
16: end while
Output:
Oanort. Osuput, +b
18

---

B.2 Algorithm of the hierarchical modulation aggregation
Algorithm 3 Hierarchical modulation aggregation
Input: M, 4/, X, Inputt subgroup cardinality M
1: while [M] 1 do
2: Subgroup M into M tokens {M1. .... , M
3: Initialize new memory bank Maeu == 0
4: for iori=11 M
[MM] do
5: Aggregate subgroup o; - ho (g0lawe Mi)
6: Store Φ; into Maex
7: end for
8: Repeat by M t- Maev
9: end while
Output: .M {0²}
C More Discussion with Related Work
Prompt compression. The amortization meta-learning scheme of MAC can also related to prompt compression methods [85, 12]. The major goal of prompt compression techniques is to reduce the context length while preserving the prediction performance. While seemingly similar to our amortization-bian based meta- meta-learning approach (as it compresses the document into few tokens), amortization network learns to extract the new knowledge that is useful to adapt the base LM's old knowledge. Namely, their goals different. Nevertheless, believe exploring the architectures suggested in other prompt compression schemes to improve our amortization network will be an interesting future direction explore.
D
More Experimental Results
D.1 Effect of train time quantization for aggregation network
Table 8: Effect of train time quantization on aggregation network. Here, we train MAC on LLaMA2 under 4bit quantization and 16bit mixed predicsion, respectively. We report exatch match (EM) and Fl score evaluation metric.
StreamingQA
SQuAD
ArchivalQA
EM
F1
EM
F1
EM
4bit quantize (nf4)
14.29
21.79
15.07
21.14
20.12
23.90
16bit (bfloat16)
19.26
27.20
16.08
22.34
21.50
26.25
We found that the main reason for the smaller improvement in larger models is due to the strong quantization applied during training, not because of our method itself. Specifically, when training large models (c.g., LLaMA4), we used 4-bit quantization for efficiency. We observed that removing this quantization (using only mixed precision training) significantly improved model performance. For example, the Fl score of Llama2 on ArchivalQA increased from 23.909 to 26.25% (as shown in the table below). This is because training with additional modules learned from scratch (e.g.. aggregation network) requires careful quantization. It is worth noting that we have only removed 4-bit quantization for training, not for the adaptation stage, thereby maintaining a fair comparison with the baseline.
19

---

D.2 Comparison with memory augmented LMs
Table 9: Comparison with memory augmented LM by compressing the context using a recent method (i.c., CCM), then learning to retrieve the relevant compressed document using retriever. Here, we train LLaMA2 (unquantized) on StreamingQA dataset. The bold indicates the best result.
EM
CCM + T5 encoder Retriever MAC
17.98
25.98
19.26
27.20
We also have conducted comparison by combining the context compression method CCM [37] and RAG to show the effectiveness of MAC. Here, we first train the CCM to compress the context, then train encoder-only model (i.e., T5 encoder) that retrieves the correct compressed contexts. For a fair comparison, we have frozen the base LLM parameter to retain the knowledge learned from the past and did not apply quantization during training. As shown in Table 9, MAC shows better performance compared to CCM combined with RAGS.
D.3 Data contamination check for evaluation datasets
Table 10: Dataset contamination check on StreamingQA dataset by comparing document adapted performance with zero-shot and few-shot in-context learning (ICL).
Model
Zero-shot
5-shot ICL
Ours
GPT2-XL
7.12
10.78
15.38
LLaMA2
12.59
13.98
21.79
We measured the base LLM's zero-shot and 5-shot in-context learning (ICL) Fl accuracies the StreamingQA dataset to verify whether the model has already learned the test set knowledge. As shown Table 10, the base LLM struggles to answer the evaluation set without adaptation the test set documents, indicating the low possibility of test set leakage.
20

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable "[No] ", it is perfectly acceptable answer "[No] provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" "we were unable find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased binary way. we acknowledge that the answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendíx. If you answer [Yes] a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims in the introduction and abstract accurately reflect the contribution and scope, which are then verified in the experiment section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
21

---

Justification: It is discussed in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We do not have theory in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have included the implementation of MAC in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
22

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide the implementation in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. /tipss.ccppublll. wttpc//cppcccggguceeeeeoo
sesisssssssosoiii iislcnnc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
23

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification: We provide the detail of the netwarkkdras//trannn//raaaaaaiia setup in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
All experiments are conducted with the same and commonly used random
seed.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the memory usage in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics ttttps://neure wwtppplic/tthiicccccoooooc.ccepe www.iiiiiieeeeeieeeeeeeceeeeeeee delines?
Answer: [Yes]
Justification:
We do not have any ethical concerns regarding the paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics. If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the societal impact in Appendix 5
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our method does not introduce risks for misuse.
25

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited all papers and datasets used in the paper (See Appendix A)
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, haperswithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have included our anonymous PyTorch implementation of MAC in the supplementary file.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
26

---

Answer: [NA]
Justification: We use existing benchmark datasets and do not have any crowdsourcing datasets experiments in the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not have human subject in the research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "YfQRTdefjy", "reviewer": "Reviewer_kPwd", "summary": "This paper presents a novel online adaptation framework (Memory of Amortised Contexts, MAC), which effectively solves the problem of rapid updating of large language models (LLMs). MAC successfully preserves the knowledge learned by the model during the original training phase and the new data streams through memory augmentation and efficient fine-tuning of parameters. Experimental results show that MAC outperforms existing online fine-tuning methods regarding online adaptive performance (Table 1 and Figure 3), time, and memory efficiency (Figures 2, 4, and 5). MAC can be combined with popular methods such as Retrieval-Augmented Generation (RAG) to improve performance further (Table 2).", "questions": "1. The current experiments are mainly validation on small models, but according to the author's description llama was also trained for 50 epochs, which seems to be problematic?\n2. The current fine-tuning is based on P-Tuning. Has there been any consideration of comparing more parameter-efficient fine-tuning methods, such as Lora fine-tuning?\n3. Is it possible to add scenario experiments for online learning related to different tasks?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. In a time of rapidly updating information, MAC provides a practical approach to model updating that helps to keep language models current. With the proposed memory-efficient technique and forward propagation optimization, MAC significantly reduces the memory footprint and time consumption during online adaptation.\n2. MAC effectively avoids the problem of catastrophic forgetting by constructing memory banks and ensures that the model integrates and utilizes old and new knowledge.\n3. This paper validated the effectiveness of MAC through experiments with multiple datasets and different models, and the results are convincing.", "weaknesses": "1. Memory bank growth issues. As online adaptation proceeds, the size of the memory bank is likely to grow, which may pose a challenge for memory management. It is recommended that the authors explore more effective memory bank reduction techniques in future work.\n2. The ability to generalize adaptations from different domains remains unknown. Often, online learning may not be the same type of task (e.g., knowledge answering and coding), and it remains unclear whether the current method can cope with such scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer kPwd,\n\nWe sincerely appreciate your efforts and insightful comments to improve the manuscript.\\\nWe respond to each of your comments one-by-one in what follows.\n\n---\n\n**[W1] Memory bank growth issues.**\n\nIt is true that one possible limitation of MAC can be the growing size of the memory bank during adaptation.\n\nHowever, we would like to remark that we have already considered such a scenario in the main paper by constraining the memory bank size (in Figure 6 and also in the table below). Specifically, we reduce the number of amortized contexts when it reaches the memory constraint of 1,250 (where the total number of contexts is 1665). Here, we consider three simple yet effective schemes: i) random pruning, ii) randomly averaging two modulations, and iii) averaging two nearest neighbor (NN) modulations based on the cosine distance, where averaging nearest neighbor modulations shows quite effective preservation. \n\n\\begin{array} {lcccc}\\newline \\hline & \\text{Random pruning} & \\text{Random averaging} & \\text{NN averaging} & \\text{Full memory} \\newline \\hline \\text{F1} & 19.80 & 20.75 & 21.00 & 21.79 \\newline \\hline\\newline \\end{array}\n\n---\n\n**[W2] Ability to adapt to different tasks (e.g., coding) and domains.**\n\nFirst, we clarify that the major reason we mainly focused on the QA task is that it is a conventional evaluation protocol for online learning LMs [1,2,3], as evaluating the updated (or preserved) knowledge is non-trivial for other tasks. In this regard, we followed the same experimental setup in [1], where the authors considered QA only for the evaluation. \n\nNevertheless, we would like to emphasize the strong adaptation ability of MAC to other domains in QA (in Table 4/also in the table below), thus showing the possibility of adapting to different tasks when the training corpus increases. Here, we show that MAC trained on the StreamingQA dataset can be used for online adaptation of different QA datasets. As shown in the table below, MAC outperforms CaMeLS in F1 score. It is worth noting that the meta-learning performance scales as the training distribution is more diverse [4], hence, we believe training MAC on larger datasets and tasks will further improve the generalization.\n\n\\begin{array} {l cc} \\hline \\text{StreamQA}\\to & \\text{SQuAD} & \\text{ArchivalQA} \\newline \\hline \\text{CaMeLS} & 8.63 & 13.43 \\newline \\text{MAC} & \\textbf{10.47} & \\textbf{13.73} \\newline \\hline \\end{array}\n\n---\n\n**[Q1] Mainly validated on small models.**\n\nFirst, we want to clarify that we followed the setup from the prior paper [1], where we actually conducted experiments on a larger model compared to the previous work (i.e., Llama2 7B). This was possible because MAC is more efficient in terms of both time and memory compared to the baseline [1]. For instance, the baseline could not train on Llama2 7B within the given memory constraint of 80GB with a batch size of 1, even with 4-bit quantization.\n\n---\n\n**[Q2] Other types of PEFT methods.**\n\nThank you for bringing this up. During our initial development, we also considered LoRA as an alternative. However, we found that P-tuning v2 outperformed LoRA when training GPT2-XL on the StreamingQA dataset. This aligns with findings from previous work [5], which also observed that P-tuning outperforms LoRA when using amortization. Additionally, P-tuning allows for efficient batch computation, enabling a single forward pass of the LLM with different modulations. In contrast, LoRA requires separate forward passes for each modulation, which increases the training time. For these reasons, we chose to use P-tuning throughout our paper.\n\n\\begin{array} {l cc} \n\\hline \n\\text{PEFT type} & \\text{EM} & \\text{F1} \\newline \n\\hline \n\\text{LoRA} & 8.67 & 15.15 \\newline \n\\text{P-tuning v2} & \\textbf{8.99} & \\textbf{15.38} \\newline \n\\hline \n\\end{array}\n\n---\n\n**[Q3] Other scenarios than QA.**\n\nWe carefully remark that we already considered an additional scenario (i.e., the language modeling) rather than QA in Table 5, where we outperformed other online finetuning baselines. For your convenience, we have presented the table below. Specifically, we adapt the LLM on a stream of documents, then give the initial 10% of the document as input and measure the perplexity of the remaining documents (the initial document is equivalent to a question in the QA task). Here, we measure the perplexity of the predicted text on two document sets: i) the documents used for LLM adaptation to measure knowledge preservation and ii) unseen documents to measure the generalization, where MAC has outperformed the baselines in both cases.\n\n\\begin{array} {lcc}\\newline \\hline &\\text{Adapted documents} &\\text{Unseen documents} \\newline \\hline \\text{Uniform} & 11.43 & 13.89 \\newline \\text{SSM} & 27.87 & 29.69 \\newline \\text{CaMeLS} & 11.31 & 14.77 \\newline \\text{MAC (Ours)} & \\textbf{10.91} & \\textbf{12.71} \\newline \\hline\\newline \\end{array}\n\n---\n\n**Reference**\\\n[1] Meta-Learning Online Adaptation of Language Models, EMNLP 2023\\\n[2] Towards Continual Knowledge Learning of Language Models, ICLR 2022\\\n[3] TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models, EMNLP 2022\\\n[4] Meta-learning without memorization, ICLR 2020\\\n[5] HyperTuning: Toward Adapting Large Language Models without Back-propagation, ICLR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uAubtvxSTc", "reviewer": "Reviewer_j5pf", "summary": "The paper proposes an online learning framework called MAC (Memory of Amortized Contexts) designed to efficiently adapt large language models (LLMs) online. By using feature extraction and memory augmentation methods, MAC compresses and stores new document information in a memory bank, retrieving relevant knowledge when answering questions. This method utilizes amortization-based meta-learning, achieving efficient modulation learning through a single forward pass, avoiding the need for gradient updates during testing. Experiments demonstrate that MAC outperforms existing methods in online adaptation performance, time, and memory efficiency, and can be combined with popular alternatives like Retrieval-Augmented Generation (RAG) to further enhance performance.", "questions": "1. How is the storage overhead of MAC controlled when facing large-scale data streams? Are there any further optimization measures?\n2. How does the method perform on other task types (e.g., text classification, generation tasks)? Have any related experimental evaluations been conducted?\n3. How were the architectures of the amortization and aggregation networks determined? Have other architectures been tried, and what were the outcomes?\n4. Is there a concern of outdated or redundant modulation parameters in the memory bank? How are these issues addressed to maintain model efficiency?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a novel memory-augmented online adaptation framework based on amortization, which can efficiently adapt to new information without requiring gradient updates.\n\n2. Experimental results show that MAC performs exceptionally well across multiple datasets and architectures, significantly improving online adaptation performance, time, and memory efficiency.\n\n3. MAC can be combined with existing methods like RAG, enhancing the quality of retrieved documents, demonstrating good scalability and compatibility.\n\n4. The use of two memory-efficient techniques during training and inference stages reduces memory requirements, ensuring the method's scalability.", "weaknesses": "1. The method involves several complex steps and model components, such as the amortization network and aggregation network, which may increase implementation difficulty.\n\n2. Although the method has been experimentally evaluated on multiple datasets, the tasks are primarily focused on question-answering. Its adaptability to other task types remains to be verified.\n\n3. The paper mainly demonstrates the method's effectiveness through experimental results but lacks in-depth theoretical analysis of certain key design choices, particularly in amortization and aggregation strategies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer j5pf,\n\nWe sincerely appreciate your efforts and insightful comments to improve the manuscript. We respond to each of your comments one-by-one in what follows.\n\n---\n\n\n\n**[W1] Possible implementation difficulty due to a somewhat complex method.**\n\nWe respectfully argue that MAC is a simple framework requiring the implementation of only two networks: the amortization and aggregation networks, both of which only need a simple modification of existing code. Moreover, we have provided the PyTorch implementation of MAC in the supplementary material, and we plan to open-source the code if the paper is accepted.\n\n---\n\n**[W2&Q2] Focused on QA: other tasks need to be verified.**\n\nWe carefully remark that we already considered an additional scenario (i.e., the language modeling) rather than QA in Table 5, where we outperformed other online finetuning baselines. For your convenience, we have presented the table below. Specifically, we adapt the LLM on a stream of documents, then give the initial 10% of the document as input and measure the perplexity of the remaining documents (the initial document is equivalent to a question in the QA task). Here, we measure the perplexity of the predicted text on two document sets: i) the documents used for LLM adaptation to measure knowledge preservation and ii) unseen documents to measure the generalization, where MAC has outperformed the baselines in both cases.\n\n\\begin{array} {lcc}\\newline \\hline &\\text{Adapted documents} &\\text{Unseen documents} \\newline \\hline \\text{Uniform} & 11.43 & 13.89 \\newline \\text{SSM} & 27.87 & 29.69 \\newline \\text{CaMeLS} & 11.31 & 14.77 \\newline \\text{MAC (Ours)} & \\textbf{10.91} & \\textbf{12.71} \\newline \\hline\\newline \\end{array}\n\nFurthermore, we clarify that the major reason we mainly focused on the QA task is that it is a conventional evaluation protocol for online learning LMs [1,2,3], as evaluating the updated (or preserved) knowledge is non-trivial for other tasks. In this regard, we followed the same experimental setup in [1], where the authors considered QA only for the evaluation. \n\n---\n\n**[W3&Q3] Lack of in-depth theoretical analysis, e.g., network design choice.**\n\nWhile our design choices were primarily guided by empirical analysis, we conducted an in-depth evaluation to determine the best architecture for our amortization and aggregation networks during development.\n\n\nFor the amortization network, we explored three types of architectures: decoder-only, encoder-only, and encoder-decoder language models. Specifically, we considered (i) the GPT2 model and (ii) the T5 encoder with learnable tokens, where the input context is compressed into these tokens. The table below shows that the encoder-decoder model outperformed the other two alternatives, based on results from using GPT2-XL as the base LLM on the StreamingQA dataset. It is worth mentioning that our architecture follows the design carefully outlined in [4].\n\n\\begin{array} {l cc} \n\\hline \n\\text{Amortization} & \\text{EM} & \\text{F1} \\newline \n\\hline \n\\text{Encoder only (T5-encoder)} & 8.53 & 15.01 \\newline \n\\text{Decoder only (GPT2)} & 8.01 & 14.87 \\newline \n\\text{Encoder-Decoder (T5)} & \\textbf{8.99} & \\textbf{15.38} \\newline \n\\hline \n\\end{array}\n\nFor the aggregation network, we initially considered combining the amortization network (context compressed into PEFT modulation) with RAGs. However, we found that the aggregation approach provided better performance. Specifically, we trained an encoder-only model (T5-base encoder) to measure the similarity between PEFT modulations and the question for retrieving the modulation, subsequently adapting the base model. We report the results using GPT-XL as the base LLM on StreamingQA. Since the aggregation network attends to multiple documents and then predicts PEFT modulation, it is more likely to outperform RAG, which has a possibility of retrieving a wrong modulation.\n\n\\begin{array} {l cc} \n\\hline \n\\text{Aggregation} & \\text{EM} & \\text{F1} \\newline \n\\hline \n\\text{MAC (retrieve)} & 7.98 & 14.51 \\newline \n\\text{MAC (aggregate)} & \\textbf{8.99} & \\textbf{15.38} \\newline \n\\hline \n\\end{array}\n\n---\n\n**[Q1&Q4] How to handle storage overhead and redundancy of the memory bank.**\n\nFirst, we carefully remark that we have considered the storage overhead scenario in the main paper (in Figure 6, which is also reported in the table below for your convenience). Here, we consider a scenario with a memory bank size constraint by reducing the number of amortized contexts when it reaches 1,250 (where the total number of contexts is 1665). Here, we consider three simple yet effective schemes: i) random pruning, ii) randomly averaging two modulations, and iii) averaging two nearest neighbor (NN) modulations based on the cosine distance, where averaging nearest neighbor modulations shows quite effective preservation. This result indicates that redundant amortized context can be merged to improve storage efficiency while effectively maintaining the performance. \n\n\\begin{array} {lcccc}\\newline \n\\hline \n& \\text{Random pruning} & \\text{Random averaging} & \\text{NN averaging} & \\text{Full memory} \\newline \n\\hline \n\\text{F1} & 19.80 & 20.75 & 21.00 & 21.79 \\newline \n\\hline \\newline \n\\end{array}\n\n---\n\n**Reference**\\\n[1] Meta-Learning Online Adaptation of Language Models, EMNLP 2023\\\n[2] Towards Continual Knowledge Learning of Language Models, ICLR 2022\\\n[3] Meta-learning without memorization, ICLR 2020\\\n[4] HyperTuning: Toward Adapting Large Language Models without Back-propagation, ICLR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k4DHKkNVqE", "reviewer": "Reviewer_QtU4", "summary": "This paper proposes Memory of Amortized Contexts (MAC) which can encode the documents into compact modulations stored in a memory bank, which can later be retrieved to answer questions.", "questions": "See the weaknesses part", "limitations": "yes the authors have addressed the limitations", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is efficient compared to the baselines. \n2. The paper is well-written and easy to follow.", "weaknesses": "1. **Unfair Comparison**: This paper only compares with [1], which may be somewhat unfair. In the paper [1], they propose to distill the information into a parameter vector $\\phi$. However, in the current paper, the proposed method MAC stores all the modulations constructed from each context, which naturally contains all the knowledge from the contexts, leading to better knowledge retention. The setting might be unfair. \n2. **Missing Baselines**: What I think could be the fair comparisons might be comparing MAC with most recent retrieval methods, such as DPR[2], BM25, RAPTOR[3], IRCoT[4] etc. Although the paper proposes to combine MAC and BM25, there is no direct comparison between them as MAC is also essentially doing retrieval. Having two retrievers is better than only having one retriever may not show the effectiveness of MAC. \n3. **Doubt on the Performances**: Even MAC shows improvements over [1] (in the paper's setting), the best performance of MAC on Llama2-7B, ArchivalQa-Seq is 20.12 (EM), while BM25 can easily achieve 52.81 (EM) as shown in Table 2. It seems that MAC performs much worse than BM25. Can I expect the more advanced RAG methods can easily beat MAC even equipped with BM25? Can I really expect there is going to be improvements over other strong RAG methods when equipped with MAC? These are unanswered questions. \n4. **Missing Related Works**: The paper talked about Retrieval Augmentation for LMs and Memory augmented LMs. However, for RAG, [2][3][4] are not mentioned in related work (there are more RAG method out there); for Memory augmented LMs, some methods such as MemoryLLM [5], Memoria [6], MemLLM [8], MemoryBank[7], Camelot[8] are not mentioned which can also perform online learning. Maybe the memory-based methods should also be used for comparison in the experiments. \n\n\n[1] Meta-Learning Online Adaptation of Language Models.   \n[2] Dense Passage Retrieval for Open-Domain Question Answering.   \n[3] RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval.   \n[4] Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions.   \n[5] MemoryLLM: Towards Self-Updatable Large Language Models.   \n[6] Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture.   \n[7] MemoryBank: Enhancing Large Language Models with Long-Term Memory.   \n[8] MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memory.   \n[9] CAMELoT: Towards Large Language Models with Training-Free Consolidated Associative Memory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer QtU4,\n\nWe sincerely appreciate your efforts and insightful comments to improve the manuscript.\\\nWe respond to each of your comments one-by-one in what follows.\n\n---\n\n**[W1] Unfair comparison: Online learning distils information into parameter vector where MAC stores the modulation.** \n\nWe would like to clarify that distilling (or compressing) the updated information into PEFT parameters rather than the full parameter space is the key novelty of our framework, which prevents the model from forgetting the learned knowledge. While we agree with the reviewer's point that MAC increases the overall parameter count by storing such parameters in the memory bank, we emphasize that MAC outperforms larger models of baselines with a smaller-sized model, thus showing parameter efficiency. For instance, we achieved a 13.31% F1 score with GPT-2 Large (774 M model parameters + 26 M memory bank parameters = 800 M in total), whereas CaMeLS reached 11.67% with GPT-2 XL (1.5B model parameters). Moreover, as illustrated in Figure 6, we have proposed an effective method to constrain the size of the memory bank (i.e., averaging similar modulations to reduce the memory), preventing the network from increasing its parameters during adaptation. In this regard, we believe the comparison is fair, as we suggested an alternative of online finetuning based on the same training/evaluation setup.\n\n---\n\n**[W2&W3] More comparison with recent RAGs/Can joint usage of MAC and recent RAGs consistently improve the performance?** \n\nFollowing your suggestion, we considered two advanced and commonly used RAG methods: DPR [1] and Contriever [2]. In our experiments, we found that BM25 remains a strong baseline, demonstrating performance comparable to RAGs in our setup. This is consistent with other literature highlighting BM25 as an effective baseline [3].\n\nMore importantly, we observed that the combined use of MAC and advanced RAGs consistently yields improvements, suggesting that the benefits from RAG and MAC are orthogonal. While RAGs are effective at capturing details from retrieved documents, they heavily rely on retrieval accuracy, which can be problematic if the wrong documents are retrieved. In contrast, MAC can attend to multiple documents simultaneously using an aggregation network, allowing the LLM to capture shared information across documents. Therefore, we believe MAC and RAG complement each other well to improve the performance.\n\n\\begin{array}{lcccccc}\n\\hline\n& \\text{Top-1} & & \\text{Top-3} & & \\text{Top-5} & \\newline\n\\hline\n& \\text{EM} & \\text{F1} & \\text{EM} & \\text{F1} & \\text{EM} & \\text{F1} \\newline\n\\hline\n\\text{BM25} & 48.53 & 54.17 & 56.18 & 63.74 & 64.74 & 71.83 \\newline\n\\text{BM25+MAC} & \\textbf{52.81} & \\textbf{56.55} & \\textbf{60.22} & \\textbf{66.82} & \\textbf{68.85} & \\textbf{74.89} \\newline\n\\hline\n\\text{Contriever} & 44.78 & 51.55 & 52.56 & 61.28 & 60.10 & 67.83 \\newline\n\\text{Contriever + MAC} & \\textbf{47.99} & \\textbf{53.23} & \\textbf{53.92} & \\textbf{63.75} & \\textbf{61.28} & \\textbf{70.01} \\newline\n\\hline\n\\text{DPR} & 48.98 & 55.01 & 57.02 & 64.27 & 65.07 & 72.24 \\newline\n\\text{DPR + MAC} & \\textbf{49.57} & \\textbf{55.98} & \\textbf{60.19} & \\textbf{67.05} & \\textbf{68.52} & \\textbf{75.00} \\newline\n\\hline\n\\end{array}\nLastly, we would like to highlight the inference efficiency of our method. While RAG requires appending retrieved documents to the context, which increases the inference cost, MAC adapts the model with PEFT modulation, thus maintaining the base LLM's inference cost. Note that we only use a prefix size of 2 for each layer as the PEFT modulation, thus enabling efficient inference. As shown in Figure 1, combining MAC with RAG minimally increases memory utilization while significantly enhancing performance.\n\n---\n\n**[W4] More related works.**\n\nWe thank the reviewer for the suggestion. In the revised paper, we will include all the references pointed out by the reviewer and discuss their relevance and differences. For instance, while MAC can be categorized as a memory-augmented system, it is particularly specialized in online learning. Our core idea is to avoid updating the parameters of the base LLM to preserve the knowledge obtained from extensive pre-training while effectively updating the knowledge through the memory. In contrast, other memory-augmented networks require architectural modifications to incorporate memory, which can lead to the potential loss of pre-trained knowledge. Therefore, our approach maintains the integrity of the pre-trained model while enabling efficient online learning.\n\n---\n\n**Reference**\\\n[1] Dense Passage Retrieval for Open-Domain Question Answering, EMNLP 2020\\\n[2] Unsupervised Dense Information Retrieval with Contrastive Learning, TMLR 2022\\\n[3] Improving Passage Retrieval with Zero-Shot Question Generation, ACL 2022\\\n[4] Compressed Context Memory For Online Language Model Interaction, ICLR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kd5gu51EKN", "reviewer": "Reviewer_pwuf", "summary": "This paper focuses on how to adapt static language models (LMs) with streaming documents during inference time. \n\nThere are two high-level challenges here: 1) how to store new domain/task relevant information, 2) how to utilize the stored information for downstream task-solving, i.e., doing question answering (QA).\n\nIn this paper, the authors propose, MAC, a parameter-efficient adaptor approach to 1) encode new documents into a sizable vector memory bank, and 2) utilize those encoded knowledge via extra attention mechanism.\nThe proposed method is compared with other fine-tuning baselines on three QA tasks tailored for evaluating online adaption scenarios, showing that the method is more effective.\nAdditional analyses and ablative studies are also provided to drive more insights into the model designs and behaviors.", "questions": "Are those methods reported in Table 1 sensitive to the input order? For example, SQuAD questions do not have any temporal dependency on documents, it is good to see the performance on different streams, e.g., recency bias (putting the relevant documents in the beginning and irrelevant in the end).\n\nWhat is the setup for decoding? Beam search or greedy? Using sampling techniques? If so, what is the temperature?\n\nWhat is the training cost? Vs baselines?\n\nWhat are the inference prompts? \n\nHow data efficient is the proposed method? Is it possible to achieve similar performance with less training?", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper is well-written with enough background and details for readers to follow.\n\n2) The authors apply their proposed method to different LMs with various architectures and training protocols, which provide support for the generalization ability of the approach.\n\n3) The design of experiments are mostly reasonable (baselines are OK, multiple datasets are good), and the results suggest the proposed method is effective. There are additional analyses and ablative studies, e.g., i) the proposed method can do better knowledge retention and help RAG, ii) most design choices are validated.", "weaknesses": "1) The proposed method is a natural extension of memory-augmented LLM (as cited in paper [76,79]) with a token compression module. Rather than comparing with only closed-book QA models, it is good to compare with context compression methods, e..g, text token compression w/ RAG or long-context models, e.g.,\n\n[1]J.-H. Kim, J. Yeom, S. Yun, and H. O. Song. Compressed context memory for online language\n\n[2]H. Jiang, Q. Wu, X. Luo, D. Li, C.-Y. Lin, Y. Yang, and L. Qiu. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression\n\n[3]T. Ge, J. Hu, L. Wang, X. Wang, S.-Q. Chen, and F. Wei. In-context autoencoder for context compression in a large language model\nmodel interaction, 2024\n\n2) The experiment settings can be problematic. \n\nAs the goal is to adapt the model to new knowledge, it is not clear whether those evaluated datasets manifest that. Both news articles and wikipedia pages are highly used in pretraining LMs, e.g., LLaMA-2.\n\nIt is necessary to report zero-shot and few-shot results with the base model. Without that, it is hard to judge the benefit of online adaption. As all reported models have very low EM or F1 scores, it is good to report the base model for sanity check.\n\nIt is noticeable that the proposed method has less improvement on more capable models (larger sizes, e.g., LLaMA-2). It is good to dig a little bit into this, e.g., experiment with similar sized models ***with or without instruction tuning*** (LLAMA and Vicuna/Alpaca). Specifically, the instruction tuning might be relevant for the model to memorize certain information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer pwuf,\n\nWe sincerely appreciate your efforts and insightful comments to improve the manuscript.\\\nWe respond to each of your comments one-by-one in what follows.\n\n---\n\n**[W1] Comparison with memory-augmented networks by combining context compression [1] with RAGs.**\n\nThank you for the suggestion. We want to clarify that the context compression method and amortization-based meta-learning approach have different goals. The major goal of context compression techniques is to reduce the context length while preserving the prediction performance. While seemingly similar to our amortization-based meta-learning approach (as it compresses the document into a few tokens), our amortization network learns to extract the new knowledge that is useful for adapting the base LM’s old knowledge.\nNevertheless, following your suggestion, we have conducted a comparison by combining the context compression method CCM [1] and RAGs. Here, we first train the CCM to compress the context, then train an encoder-only model (i.e., t5 encoder) that retrieves the correct compressed contexts. For a fair comparison, we have frozen the base LLM parameter to retain the knowledge learned from the past and did not apply quantization during training. As shown in the table below, MAC shows better performance compared to CCM combined with RAGs. \n\n\\begin{array} {l cc} \n\\hline \n\\text{Llama2} & \\text{EM} & \\text{F1} \\newline \n\\hline \n\\text{CCM} & 17.98 & 25.98 \\newline \n\\text{MAC} & \\textbf{19.26} & \\textbf{27.20} \\newline \n\\hline \n\\end{array}\n\\* Both methods did not apply quantization during training; therefore, the reported score of MAC is higher than the paper’s result (see more details in [W2-2]).\n\n---\n\n**[W2-1] Is the adaptation dataset already seen by the language model? Performance of zero/few-shot.**\n\nWe first clarify that we used the same setup from the previous paper [2], in which the authors carefully selected the datasets that are not trained by LMs. \n\nNevertheless, we also agree with the reviewer’s concern and measured the base LLM's zero-shot and 5-shot F1 accuracies on the StreamingQA dataset to verify whether the model has already learned the test set knowledge. As shown in the table below, the base LLM struggles to answer the evaluation set without adaptation to the test set documents, indicating the low possibility of test set leakage.\n\n\\begin{array} {l cc} \n\\hline \n& \\text{Zero-shot} & \\text{5-shot} & \\text{Ours} \\newline \n\\hline \n\\text{GPT2-XL} & 7.12 & 10.78 & 15.38 \\newline \n\\text{Llama2} & 12.59 & 13.98 & 21.79 \\newline \n\\hline \n\\end{array}\n\n---\n\n**[W2-2] Less improvement on larger models.**\n\nThank you for pointing this out. We found that the main reason for the smaller improvement in larger models is due to the strong quantization applied during training, not because of our method itself. Specifically, when training large models (e.g., Llama2), we used 4-bit quantization for efficiency. We observed that removing this quantization (using only mixed precision training) significantly improved model performance. For example, the F1 score of Llama2 on ArchivalQA increased from 23.90% to 26.25% (as shown in the table below). This is because training with additional modules learned from scratch (e.g., aggregation network) requires careful quantization. It is worth noting that we have only removed 4-bit quantization for training, not for the adaptation stage, thereby maintaining a fair comparison with the baseline.\n\n\\begin{array} {l cccccc} \n\\hline \n& \\text{StreamingQA} & & \\text{SQuAD} & & \\text{ArchivalQA} & \\newline \n\\hline \n& \\text{EM} & \\text{F1} & \\text{EM} & \\text{F1} & \\text{EM} & \\text{F1} \\newline \n\\hline \n\\text{4bit quantize (nf4)} & 14.29 & 21.79 & 15.07 & 21.14 & 20.12 & 23.90 \\newline \n\\text{16bit (bfloat16)} & 19.26 & 27.20 & 16.08 & 22.34 & 21.50 & 26.25 \\newline \n\\hline \n\\end{array}\n\n\n---\n\n**[Q1] Sensitivity to the input order.**\n\nWe clarify that the input order does not change the output, as our aggregation network (i.e., cross-attention network) is permutation-invariant.\n\n---\n\n**[Q2] Decoding setup.**\n\nWe have followed the same decoding setup from CaMeLS [2], where we use beam search with 12 beans and do not perform sampling. \n\n---\n\n**[Q3] Training cost compared to baselines.**\n\nThank you for pointing this out. MAC is highly efficient compared to the major baseline CaMeLS. For instance, MAC is 32 times efficient in terms of the memory usage on the same sized model (CaMeLS requires a 80GB GPU memory to train DistilGPT (82M) with a batch size of 1 while MAC can train a batch size of 32), thus showing more than 20 time faster training speed. \n\n---\n\n**[Q4] Inference prompts.**\n\nWe do not have a specific inferent prompt or prompt template. We give the raw question to the base model (e.g., GPT2) and the raw context to the amortization network.\n\n---\n\n**[Q5] Data efficiency of the proposed method.**\n\nThank you for the interesting question. We have trained MAC using 21,000 document question pairs for StreamingQA (in the paper), where we reduced the document by 20% and 50% to measure the data efficiency. Here, we found that MAC is somewhat data-efficient: removing 20% of the data still shows a good performance, achieving 21.01% of F1 score in StreamingQA, where the original F1 score is 21.79% where removing more than 50% of the dataset can drop the performance to 19.75%. We believe diverse and complex document sets indeed help to train the aggregation network to better aggregate the optimal modulation.\n\n---\n\n**Reference**\\\n[1] Compressed Context Memory For Online Language Model Interaction, ICLR 2024\\\n[2] Meta-Learning Online Adaptation of Language Models, EMNLP 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers and AC,

We sincerely appreciate your valuable time and effort spent reviewing our manuscript. 

As reviewers highlighted, we believe our paper provides a novel (kPwd,j5pf), efficitent (all reviewers) yet effective (kPwd, pwuf) framework for online adaptation of LLMs followed by a clear presentation (all reviewers). 

We appreciate your constructive comments on our manuscript. In the attached pdf, we have run the following additional experiment to clarify the reviewer's comment:
- Memory efficiency and performance curve of RAG and MAC (combined with RAG), in Figure 1
- Comparison between RAG and MAC (combined with RAG), in Table 1

We strongly believe that MAC can be a useful addition to the NeurIPS community, in particular, due to the enhanced manuscript by reviewers’ comments helping us better deliver the effectiveness of our method.

Thank you very much!\
Authors

Meta-review: This paper introduces a novel framework for efficient online adaptation of LLMs. The method uses feature extraction and memory augmentation to compress and store information from new documents in a memory bank, which can be retrieved when answering questions. The method employs amortization-based meta-learning, allowing for efficient modulation learning through a single forward pass without requiring gradient updates during testing.

The reviewers generally agree that the proposed method presents a novel and efficient approach to online adaptation of LLMs. The method's ability to outperform baselines while maintaining memory efficiency is particularly noteworthy. The authors have addressed many of the reviewers' concerns in their rebuttal, such as clarifying the fairness of comparisons, providing additional experimental results, and explaining the impact of quantization on larger models.
Decision: Accept (poster)