Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass
Authors: Tong Chen
Hao Fang
Patrick Xia
Xiaodong Liu
Benjamin Van Durme
Luke Zettlemoyer
Jianfeng Gao
Hao Cheng
Venue: ICLR 2025 Poster (2025)
Keywords: language model; efficient adaptation; fine-tuning; prompting
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models (LLMs) acquire substantial knowledge during pretraining but often need adaptation to new contexts, tasks, or domains, typically achieved through fine-tuning or prompting. However, fine-tuning incurs significant training costs, while prompting increases inference overhead. Inspired by fast weight memory, we introduce GenerativeAdapter, an effective and efficient adaptation method that encode test-time context into language model parameters with a single forward pass.
GenerativeAdapter augments a frozen pretrained LM with a lightweight adapter generator, trained via self-supervised learning, to produce parameter-efficient adapters.
Notably, our generator is general-purpose, i.e., one generator can adapt the corresponding base model for all langauge processing scenarios.
We apply GenerativeAdapter to two pretrained LMs (Mistral-7B-Instruct and Llama2-7B-Chat) and evaluate the adapted models across  knowledge acquisition from documents, learning from demonstrations, and personalization for users.
In StreamingQA, our approach is effective in injecting knowledge into the LM's parameters, achieving a 63.5\% improvement in F1 score over the model with supervised fine-tuning (from $19.5$ to $31.5$) for contexts as long as 32K tokens.
In the MetaICL in-context learning evaluation, our method achieves an average accuracy of $44.9$ across 26 tasks, outperforming the base model. 
On MSC, our method proves to be highly competitive in memorizing user information from conversations with a 4x reduction in computation and memory costs compared to 
prompting with full conversation history.
Overall, GenerativeAdapter provides a viable solution for adapting large LMs to evolving information and providing tailored user experience, while reducing training and inference costs relative to traditional fine-tuning and prompting techniques.

Published as a conference paper at ICLR 2025
CONTEXTUALIZING LAN- GUAGE MODELS IN PARAMETERS WITH A SINGLE FORWARD PASS
GENERATIVE ADAPTER:
Tong Chen Hao Fang " Patrick Xia 0 Xiaodong Liu Benjamin Van Durme Luke Zettlemoyer* Jianfeng Gao Hao Cheng** University of Washington < Microsoft
Microsoft Research
ABSTRACT
Large language models (LMs) are typically adapted to improve performance on new contexts (e.g., text prompts that define new tasks domains) through fine- tuning or prompting. However, there an accuracy compute tradeoff fine- tuning incurs significant training cost and prompting increases inference over- head. We introduce Generative Adapter, an effective and efficient adaptation method that directly maps new contexts to low-rank LM adapters, thereby sig- nificantly reducing inference overhead with no need for finetuning. The adapter generator is trained via self self-supervised learning, and can be used to adapt a single frozen LM for any new task simply by mapping the associated task or domain context to a new adapter. We apply Generative Adapter to two pretrained LMs (Mistral-7B 7B-Instruct and Llama2-7B 7B-Cht) and evaluate the adapted models in three adaption scenarios: knowledge acquisition from documents, learning from demonstrations, and personalization for users. In StreamingQA, our approach is effective in injecting knowledge into the LM's parameters, achieving a 63.5% improvement in Fl score over the model with supervised fine-tuning (from 19.5 31.5) for contexts as long 32K tokens. In the MetalCL in-context learn- ing evaluation, our method achieves an average accuracy of 44.9 across 26 tasks, outperforming the base model. On MSC, our method proves to be highly com- petitive in memorizing user information from conversations with 4x reduction in computation and memory costs compared to prompting with full conversation history. Together, these results suggest that Generative Adapter should allow for general adaption to a wide range of diffcrent contexts. The code is available at Chttps://gitan com
INTRODUCTION
Adaptation is essential for language models (LMs) to acquire new world knowledge (Jiag et al.,
2024| Hu et al., 2023; Mecklenburg et al. 2024), learn new tasks (Min al.] (022). and person- alize to individual users (Salemi et al. 2024). Existing adaptation methods typically involve either
prompting or fine-tuning (Brown et al. 2020). As the scale of LMs continues to increase, adapting
them becomes increasingly difficult due to efficiency constraints during both training and infer- ence (Hu et al. 2022).
Prompting with task-specific demonstrations (i.e., in-context learning (Brown et al. 2020)) or back- ground knowledge (i.e,, retrieval-augie -aggeeee- generation (Lewis et al.) 2020)) is one way to enable models to temporarily encode such relevant information, allowing flexible adaptation tasks. However, to maintain additional memory across sessions, some extra prompts must be added to the input, which incur an inference-time or storage overhead (Chevalier et al. 2023). Fine- tuning is another way to embed new information into the LM's parameters, retaining long-term memory. Nevertheless, it requires a training phase that is more computationally expensive than single forward pass, and acquiring knowledge through continual pretraining has shown to be data- inefficient (Yang et al., 2024 Allen-Zhu & Li 2024). Thus, we are interested in exploring alternative approaches for effectively and efficienlly adapting pretrained LMs.
various
WWrk done during internship at Microsoft Research.
1 "Correppondence (chentong wwshington.cddee chehao@ miccooomicrosoot.commm

---

Published as a conference paper at ICLR 2025
In this work, present Generative Adapter, a novel method for training neural network (adapter generator) to generate adapters that contextualize pretrained LMs on-the-fly fly with temporary knowl- edge from incoming contexts. Inspired by fast weights (Ba et al., 2016; Schmidhuber, [992] inter alia), our approach incorporates a lightweight adapter generator on top of pretrained LM as the slow network to produce updated parameters for the fast network (the adapted L.M.. As far we know, we are the first to explore this direction. Specifically, the pretrained base LM remains frozen while train the LM-specific adapter generator to generate layer-by-layer additive updates, similar to recent parameter- efficceent fine-tuning (PEFT) techniques (Houlsby et al. 2019 Hu et al. (022). For each layer, an adapter generator network uses the outer product of past context hidden states from the corresponding base LM layer to generate delta weights. These generated delta weights are then added to the base LM weights to form an adapted LM for future predictions. Similar to previous work on fast weights, our method achieves test-time adaptation using only forward passes, allowing dynamic updates as new context arrives in sequential chunks. We train the generator end-to-end in self-supervised manner by compressing the context into a generated adapter and then computing the next-token prediction loss on a target sequence using the adapted LMM. Once trained, our method can produce adapted LMs that effectively capture knowledge from the context to solve multiple downstream tasks, thus improving the adaptability of off-the-shelf pretrained LMs.
We evaluate our method on three scenarios where on-the-fly contextualizing pretrained LMs is cru- cial: acquiring new factual knowledge, learning from demonstrations, and personalizing for in- dividual users. These scenarios involve diverse forms of context with varying lengths, including documents with background knowledge, task-specific input-output examples and user-specific con- versations. In the knowledge acquisition scenario, Generative Adapter effectively memorizes fac- tual knowledge from provided documents, with minimal information loss compared to full-context prompting at short context lengths. Notably, our method excels in memorizing long-contex: doc- uments, managing to handle context lengths up to 32K on StreamingQA (Liska et al.) 2022) and 8K on SQuAD (Rajpurkar et al., 2016) better than continous pretraining. In learning from demon- strations on MetalCL (Min et al. (2022). Generative Adapter follows demonstrations effectively, achieving superior accuracy compared to the in-context learning of its base model. This exempli- fies the model's s ability to adapt to new tasks efficiently. For personalization, Generative Adapter is highly effective in retaining user information from conversations, achieving a fourfold reduction in computation and memory costs compared to full conversation prompting. In practical scenarios with many queries from the same user on edge computing devices, the benefits of our method are even more evident. This positions Generative Adapter as a highly efficient tool for personalized LMs.
Our contributions are summarized as follows:
1. We introduce Generative Adapter, a novel method for efficiently adapting pretrained LMs on-the-fly using test-time contexts. To our knowledge, we are the first to explore retaining the relevant temporary knowledge through generated parameter-efficient model updates for state-of-the of-the-art pretrained LMs.
2. We develop an adapter generator network on top of frozen pretrained LMs to transform text contexts into updated model parameters (adapted LMs) for future queries. We also design an efficient end-to-end training process to enhance the LMs' adaptability, i.e., the resulting generator augmented LM can be used for various downstream tasks using only forward passes.
3. We validate the proposed method on two representative pretrained LMs. Empirically, we show the effectiveness of Generative Adapter in various adaptation scenarios, including knowledge acquisition from documents, learning from demonstrations, and personalized user interactions. Our method proves to be generalizable across different types of contexts and applicable to multiple downstream tasks.
METHOD
We present Generative Adapter, efficient and effective framework for directly generating additive weight updates to contextualize the pretrained LM (a frozen base LM) at test time. Unlike continual pretraining and supervised fine-tuning which update the pretrained LM via gradient descent, our method achieves adaptation using forward passes only. In the following sections, we first provide a task formulation and an overview of Generative Adapter ($2.)) Then describe the the adapter
2

---

Published as a conference paper at ICLR 2025
Test-Time Contextualization
Inference
1
y
response
+ base
1
C1
Base
Adapter Gonerator
Adapter
A
H
+
base
4
base
H
G
C
Base LM
Adapter Generator
Adapter
x
prompt
During test-time contextualization, the adapters A....., Ar are generated sequentially for the stream context chunks Cj...., . C.. At given time step f, the context chunk C₂ is encoded by the base LM Obsee into hidden state vectors H, Then the generator 9 produces a new adapter D, based on the collection of hidden state vectors HI...., H2 representing the accumulated context. Right: During inference, combine the latest adapter A with the base LM base to generate responses for input prompts.
Figure 1: Overview of Generative Adapter. Left:
generator 2.2). followed by the self-supervised pretraining tasks ($2.3) and the normalization tech- niques for improving training stability ($ 2.4).
2.1 ADAPTATION WITH TEST-TIME CONTEXTUALIZATION
To contextualize base model, basee given context C. our goal to obtain updated model, Oc, that can respond to user instructions using the information provided the context C. In prac- tice, the context include different types of data, such documents, dialogues, or task description and few-shot demonstrations.
We specifically focus on lest-time contextualization, where context arrives incrementally stream of data, such as continuous flow of documents or dialogue sessions. We represent this streaming context up time step & as (t) (CI..... Ct), where C, is the context chunk arriving time step t. In this online adaptation scenario, the model must be efficiently adapted to each new context chunk as becomes available.
As shown in Figure I we propose Generative Adapter as a framework that adapts the base model Ohaso to new contexts through single forward pass as each context chunk arrives. Specifically, given test-time context E()), we adapt the base model Opase to a new model θ using context- dependent additive adapter A,, i.e., +z()) = b + Ar- More details regarding the adapter A, will be provided in 22.2 After this adaptation, the modified model θz()) can be utilized for any test input relevant to the context E(t) during inference. For example, if the context Σ(t) consists of user's past conversations, the modified model @z(1) can effectively summarize or answer questions about these conversations.
2.2 GENERATIVE ADAPTER
In this paper, propose using learned adapter generator directly produce the adapter A based on the streaming context Σ. The core idea is to use the adapter generator to project context token embeddings, encoded by the base language model (LM), into the matrices of each layer in the L.M Specifically, we consider only adapting the linear projection layers of the base Transformer model, i.e., the keylquery/value/outpu layers of the multi-head attention unit and the down/up projection layers of the feed-fo network.
Concretely, linear projection layer in the 1-th Transformer block (1 = 1, 1.2,.... L) can be written as Whh, where w() E do xdm is the weight matrix, h Rdi is the input vector, Redu

---

Published as a conference paper at ICLR 2025
is the output vector, and we omit the bias term for simplicity. For the adapted LM, we parameterize
W
base ( where w and w are the corresponding weight matrices in the base hase model base and the context-dependent adapter A, respectively.
w
To generate the weights of the context-dependent adapter A, we first encode the streaming context Σ using the base model 6 base and obtain the sequence of hidden states h,") h ..... ,,,,hh, h E RRdh (i.e. the outputs of the I-th Transformer block), where M the number of tokens in the context E, and de, the dimension of hidden states. These hidden states packed in matrix H E RM Then we use the hidden states from the (1 1)-th Transformer block to generate the adapter's weights Rd Xd.
w(O for the 1-th Transformer block, ie.. w = g (H(-))). where ()) denotes the layer-specific adapter generator which can transform any hidden state sequence of arbi- trary length into siedddimensiooal weight matrix. For conciseness, we will omit the superscript denoting the layer number / when it does not cause ambiguity.
To obtain generator (-)) without the undesirable dependency on the context length M, we use bi-linear function as following,
WA - H(B,B.)
(2 H(B.))
=
Σ hm & hm) (B,
where @ denotes the outer product operator, A E BRd A2 E Rd, B E Rdaxd, B2 E Id xd are all learnable parameters, and we set the dimension d, to much smaller than dma dout and dy, to keep the number of learnable parameters within an acceptable range.
Dynamic Streaming Update In practice, the context can arrive in chunks sequentially, The matrix of hidden states H2 at step is computed based all previous context chunks E(t 1). This hidden state is then used to generate an adapter for the current chunk E(t), which, in turn, is also used to compute the hidden states for future context steps. Based on [Equation I compute the adapter of Σ(t) we need to concatenate all hidden states (i.e., [H H.] M.) of the context chunks in [t) to generate the adapter, i.e., WA. = - H.)).
Fortunately, our formulation allows efficient updating mechanism without explicitly storing his- tory hidden states, noting that
WA = (AIA2) H. H (B,)( = (A1 Σ
H H, (B,
(2)
(=1
Thus, the update can be efficiently computed
S, + S A2H, H,B, WA. + 2
(3)
(4)
where H, E RR xda stores the hidden states for the t-th context chunk, and the partial sum S, E Rd. acts as the memory of history context chunks with So initialized all zeros. Note directly storing WA, E RRd xdin or Σ, H. H, E Rda xds would require much more memory because we control d, < min(dm. dout d...
Our preliminary experiments find that this architecture exhibits some empirical instability because the generated matrix Wa. can transform an input vector x into vector containing values with either extremely large or near-zero magnitudes, due to its skewed distribution of its singular values. In $2.4
we will explain how normalization can address the instability issue.
2.3 LEARNING TO UPDATE WITH SELF-SUPERVISED PRETRAINING
To preserve the language modeling capability of the adapted models Oxyee) for )1, pretrain the weight generator using the next-token prediction loss of O in a self-supervised manner web corpora. In other words, the adapter generator is trained top of the frozen base model Obase in an end-to-end fashion. Specifically, we use two self-supervision pretraining tasks: reconstruction and completion.
The reconstruction task (Ge et al. 2024) draws inspiration from autoencoders and aims to train the weight generator G to embed contextual information into the generated weights. This process
4

---

Published as a conference paper at ICLR 2025
compresses the imput context (I1, Tmm into generated adapter, (z11m) which is subsequently used to reconstruct the imput. Formally, this accomplished by maximizing the log-likelihood the input tokens with the adapted LM, using weights updated from the same text: Preconstruction (9)
log (Z1,...,, Zm base + (a))s)))
The completion task (Zhang al. 2024 Kim et al. 2024) trains the adapted LM gener- ate the continuation of the given context. The goal is to maximize the log-likelihood of to- kens 2m+1.... In. which represent the continuation of the context , in the dataset: Ccomplction (9) = log P (Im+lI..., O + (TTmm))
We observe using both of the task can make the generated adapter memorize and utilize the contex- tual information. Similar to prior work (Ge et al. 2024), the generator is trained to maximize the sum of the objective functions of the two task:
max Creconstruction (g) + Ceompletion (g)
(5)
g
2.4 NORMALIZATION FOR GENERATED WEIGHTS
In preliminary experiments, we find that using the naive outer product for generating weights led to instability during training, causing convergence issues. When multiplying the generated matrix with the input vector, the resulting output can either diminish to near-zero or grow excessively large.
To address this instability, we introduce normalization into the formulation, i.e.,
WA, < Ai norm(S.)B. A norm
A2
Σ HH H.) B
B2.
(6)
Our pilot experiments find that normalization based on singular value decomposition (SVD) is par- ticular effective, among other normalization strategies.
SVD Normalization The SVD normalization technique ensures the singular values of the outer product are normalized to 1. Given a matrix M, we define SVD normalization as:
norm (M) = UV
where M = T is the SVD factorization. This normalization resets the positive singular values of the matrix to one, preventing the vectors from excessively shrinking exploding.
Low-Rank SVD and LoRA An additional benefit of SVD normalization is that it can naturally produce low-rank matrices. Instead of performing full-rank decomposition,
approximate the
input matrix with a rank-r SVD decomposition, where r is a hyperparameter set in advance. Con- sequently, the matrix be written the product of two low-rank matrices, similar LoRA adapter (Hu et al. )022):
WA, A₁ norm B2
(8)
(H)) (H)B2),
where U(H) and (HH are the matrices resulting from SVD normalization. This low-rank approx- imation reduces both computational cost and memory usage.
EXPERIMENT SETTINGS
We experiment with using both Mistral-7B-Instreee (v0.2) (Jiang et al. 2023) and Llama2-7B-CHl Touvron et al. 2023) as the base LMs. For efficiency, our main experiments train adapter generators to only update the output projection layers of the multi-head attention unit in Transformer. We study a more capable implementation in 5 and defer the full exploration other modules for future work.
Hyperparameters The intermediate dimension d, and SVD rank spectively. Approximately, this leads to 500 million parameters for the generator, with the generated adapter of 32 million parameters.
1,024 and 128,
Training Following the standard training pipeline of LM development (Jiang et al. 2023 Touvron al. 2023), the training of adapter generator includes pretraining phase described in 2.3
5

---

Published as a conference paper at ICLR 2025
followed by instruction tuning. For pretraining, randomly sample 1 billion tokens from SlimPa- jama (Soboleva et al., 2023) which are split into segments of 8,192 tokens each. For instruction tuning, use a mix of tasks such question answering, in-context learning, and general instruc- tion following, which we ensure that there is no overlap with downstream tasks, with a detailed list provided in the appendix. The context is divided into chunks of 1,024 tokens utilize the dynamic updating mechanism described in 2.2
4
MAIN RESULTS
We evaluate Generative Adapter on three representative scenarios where contextualizing pretrained LMs is crucial, i.e., acquiring new factual knowledge (54.1). learning from demonstrations ($4.2).
and personalizing to individual users ($4 ($4.3)
4.1 DOCUMENT-BASED QUESTION ANSWERING WITH VARYING CONTEXT LENGTH
The factual knowledge stored in the parameters a LM remains static after pretraining. Here, we consider the scenario where the model needs to adapt to new knowledge based on documents. After adaptation, it is expected to correctly answer information-seekee seeking questions about these documents.
Setup and Baselines To evaluate the fact recall ability of the adapted model, use two question answering (QA) datasets, SQuAD (Rajpurkar et al.) 2016) and StreamingQA (Liska et al. 2022), where each test case consists of passage and corresponding question about some information from that passage. To analyze the impact of context length on performance, we conduct an evalua- tion using contexts of varying lengths.
We divide the documents in the corresponding test set evenly into groups, with each group having an average length of k tokens (k {512, 1K, 2K, 4K, 8K, 16K, 32K)). Thus, the model should contextualize on the article in each group and evaluate fact recall by the question associated with the articles. The QA accuracy is evaluated by comparing the generated output with the gold answer for all questions associated with the documents within the group. Following Rajpurkar et al. (2016), F1 score is used as the metric for evaluation.
We also analyze the computational and storage requirements of Generative Adapter, which com- prises three phases: seneral-purposs, pretraining, contextualization, and inference. The generator is pretrained once and can subsequently be used for any task. During the contextualization phase, Generative Adapter encodes the context into an adapter with a single forward pass. the inference phase, the adapted model generates responses based on the input. Beyond the LM parameters, the extra storage required includes the parameters of the generative adapter.
Here, we consider both full parameter fine-tuning and full context prompting using the same base model as baselines. For fine-tuning, we consider two variants. The first approach, supervised fine- tuning (SFT), trains the base model exclusively on training set of qnestion-answer pairs sourced from articles distinct from those in the test set. The second variant, known as continual pretraining (CPT), involves first training the base model all documents in the test set, followed by further adaptation through SFT using the the training set of question-answe answer pairs. During inference, we evaluate the fine-tuned model in a closed-book manner, i.e., the model is tasked with directly pro- ducing the answer to a given question. For prompting, we simply concatenate all documents in the group as single context and prompt the base model to respond accordingly. Specifically, for Mlmaa-7B-Chhne if the context length exceeds the maximum limit of 4K tokens, we truncate the prompt to include only the last 4K tokens. For Generative Adapter, we create an adapted model for each document group, which is similar to how the context is encoded as prompting. After that, the adapted model is asked to answer the question again in a closed-boue fashion, akin to fine-tuning.
We present the QA accuracy results for SQuAD and StrcamingQA and the computation costs for StreamingQA in Figure 2 and Figure 3] respectively. Both fine-tuning methods (SFT and CPT) are evaluated in a closed-book manner, resulting in constant QA performance regardless of varying context lengths. In contrast, both Generative Adapter and prompting are evaluated on vary- ing context lengths, where recalling facts can become more difficult as the context length increases.
Results
As expected, both our method and prompting achieve improved QA performance by using relevant contexts compared supervised fine-tuning baselines. Notably, Generative Adapter is highly effec-

---

Published as a conference paper at ICLR 2025
Clowed-book Prompking
Supervised Fineluring
Conkinual Pretraiiing
Incorteed Prompling
Genara Derwatiewhdapler
SQuAD
StrearmingQA
SQuAD
StreaminngQA
50
50
4.0
30
30
30
20
20
20-
10
10
512
1K
26
4K
BC
16K
32K
512
1K
BC
16K
32K
512
1K
BK 16K
32K
512
1K
2K
-KK
10K
32K
Context
Context
Length
(a) Mistral-7B-Instteee
(b) Llama2-7B-CCin
Figure 2: Document-basee QA performance of Generative Adapter across different context lengths. Both fine-tuning methods (supervised fine-tuning and continual pretraining) are evaluated in a closed-book manner and remain consistent Fl across context lengths. Generative Adapter achieves the same inference time as fine-tuning methods while demonstrating higher knowledge recall.
Continual Pretraining
Prompting wol Cache
Prompting KV Cache
Generative Adapter
Contextual ization Computation
Inference Computation
Extra Storage
20a
10a
10
Rloass (Milloo))
FLOP CD
FLOPS CD
200
10a
100
307
512
1K
8K Context Longth
2K
4K
16K
32K
512
1K
2K 4E 3K
16K,
32K
512
1K
28
4K
16K
32K
Context
Context Longth
Figure 3: Computation and storage requirements for Generative Adapter and baseline methods on StreamingQA. For Generative Adapter, the context is converted into adaptor during contextual- ization and then stored for inference. For the prompting method, the key-value (KV) cache can be generated during contextualization and reused during inference.
tive when the context is relatively short (< 1K tokens). Moreover, it avoids the additional inference overhead associated with prompting, which requires attention computation over the context input regardless of using key-value (KV) caches (illustrated by the green lines in Figure 3) This overhead issue worsens with longer contexts.
In most cases, Generative Adapter outperforms CPT, especially when the context length is less than 8K tokens. Importantly, although both approaches adapt model parameters using documents, Gener- ative Adapter requires preprocessing time (forward passes only) that is orders of magnitude smaller than CPT (which involves multiple forward and backward passes), as demonstrated in Figure
Zero-shot Prompting
Fine tine-tuning
In-context Prompting
Generative Adepter
Classification Tasks
Non-classification Tasks
Classification tion Tasks
Non-classification Tasks
0.150
0.6
0.125
0.10
Accurcy
Accurayy
0.08
0.4
0.075
0.04
0.025
0.02
0.0
0.000
0.0
Kishot
Kishot
Kishot
Kishot
(a) Mistral-7B-IIstteee
(b) Llama2-7B-Chat
Figure 4: Accuracy plots on MetaICL with varying K-shot in-context examples. Both fine-tuned and zero-shot prompting baselines are instructed complete the task without any in-context examples.
7

---

Published as a conference paper at ICLR 2025
4.2 IN-CONTEXT LEARNING WITH VARYING IN-CONTEXT EXAMPLES
In the prompting paradigm, one emerging ability of pretrained LMs is that they can perform a task with few task-specifie input-output examples as context on unseen cases, also known as in-context learning (Brown et al. 2020). Here, we are interested to see whether Generative Adapter can provide further benefits in enhancing the base LM's in-context learning ability.
Setup and Baselines We conduct experiments using MetalCL (Min et al. 2022), consisting of 26 test tasks. We also ensure that none of these test tasks were seen during the training of adapter generator. For each task, we use 1, 2, 4, 8, and 16 demonstrations randomly sampled from the corresponding development split following the MetalCL evaluate pipeline. To reduce evaluation variance, we repeat the sampling process five times for each few-shot setting. We report separate average accuracy for classification and non-classification tasks. For classification tasks, achieving high accuracy requires the model to learn both the candidate options and the input-output relation- ships from the provided examples. For non-classification tasks, the model also needs to learn the output style.
We consider three baselines: 1) zero-shot prompting with the base LM where only task instruction
is provided; 2) standard few-shot prompting with the base LM (Min al. 2022) where each test case is prepend with those few-shot examples; and 3) fine-tuning the base LM on each evaluation task using 16 input-output pairs, corresponding the maximum number of shots in our evaluation.
Results Figure 4 summarizes the results for MetalCI, with various number of in-context examples for both classification and non-classification tasks. The performance of fine-tuned models (blue lines) and zero-shot prompt baselines (grey lines) is evaluated without demonstrations, resulting in constant performance across different numbers of shots. While fine-tuned models generally achieve higher accuracy on classification tasks, their performance on non-classification tasks is lower. We speculate that the few-shot setting (16 shots) is insufficient for the model learn the desired out- put style through fine-tuning. In contrast, Generative Adapter outperforms few-shot prompting in most cases, with more significant improvements observed in the more challenging non-classification tasks, where the model must adapt to specific output styles. This indicates that the generated adapter not only retains the in-context learning ability but also enhances the base model.
4.3 PERSONALIZATION
Using LMs to analyze users' behaviours and memorize their preferences is the key to unlocking a tailored and engaging user experience, i.e., personalized LMs. Towards this goal, we focus on evaluating the LM's ability to memorize user information in conversations.
Setup and Baselines We use the our experiments, following Packer al. (224). Each test case comprises multi-session human- human conversation between two participants, along with a question regarding information men- tioned within the conversation. The average length of the conversational context is 2.5K tokens, which makes it inefficient to prompt the model repeatedly with the entire conversation history for the same user. Similar document-based QA (64.1). we evaluate the model quality using the Fl score by comparing the generated answers to the ground truth. We also report computation and memory costs. Here, we use Mistral- Instruct as the base LM.
Conversation (MSC) dataset (Xu et al. 2022) for
As baselines, we include both closed-book and full-conversation prompting based on the base LM, where the former involves random guesses and the latter incurs higher computation and memory costs by storing the entire long conversation. We also include the state-of-the-- prompt compres- sion method, UltraGist (Zhang al.) 2024), which reduces the context into fewer token embeddings, thereby saving computation and memory costs.
As expected, the closed-book approach, which does not memorize any user information performs very poorly. In contrast, methods that uti- lize proper user conversations as context can accurately recall user information, achieving reasonable answer accuracy. Although using the entire conversation leads to better accuracy, full conversation prompting incurs significant computation and storage costs, i.e., 4x those of Generative Adapter. Such costs are highly undesirable for personalizing LMs for individual users, especially since most computations occur on edge devices without power GPUs. Comparing to UltraGist at the same level of storage cost (compressed into 512 tokens), Generative Adapter further reduces inference
Results The results on MSC are summarized in Table
8

---

Published as a conference paper at ICLR 2025
Table 1: Performance comparison on MSC. higher FI indicates better performance, and lower inference computation and extra storage costs are preferable. For Ultragist (Zhang et al. (024), fewer compressed tokens (noted in parentheses) correspond to lower computation and memory costs.
Inference Computation (TFLOPS)
Extra
Model
Storage (M fooats)
Closed-book
0.505
Full colversnettttoonn Prompting
66.0
2:059
128+
Uhragist (64 Tokens)
26.5
0,514
Ultragist (128 Tokens)
32.2
Ultragist (256 Tokens)
38.3
0.627
Ultragist (512 Tokens)
40.8
0.772
Ultragist (IK Tokens)
44.4
1.067
64
Ultragist (2K Tokens)
42.4
1.658
128
Generative Adapter
cost without performance drop. In real world scenarios with many queries from the same user, the benefits of our method are even more pronounced.
ANALYSIS
5.1 MODEL DESIGN OPTIONS
Here, we exam different Generative Adapter design choices. Specifically, we train adapter gener- ators for M under various configurations and assess their quality based on recon- struction and completion perplexities on the validation set. [Tble 2 summarizes the results. These metries have shown a strong correlated with model quality, e.g., the default setting (row 1) achieves an Fl score of 40.2 on MSC, while using the Frobenius norm (row 4) reduces the score to 27.1.
Mixing pretraining tasks improves generalization. Training only on one task degrades perfor- mance. Without the completion task, completion perplexity deteriorates, suggesting overfitting to memorization. This highlights its role as a regularizer, helping Generative Adapter distill contextual information into adapters for better future predictions.
SVD is a more effective normalization. We compare SVD-based normalization (default) to Frobe- nius norm. While computationally simpler, Frobenius norm exhibits inferior performance, likely due to excessive shrinkage in certain directions, reducing model expressiveness.
More updatable parameters improve performance. By default, insert adapters in the attention output projection layer. Switching to the feedforward down-projection layer (tripling the number of updated parameters) enhances both perplexities. Due computational constraints, leave further exploration to future work.
5.2 COMPATIBILITY WITH BASE LM AND RAG
First, we exam whether Generative Adapter preserves the capabilities of the base model by adding the adapter layers. We generate an adapter using the prompt "You are a helpful AI assistant" and evaluate it on MMLU (0-shot) (Hendrycks et al. 2021). The base model (Mistral-77--Instreee-e scores 0.574, and Generative Adapter achieves 0.576, indicating a negligible impact on accuracy.
Generative Adapter can also be seamlessly combined with RAG (Lewis et al. 2020). To illusrate this, we combine Generative Adapter with RAG by prepending the most relevant 100-token chunk (retrieved via BM255) to the query at inference. On StreamingQA with 1 K-token context, Genera- tive Adapter alone achieves an accuracy of 49.3, while Generative Adapter + RAG reaches 63.6 with only O.IK additional tokens. In comparison, full-context RAG requires 1K tokens to achieve 67.8 accuracy. This highlights the effectivenss of using Generative Adapter alongside RAG to enhance performance with minimal additional context. The full results shown in Table of Appendix.

---

Published as a conference paper at ICLR 2025
Table 2: The validation set perplexity of the pretrained model under different design choices.
Reconstruction Perplexity
Completion Perplexity
Factor
Setting
Default
1.75
7.40
Presraining
Reconstracion Only
1,25
34.34
Task
Completion Only
Normalization
Frobenius
7.72
7.32
Module
Feedforward
1.68
7.26
6 RELATED WORKS
Fast Weights: Our proposed method is closely related to the idea of "fast weights" (Hinton & Plaut 1987 Ba al., 2016 Schlag et al. 2021), which makes the model weights being adaptive to the model input. Context-dependen fast weight programmers (FWPs) introduced by [Schmidhuber [1992 1993 use a slow network with slow weights to reprogram the fast weights of the correspond- ing fast network. Schlaag al. (222T) point out that self-attention without softmax and other linear Transformer variants Saai et al. 2019 Katharopoulos et al. 2020 Choromanski et al., 2021 Peng
al. 202T can be viewed as FWPs. Clark et al. (2022) propose fast weight layers which are added on top of the Transformer model after the last attention layer for language modeling. Different from previous work mainly focusing on specific tasks, our goal is enhance frozen pretrained LMs with fast associative memory for general language processing. Instead of using slow network to pro- gram a separate fast model, our method can be viewed as self prograaaggn model, i.e., context encoded by the base LM is used to update the base LM itself. Our work is also related to hyper- networks (Ivison al., 2023 Vladymyrov introduce additional layers to improve multi-task learning and in-context learning. However, our method directly integrates generative fast weights into the Transformer architecture, and our method can recall user-provided facts for tasks such as question answering beyond in-context learning.
af., 2024 von Oswald et al.! 2020), which typically
Adapting LMs via Meta-Learning: Recent work explores adapting pre-trained LMs to an online stream of documents using meta-learning, Hu al. (2023) propose context-aware meta-learned loss scaling, which reweights token-level losses during online fine-tuning, addressing the inefficacy of naive fine-tuning for downstream QA. Tack et al. (2024) introduce meta-learned amortization network that predicts parameter-efficien fine-tuning modulations for individual context documents, which are then aggregated for QA. Unlike these approaches, which typically require nested training loop, our adapter generator augments pre-trained LMs and enables end-to-end training with self- supervised objectives.
Grramtter-Efficien Fine- Fine-Tuning (PEFT): Generative Adapter employs a low-rank adapter akin LoRA (Hu et al., 2022). which was originally designed for PEFT. Several derivatives of LoRA exist such as AdaLoRA (Zhang et al. (202)) and DoRA (Liu et al. (2024), along with various other PEFT strategies such as serial adapters (Houlsby et al. 2019) and prefix tuning (Li & Liang 2021). A thorough survey of PEFT methods is presented by Han et al. (2024). Most work focuses on task- specific fine-tuning scenarios. Instead, Generative Adapter is general LM and does not require a downstream dataset for adaptation.
CONCLUSION
In this work, introduce Generative Adapter, method for efficiently adapting pretrained LMs on-the-fly using test-time context through forward passes only. We design an adapter generator net- work on frozen pretrained LMs transform text contexts into updated model parameters. Trained end-to-end with the frozen LM using two self-supervised tasks on web corpora, Generative Adapter evaluated in three scenarios: acquiring new factual knowledge, learning from demonstrations, and personalizing to individual users. Our experiments show that Generative Adapter reduces infor- mation loss compared to continual pertaining in retaining factual knowledge from new documents. Additionally, the model effectively adapts to new task instructions when learning from demonstra- tions. Finally, Generative Adapter achieves comparable fact recall to efficient prompting methods while using less inference-time computation, making it well-suited for personalization. Future work could explore scaling the adapter generator, such as integrating adapters into additional layers, and investigating more selective update rules (Schlag al. 2021
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank the members of the Deep Learning Group at Microsoft Research and Semantic Machines for their invaluable discussions and support. We also appreciate the insightful feedback provided by members of Hannaneh Hajishirzi's and Luke Zettlemoyer's labs.
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction, 2024. URL Tttt:s://arxi..or///at.
143166
Jimmy Ba, Geoffrey E Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 29, Cur- ran Associates, Inc., 2016. URL
ottp.e/ederrreeerrrrrsoc...pppre pillsp paper/2016/Eilee oaa.a//222..44442227777772323323
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. Ms marco: human generated machine reading comprehension dataset, 2018. URL https: //ax:v/////ivva/aa org / abs / 1611.111. 09268
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Vosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess. Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle. M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL httpps://prrcceed..ngg hewp:rocceee.ddo.........o....e. neurips cc/ cc/paper.p.pp....pp fww.flll///feo pess/ppprr/222. File//65cccbrrr00511118sttt22e21 Pataaaaapppppppp...
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3829-3846, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1/200 emnlp-main.22 2322. URL https: //acl anthology org/2023 emn 1p --ma--------- i n. 232
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with per- formers. In International Conference Learning Representations, 2021. URL https:
//openreview net
Kevin Clark, Kelvin Guu, Ming- Wei Chang, Panupong Pasupat, Geoffrey Hinton, and Mohammad Norouzi. Meta-leaming fast weight language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pp. 9751 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v h/2022.cmmlllmain.noo.. https://aclantholegg org. 20tg./222.cco emm22emn222......mmmmmmmmmmmm
URL
main. 661
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 2368 2378,, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/NI1 1ANN19--246.. 1246. URL https:
7/acl aothlannhholooooooooo ogy. htt.hholooooyooooo11111111100 9-
11

---

Published as a conference paper at ICLR 2025
Tao Ge, Hu Jing, Lei Wang, Xun Wang. Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in large language model. In The Twelfth International Confer- ence Learning Representations, 2024. URL https://pponneeeww..eeee/eeoo (RuGGZEGGJE
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter--efficien fine-tniing for large models: A comprehensive survey, 2024. URL httpss//aar.....or/////////000 14608
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In The Twelfth International Conference on Leurning Reprexentations, 2021.
1dd=77B3mm 3Gm
Geoffrey E Hinton and David C Plaut. Using fast weights to deblur old memories, In Proceedings of the ninth annual conference of the Cognitive Science Society, pp. 177-186, 1987.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In Proceedings of the 36th International Conference Machine Learning. 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https: https://openzeev///
net forum?
id=nze>/keePES
Nathan Hu, Eric Mitchell, Christopher Manning, and Chelsea Finn. Meta-learning online adapta- tion of language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 4418 4432,, Singapore, December 2023. Association for Computational Linguistics. doi: 18.18653/v 1/2023. emnlp-main.268. URL https:::p//... www.ooooontoooo..oooooooooooooo. org/ emn 1p 1al1pppmmm-- i n. 268
Hamish Ivison, Akshita Bhagia, Yizhong Wang. Hannaneh Hajishirzi, and Matthew Peters. HINT:
Hypernetwork instruction tuning for efficient zero- and few-shot generalisation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 11272-1 1288, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.116533w1/2223.oooo aclloo. 631. URL https: httttsss::/aaaatolllooooo org 2023 acl-laalllnol 631
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William Sayed. Mistral 7b, 2023. URL https:
0682
Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Lin, Wen-tau Yih, and Srini Iyer. Instruction-tuned language models are better knowl- edge learners. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pp. 5421-5434, Bangkok, Thailand, August 2024. Association for Computational Lingutt 1/2653////222.4c1lloooo
tics. doi: acl- scl-iong.e. 296
URL hetps://aclanttain org.2/000oyor//222.....
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.). Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLL) pp. 2567-2577, Hong Kong, China. November 2019. Association for Com- putational Linguistics. doi: 10.18653/v 1/D19- 1259.
URL https: http//aaaaanttooo..oooooo
D19-1259
12

---

Published as a conference paper at ICLR 2025
Angelos Katharopoulos, Apoory Vyas, Nikolaos Pappas, and François Fleuret. Transformers RNNs: Fast autoregressive Transformers with linear attention. In Proceedings of the 37th Inter- national Conference on Machine Learning, 2020.
Jang-Hyun Kim, Junyoung Yeom, Sangdoo Yun, and Hyun Oh Song. Compressed context memory for online language model interaction. In The Twelfth International Conference on Learning
Representations, 2024. URL BtLhttts: http://:////pe.eeeeeeee net/ /tte//ffrumttt 2om
Tomáš Jonathan Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018. doi: 10. 1.00tac.
URL
Wojeiech Kryscinski, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. BOOKSUM: A collection of datasets for long-form narrative summarization. In Yoav Gold- berg. Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 6536-6558, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v 1 1/2022. htwt:///......nnsemmmm.... wwiniingss-ennl..4.5.. URL
wwwphhhwwww..ggggg..oooo wrg. findings emn 1p 488
Patrick Lewis. Ethan Perez, Aleksandra Piktus. Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocküäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensiive nip tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 9459-9474. Curran Associates, Inc., 2020. URL httppp:://prccceee.nggg... heur: hwtp::ss...erp.......ceoooeaeaee ccppppr. saappprrrflleeeeeeeeeeereeeeeees fwweefilleeeeeee htt1es//papp//2000 E1 /tt//669223220557380cc2222222233
Xiang Lisa Li and Perey Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li. and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 4582 4597, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 acl- long.333. URL httpps::/acaaaathlllogoo.oo/ org/ 2021 aal.aaallloonog 353
Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. RA-DIT: Retrieval-augmented dual instruction tuning. In The Twelfth International Confer- ence on Learning Representations, 2024. URL 220Tbutug9
net wtt.//wwwwwew/eente/ee forum?id i
Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien De Masson D' Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen Gilsenan- Mcmahon, Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. StreamingQA: A bench- mark for adaptation to new knowledge over time in question answering models. In Kamalika Chaudhuri, Stefanie Jegelka, Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference ceedings of Machine Learning Research, pp. 16604-13622. PMLR, 17-23 Jul 2022. URL http:/gg...........//p.
Machine Learning, volume 162 of Pro-
https: http:://rrrcceeie ngs ml prwsss/vssvv wttll////lllaaaaa.e. wwl.iskkkaaah....m
Shih-Y Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomppose low-rank adaptation. In Proceed- ings of the 41th International Conference on Machine Learning, 2024.
Nick Mecklenburg, Yiyou Lin, Xiaoxiao Li, Daniel Holstein, Leonardo Nunes, Sara Malvar, Bruno Silva, Ranveer Chandra, Vijay Aski, Pavan Kumar Reddy Yannam, Tolga Aktas, and Todd Hendry. Injecting new knowledge into large language models via supervised fine-tuning, 2024.
URL https:://axiiiii http://axxxi.vooaa/////aaa/////a http://////ab.//.////////// 00213
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannanch Hajishirzi. MetalCL: Learning to learn in context. In Marine Carpuat, CatherCa de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association
13

---

Published as a conference paper at ICLR 2025
for Computational Linguistics: Human Language Technologies, pp. 2791-2809, Seattle, United States, July 2022. Association for Computational Conmuuaaiionalllgguiiisssss Linguistics. doi: 10. (00118653/11/2222.mo. waa..naacllaaa main. 201. URLL/attpps//accaaathooooo
Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, lon Stoica, and Joseph E.
Gonzalez. Memgpt: Towards ilms operating systems, 2024. URL https:://axxx..ooott
abs/ 2310..1..8850
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature attention. In International Conference Learning Representations, 2021. URL
https://openneville net
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Perey Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su. Kevin Duh, and Xavier Carreras (eds.), Pro- ceedings of the 2016 Conference on Empirical Methody in Natural Language Processing, pp- 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-122 Utt.httss:://// http://claaattoooooo ogy. ogw.orgg//6611100////
Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: conversational question answering 2019. doi:
challenge. Transactions of the Association for Computational Linguistics, owgp:/ooooor//000o0/-0//0/01 1016
10.1162/tacl a 00266. URL
Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. Getting closer to AI complete question answering: A set of prerequisite real tasks. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY. USA, February 7-12, 2020, pp. 8722-8731. AAAI Press, 2020. URL https: /aaaaaaa............aa..//aa..aa ojs/ http//nddxxx///// php/ /AAA//AA///
article/view 6398
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. LaMP: When large language models meet personalization. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 7370-7392, Bangkok, Thailand, August 2024. Asso- ciation for Computational Linguistics. doi: 10.18653/v1120000 1865 URL https:
//aclanthology.ozqg aww.-///ooo/222411oooo.o.... 399
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear Transformers are secretly fast weight programmers. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Jürgen Schmidhuber. Learning to control fast- memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139) 1992.
Jürgen Schmidhuber. Reducing the ratio between learning complexity and number of time varying variables in fully recurrent nets. In Proceedings of International Conference on Artificial Neural Networks (ICANN), pp. 460-463, 1993.
Daria Soboleva, Faisal Al- Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama, 2023. URL https: h cwtpsr/eeeeeeeeeeeeeeeeee S1 imp a ama- 627B
//buggingfaace. co/ dtt.datttattaaeeeeeeeeeadatss
Jihoon Tack, Jaehyung Kim, Eric Mitchell, Jinwoo Shin, Yee Whye Teh, and Jonathan Richard Schwarz. Online adaptation of language models with memory of amortized contexts, 2024.
URL https: /tttps://arivv. org. hbtpg//b////400 /403. 04317]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anna Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra,
14

---

Published as a conference paper at ICLR 2025
Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. BRLLttpps /ttppp::/aarxaa /ttp///axvvvv///a///i///a...awaa www/oogg//..... 22t./23070.000 htt.or///bbb//330002888 09288
Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada. Louis-Philippe Morency, and Ruslan unified understanding for transformer's s attention
Salakhutdinov. Transformer dissection: via the lens of kernel. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UJCNLP), pp. 4344 4353, Hong Kong, China, November 2019. Association for Computational Linguistiss doi: 10.18653/v1/D19 14433... URL/ittps://aeaaaaooooooo www.:/oooooorr//1-11111-10
Max Vladymyrov, Andrey Zhmoginov, and Mark Sandler. Continual hypertransformer: A meta- learner for continual few-shot learning. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL hntps::/openreview d Sql2nkkk
Johannes von Oswald, Christian Henning, Benjamin F. Grewe, and João Sacramento. Continual learning with hypernetworks. In International Conference on Learning Representations, 2020.
URL httpp:://peenrevvew..n httpi/reeieeweeeeeeeeeee fontt/orrmmiddiii BJg.Nerky
Jing Xu, Arthur Szlam, and Jason Weston. Beyond goldfish memory: Long-term open-domain conversation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 5180-5197, Dublin, Ireland, May 2022, Association for Computational Lin- guistics. doi: 10. 10.186663/112222.2cllllogg acl-long.356
Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candès, and Tatsunori Hashimoto. Synthetic continued pretraining, 2024. URL https: http://aaxvv.......... abs 2409. 07431
Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Yc, and Zhicheng Dou. Compressing lengthy context with ultragist, 2024. URL. httts: wtt./oo/////o.ooooooo//.//oooe// 2405. 16635
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh In- ternational Conference on Learning Representations, 2023. URL httpt::/ pestps://opann /ttp::///peeeeeeee// het//forum:idd=11@22WRR)
15

---

Published as a conference paper at ICLR 2025
A DATASET DETAILS
Pretraining, We pretrain our models using a randomly sampled subset of IB tokens from the SlimPajama corpus. For validation, we sample an additional 100 segments, each containing 2K tokens, from the same corpus.
Instruction Tuning. We perform instruction tuning using a combination of question answering, in-context learning, and instruction following datasets, following prior studies (Lin al. 2024 Ge et al. 2024 Zhang et al. 2024).
B TRAINING SETUP
Implementation We empirically found that normalization is crucial for Generative Adapter to function effectively. For SVD normalization, we implemented it using torch. svd lowrank (), setting the number of iterations to 1.
Generative Adapter is able to generate the adaptors for prefixes of chunks simultaneously by process- ing the context chunks in parallel. The computation proceeds by processing the hidden states of each Transformer block for context chunks layer by layer. Given the hidden states of (1), (t)
from the (7 (1-1) 1)-th Transformer block, denoted by (r(11)) we first compute the accumulated outer product S using Equation 2 We then normalize this outer product to obtain the additive matrix w(() A. using Equation 6 and finally get the output of the I-th Transformer block H! by the base model.
Pretraining. For Mistral-7B-Instruuc (hereafter referred to as Mistral), we use learning rate of 5 x 10-3, and for Llama2-7B-Chat (Llama2), we use 1 10-4, We apply weight decay of 0.01 and no dropout. The adapter added to the base model are scaled by 1/16 for Mistral and 1/8 for Llama2. We employ a WarmupDecayLE learning rate scheduler with a 100-step warmup and use the Adam optimizer. The global batch size is set to 8. Pretraining the adapter generator on 1B tokens takes approximately 20 hours using 8 NVIDIA H100 GPUs.
Instruction Tuning. For instruction tuning, largely follow the same configurations in pre- training, with some adjustments. We set the learning rate to 5 x 10-5 for both Mistral and Llama2 models. We train the models for epochs and use a batch size of 32.
C EXPERIMENT SETUP
Document-based QA. We set up experiments for document-based question answering (QA) using both supervised fine-tuning and continuous pretraining. For supervised fine-tuning answer pairs, we train on the training split of each dataset, evaluate on a validation set, and employ early stopping when the validation loss increases. We use a learning rate of 1 x 10-5 and global batch size of 64. For continuous pretraining, we train for 8 epochs using the log ligleikll of the document the training loss, with learning rates of 10 for Mistral and 3 x 10-5 for Llama2. Each passage treated as a training sample, and we use a global batch size of 16.
question-
For closed-book prompting and in-context prompting, we apply instruction template encourage the model to generate a short answer. The prompts are shown Figure 7
In-Context Learning. We explore in-context learning using both fine-tuning and prompting meth- ods. For fine-tuning, conduct task-specific fine-tuning 16 samples for each dataset. We use a learning rate disjoint from the training is collected from the same dataset. We train the model for maximum of 40 epochs, employing early stopping if the validation loss increases for three consecutive epochs.
10 for Mistral and X 10 for Llama2. A validation set of 16 samples,
For in-context prompting, we observe that omitting additional instructions yields better performance for Mistral, whereas adding an instruction template improves performance for Llama2. The prompts shown Figure
16

---

Published as a conference paper at ICLR 2025
Table Statisties of data used in the instruction tuning.
Type
Dattset
#Doess
Alestractions
Centect les
Inciniciio lan
Resporse Imn
COOA Reddy al. 2011
1758
57.2
1083.5
3,5
2.7
DROP Dia 2019
1379
38.4
$48.6
11.0
1.4
Question Answering
NamaiveQA
PubMadQ
2016
1047
29.4
1.0
200.2
574.5
8,5
40.7
4,4
Quail
332.7
2018
Inecontest Leaming
MetaCCT, Minet
1776.
2.9
Instrutionn Following
BookSun
Pot
Krysciskk ai
10 8 m 0924
2022
13102
2914
12.4
1.0
1158.6
348.1
10.3
7.0
205.3
23.3
D ADDITIONAL RESULTS
This section presents extra results for document- question answering and in-context learning evaluation.
D.1 DOCUMENT-BASED QA
In Section 4.1, we evaluate the knowledge recall capability Generative Adapter and baseline models. The complete numerical results are provided in Table 5
We also assess the performance of Ultragist (Zhang et al. 2024) on document QA. Since Ultragist requires predefined compression rate for token reduction, direct comparison with method is not easy. To address this, we report results for compression ratios of 2, 8, and 32, as shown in Table 7 and Figure As expected, Ultragist's 's performance degrades significantly as the compression ratio increases.
Unlike Ultragist, Generative Adapter modifies the model parameters directly, ensuring that its infer- ence time remains identical to that of the base model after contextualization. In contrast, Ultragist's inference and storage costs depend on the number of gist tokens, which approximately equal the original context length divided by the compression ratio. Given its efficient inference, Generative Adapter particularly suitable for scenarios where the model undergoes contextualizatoon once and then reused multiple times, making it effective solution for resource- constrrinn environments such as edge computing.
D.2 IN-CONTEXT LEARNING
Expanding on Section we further evaluate Ultragist on MetaICL, with results summarized in Table 8 Ultragist performs worse than in-context prompting on classification tasks but shows a slight improvement on con-classification tasks. In contrast, Generative Adapter consistently outperforms both Ultragist and in-context prompting across various tasks, demonstrating its effectiveness.
D.3 RETRIEVAL-AUGMENTT GENERATION
Generative Adapter can be combined with prompting techniques or other prompt compression meth- ods provide complementary benefits. To illustrate this, integrate Generative Adapter with Retrieval-Aur -uggentl- Generation (RAG). In this hybrid approach, the contextualization phase re- mains unchanged, and at inference, we prepend the relevant 100-token chunk (retrieved using BM25) to the query. Additionally, for comparison, we report results where the entire context is prepended to the query (denoted as "Generative Adapter + Context"). The results are presented in Table 6
17

---

Published as a conference paper at ICLR 2025
Table 4:
Training and test datasets of MetalCI.
Train
piga, hale spe.cheecchhooccco offensive, google www.wwwwll.eweeeooeo query, social.i.q, circa, yuuref, glue-sst2, scitail, emo, cosmos freebase.qu, ag.news, art, paws, kilt,ay2, glue-quli. quail, ade corpus v2-classification, sciq, hatexplain, emotion, glue-qqp. kilt fever, kill.ng. ny, dbpedia. 14, kilt zsre, hellaswag, squadwith context, holpol qu, glue-moli, ropes, squad-no context. kilt holpolqu discovery, superglue-recoru. race-middle, race-high, lama-trex, swag. gigaword. umazon polarity. biomre, tab fact, tweet eval-emoji, tweet evee-evllooo oval-offenssie, tweet -val-ment,,nnt, tweet qa, imdb, lama-conceptnet, liar, anli, wiki.qu, Irex, wikisql, wino-gramde, wiqu, search.qa,
review xsum, yahoo answers topics, yelp polarity, revier full quarel. financial financial.phrasebon phrasebank, openbookgu, codah. qasc. glue-mrpe. dream, sick, commonsensc qa, medical questions pairs, quartz-with knowledge, poem sentiment, quartz-no knowledge, glue-wnli, climate_fever, ethos-mational.orgg .orignn, ethos-race, ethos-religion ai2.arc. hate apeeesse 88, glue-rte, supergluecb, superglue-copa. tweet. eval-hate, tweet, oval-stance athcism. tweet eval-stance feminist
Test
Zero Zero-shot -hot Prompting
Fine tuning
In. contxxt Prampting
Generative Adapter
Stuncial paranbaa
sparocckga
codan
600
6402
A200
183
medel
1.2
0.31
0.1
wigg
2706 D30
211
Telr Spedll
2.44
cox
xepeplinc
Inedl www.herahll
0.00
Figure 5: In-context learning evaluation of Generative Adapter, based on Llama2-7B-CC across 26 test datasets from MetalCL.
18

---

Published as a conference paper at ICLR 2025
Zero shot Prompting
Fine-tuning
in-context Prampting
Generative Adapter
Ssancil
epriesop
1.11
5.05
0.30
1331
$.500
Quatloee, uneelddg
$1own1
11et deiddre Free
6ds
speedd
022
Inaal www.vitaooo.eeeee... 10m
Figure 6: In-context learning evaluation of Generative Adapter, based on Mistral-7B-Instruuttteo across 26 test datasets from MetalCL.
19

---

Published as a conference paper at ICLR 2025
512
1K
2K
4K
8K
16K
32K
Zoro-Sbot Prompting
Supervised Finc-tuning
20.7
Pretraining
SQuAD
In-coniext Promping
45.4
44.9
43.6
42.6
42.5
38.6
35.1
GenerativeAdapter
39.9
30.3
28.0
Zero-Shot Promping
13.6
Supervised Fine-tuning
19.5
Continuous Pretraining
22.2
StreamingQA
In-contcctt Prompting
47.2
48.7
48.1
48.7
48.0
46.0
39.3
GeacrativAdapptrr
49.3
32.0
Zeto Shot Prompiig
14.6
Supervised Fine-tuning
20.7
Continuous Pretraining
23.9
SQuAD
In-contect Prompting
60.6
55.4
44.9
25.2
9.6
6.4
GenerativeAAapiter
36.2
32.5
31.0
28.9
28.2
24.9
23.6
Zem-Shot Prompting
18.0
LLama2
Supervised Finc-tuning
Continuous Pretraining
20.5
StreamingQA
Prompting
AenerativeAdapter
42.9
(37.8
34.4
32.6
28.7
26.0
25.7
Table 5: All results of the QA accuracy on SQuAD and StreamingQA.
Prompting for Document-based Question Answering
(Context)
## Instruction:
Answer the question based on the context above. Respond with short phrase only. Keep the answer short and concise, without any explanation or additional words
Question:
(Question)
Answer:
Prompting for MetaICI
Input: {demo input}
Output: (demo output)
{... k-shot demonstrations. ...
## Instruction: Based on the demonstration above, provide short and concise answer, with- out any explanation or additional words.
Input: finput)
Output:
Figure 7: Prompts used in the document-based QA and in-context learning evaluation.
20

---

Published as a conference paper at ICLR 2025
Table 6: The Fl scores (along with the number of gist tokens in parentheses) on the Streaming QA dataset show complementary benefits on top of RAG.
Contest I Leigth
512
IK
4K
XK
16K
32K
GenerativeAdagee Adapter
51.5 (OK)
49.3 (IK)
44.7 (OK)
40.9 (OK)
36.7 (OK)
32.7 (0K)
32.0.00)
Generative Adapter Context
67.8 (0.5K)
67.8 (1K)
61.1 (28)
CenerativeNapperr RAG
619 (0.IK)
63.6 (0.1K)
60.8 (0.1K)
60.9 (0.1K)
80.1 (0.K)
59.1 (0.1K)
(6.5(0
Table The scores (along with the number of gist tokens in parentheses) on the Streaming dataset for Ultragist with different compression ratios.
Langth
Ultngiat (Conpression Ratio-2)
(0.3K)
(1K3
61.8(4K)
62.1 (8K)
(160(
Ulingiel (Compression Ratinz))
55.7 (0.1K)
35.4 (0.3K)
55.7
(4.0
(2K)
51.1 (4K)
Ultmgist (Compression Ratio=12)
32.5 (0.1K)
31.1 (D.IK)
30.1 (0.)K)
32.8 (0 (K)
(3.0.0..KK (0.5K)
32.0 (0.5K)
31.4 (IK)
GeneraliveAdggter
51.5 (0K)
(9K3
44.7 (0K)
40.9
36.7
32.7 (08)
32.0 (OK)
Table 8: Comparison between Generative Adapter and Ultragist on MetalCL.
Method
Classification
Non-classification
Ultragist (256 tokens)
41.1
7.5
In-context prompting
60.5
6.7
Finetune
71.8
GenerativeAdapter
63.7
14.9
Prompting wo/ KV Cache (Compression
Prompting w/ KV Cache UltraGist (Compression
Generative Adapter (Compression
Inference Computation
Extra Storage
10a
Fleats (Hiitent
FLOPS
10.
102
512
16K
512
3K
26
Langth
Figure 8: Inference computation and storage requirements for Generative Adapter and baseline methods on StreamingQA.
21


Reviews:
[{"review_id": "cyED24TYU8", "reviewer": "Reviewer_1mHi", "summary": "This paper proposes a neural module called generative adapter, which once trained can accept a stream of input contexts to adapt transformers on the fly. The paper shows that generative adapters work well at lower computational cost than in-context learning on several NLP tasks.", "questions": "- From what I understand, generative adapters are Hypernetworks [[Ha et al](https://arxiv.org/abs/1609.09106)], but that hasn't been discussed anywhere. Do the authors agree with this assessment, and if so, how would they contextualize similar work using hypernetworks for continual learning [[Oswald et al](https://arxiv.org/abs/1906.00695), [Vladymyrov et al](https://arxiv.org/abs/2301.04584)]?\n- Why is the continual pretraining baseline so much lower than the generative adapter baseline? This sticks out and is surprising to me, and makes me think it is poorly tuned.\n- I think a valuable baseline (especially since continual pretraining seems to underperform) might be using the generative adapter loss function (Eqn 5) but optimize the weights of the model directly instead of the generative adapter. If I understood correctly, I don't see any reason why this shouldn't be possible, and if generative adapter is in the ballpark of this baseline, it's a positive signal. Do the authors agree with this assessment?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The method is simple and well-motivated.\n- The method works nicely in a streaming setting to continually update weights using lower number of FLOPs due to being able to reuse weights computed at the previous time step of the stream.\n- The paper is clearly written and easy to follow.", "weaknesses": "- I think the Ultragist baseline should be reported in $\\S 4.1$ and  $\\S 4.2$, not just $\\S 4.3$. Gisting is a natural comparison and I am glad to see it compared against in some form, but I think it could be reported more widely in the paper.\n- Using llama-2-7b doesn't provide much value in my opinion. The 4k context length makes Figure 2b hard to reason about, and I don't see why all llama-2-7b experiments couldn't be done with llama-3.1-8b instead for similar cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UBGHfbqA5c", "reviewer": "Reviewer_apSm", "summary": "The paper introduces GenerativeAdapter, a method for adapting pretrained large language models (LLMs) using lightweight, parameter-efficient adapters generated by a self-supervised generator network. This approach aims to enhance LLM adaptability without incurring the computational cost of fine-tuning or the inference overhead of prompting. Evaluations are conducted across three main areas: factual knowledge acquisition, learning from demonstrations, and user personalization, showing that GenerativeAdapter is effective in reducing computational load and increasing model adaptability to various contexts.", "questions": "How does GenerativeAdapter perform in scenarios where the context information is dynamic or frequently changing, such as real-time conversational agents? It would be interesting to see if the method’s efficiency and adaptability hold in continuously evolving contexts.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper presents an efficient approach to adapting LLMs in real-time by using a lightweight, bilinear adapter generator. This mechanism effectively reduces computational costs associated with full model fine-tuning and inference-time overhead due to prompting, which is particularly advantageous for applications on edge devices.\n\n2. The authors evaluate *GenerativeAdapter* across diverse, practical contexts such as document-based knowledge acquisition, in-context learning, and user personalization. These evaluations, especially in a personalized setting, demonstrate the adaptability of the method and its effectiveness in various contexts that align with real-world demands.", "weaknesses": "1. The authors provide detailed descriptions of the model architecture, adapter generation mechanism, and datasets. However, hyperparameter settings and implementation details could be expanded, especially regarding stability mechanisms like SVD normalization, which could impact reproducibility. It is unclear if the code or models will be open-sourced for verification.\n2. The analysis of *GenerativeAdapter*’s performance in different scenarios is comprehensive; however, the lack of prompt  engineering effort for baselines could introduce an unfair comparison. A clarification on the extent of prompt engineering would ensure a fairer assessment.\n3. While *GenerativeAdapter* achieves superior performance over specific context lengths and adaptation scenarios, claims of general \"efficiency\" for \"personalization\" are strong, given the specific constraints of the study. The authors should emphasize that observed benefits apply to controlled experiments rather than broad generalizations.\n4. The paper introduces concepts like \"test-time contextualization\" and \"parameter-efficient\" without specific definitions, which could hinder comprehension for readers less familiar with these terms. Explicit definitions or brief explanations in the methodology section would help maintain clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3OGkTaqJbG", "reviewer": "Reviewer_mTD9", "summary": "The paper proposes GenerativeAdapter, which is a method to introduce unseen knowledge to LMs without finetuning cost or extra prompts. It generates a new adapter for the target unseen knowledge and utilize it to adapt the LMs.", "questions": "- Is there a suspected reason why the unnormalized adaptor contains extreme values?\n- In the formulations in section 2.3, there is a  G(x_{1:m}). In this case, shouldn't the hidden states input to the generator, not the input tokens?\n-  In the completion loss formulation, shouldn't the output just x_{m+1}, not x{m+1},...,x_n?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses the critical challenge of adapting LMs to incorporate new knowledge effectively, a fundamental issue since LMs can only utilize the information they were originally trained on. \n- It introduces an interesting approach of training another generator to generate an adaptor and optimize adaptation cost", "weaknesses": "- Some of the important experiments are either missing or have questionable setting\n    - It is suspicious that the the settings for experiment 4.1 are fair for all of the baselines. It seems like the knowledge that each baseline has seen are different.\n    - In section 5, the paper mentions the preliminary study shows the quality of the adapter generator is highly correlated with the final evaluation metrics. It would have been better if the results that support this claim is included in the paper. Many times, the training objective is not exactly aligned with the downstream task.\n    - It would be better if the experiment to see how the adaptor affects the original ability of the LMs is included in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RrcOfKXLuI", "reviewer": "Reviewer_PYAH", "summary": "This paper proposes a novel method to integrate the information into the parameters. The experimental results are comprehensive and demonstrate the effectiveness of this method.", "questions": "I don't have any questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The topic is important. \n2. The idea of this paper is novel and clean, inspired by fast weights. \n3. The experimental results show the effectiveness of this method. it is impressive to see that model is able to inject 32k tokens into the parameters.", "weaknesses": "1. **Missing related works**: GenerativeAdaptor adds the adaptor which is quite similar to memory-based methods, where people would add a memory module on the large language models [1,2]. Perhaps the paper should discuss some related papers and the correlations between them. \n\n2. **Size of the additional parameters**: As stated in line 264, GenerativeAdaptor introduces 500 million parameters, however, even using 500 million parameters cannot achieve comparative performance as In-context Prompting (as shown in Figure 2(a)). Consider saving all the tokens in the context, which leads to 32768 * 4096 = 134 million parameters. This is fine, as the proposed method has smaller computation cost than storing all the previous tokens, but this might be something to consider, as it shows that the parameters may not be fully utilized, or the information in the parameters might be overly sparse. \n\n[1] Memllm: Finetuning llms to use an explicit read-write memory   \n[2] MemoryLLM: Towards Self-Updatable Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)