PrefixMemory-Tuning: Modernizing Prefix-Tuning by Decoupling the Prefix from Attention
Authors: Haonan Wang
Brian K Chen
Siquan Li
Liang Xinhe
Hwee Kuan Lee
Kenji Kawaguchi
Tianyang Hu
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Fine-Tuning
Primary area: foundation or frontier models, including LLMs

Abstract: Parameter-Efficient Fine-Tuning (PEFT) methods have become crucial for rapidly adapting large language models (LLMs) to downstream tasks. Prefix-Tuning, an early and effective PEFT technique, demonstrated the ability to achieve performance comparable to full fine-tuning with significantly reduced computational and memory overhead. However, despite its earlier success, its effectiveness in training modern state-of-the-art LLMs has been very limited. In this work, we demonstrate empirically that Prefix-Tuning underperforms on LLMs because of an inherent tradeoff between the contribution of input prompt and parameterized prefix within the attention head. This motivates us to introduce PrefixMemory-Tuning, an architecture that generalizes the principles of Prefix-Tuning while addressing its shortcomings by shifting the prefix module out of the attention head itself and improving its expressiveness. Our experiments show that, across diverse benchmarks, PrefixMemory-Tuning consistently outperforms existing Prefix-Tuning methods. Notably, it achieves competitive performance with modern PEFTs on several general benchmarks, highlighting a potential extension of Prefix-Tuning approaches to become state-of-the-art. Our findings suggest that by overcoming its inherent limitations, Prefix-Tuning can remain a competitive and relevant research direction in the landscape of parameter-efficient LLM adaptation.

Published as a conference paper at ICLR 2026
PREFXMEMORY-TUNING MODERNIZING PREFIX- TUNING BY DECOUPLING THE PREFIX FROM ATTEN- TION
Haonan Wang'.* Brian K. Chen Siquan Li Xinhe Liang*
Hwee Kuan Lee Kenji Kawaguchi Tianyang Hu
National University of Singapore Bioinformatics Institute, A*STAR
The Chinese University of Hong Kong, Shenzhen
ABSTRACT
Parameter-Efffcieet Fine-Tuning (PEFT) methods have become crucial for rapidly adapting large language models (LLMss) to downstream tasks. Prefix- Prefix-Tuning, an early and effective PEFT technique, demonstrated the ability to achieve perfor- mance comparable to full fine-tuning with significantly reduced computational and memory overhead. However, despite its earlier success, its effectiveness training modern state-of-the-an LLMs has been very limited. In this work, we demonstrate empirically that Tuning underperforms on LLMs because of an inherent tradeoff between the contribution of input prompt and parameterized prefix within the attention head. This motivates us to introduce PrefixMemory- Tuning, architecture that generalizes the principles of Prefix- shortcomings by shifting the prefix module out of the attention head itself and im- proving its expressiveness. Our experiments show that, across diverse benchmarks, PrefixMemory Tuning consistently outperforms existing Prefix-Tuning Notably, it achieves competitive performance with modern PEFTs on several gen- eral benchmarks. highlighting potential extension of Prefix- Prefix-Tuning approaches to become state-of-the-arr. Our findings suggest that by overcoming its inherent limitations, Prefix-Tuning can remain a competitive and relevant research direction in the landscape of parameter-efficient LLM adaptation.
while addressing its
Tuning methods.
INTRODUCTION
Large Language Models (LLMs) have advanced at remarkable pace in recent years, driven primarily by larger model architectures and bigger training datasets (Kaplan et al., 2020; Rae et al., 2022). This rapid progress, however, comes with soaring computational costs, making full-parameter fine-tuning on state-of of-therrtt models prohibitively expensive for but the biggest players. To address this, parameter-efficient fine-tuning (PEFT) methods have been introduced. One such approach is Prefix- Tuning (PT) (Li Liang, 2021), a technique which prepends trainable vectors to future inputs of each attention layer in the transformer. PT is both computationally cheap and effective, matching even surpassing more complex methods in several early benchmarks.
However, LLMs have scaled to unprecedented depths and parameter counts, PT has struggled to retain its effectiveness, leading to its replacement by newer methods such as LoRA (Hu et al., 2021) and GaLore (Zhao et al., 2024). The rapid pace of LLM research often leads methods being abandoned before their limitations are fully understood or addressed. In the case of PT, its decline in popularity may reflect a lack of deeper investigation and adaptation. Despite having inherent advantages which are not directly reflected in performance, such as interpretability and relation to concepts such as memory, its poor performance has prevented further exploration of the methodology. its apparent limitations
This motivates a re-examination of PT in the context of modern may be addressed and remedied.
Equual contribution. Cooresponndence Tianyang Hu. Email: hut i www.aannnnnnakkddu.....

---

Published as a conference paper at ICLR 2026
Earlier studies have primarily attributed PT's underperformance to its inability to reshape attention patterns within attention heads (Petrov et al., 2023). We revisit this claim and show empirically that, while this explanation may hold for shallower transformers, it does not generalize deeper architectures typical of modern LLMs. In this work, we argue that the real reason PT performs sub-optimally is its inherent tradeoff' between prefix and input contribution. When the prefix is long relative to input length, the model risks losing input specificity and being dominated by the prefix. When the input is long relative to prefix length, the impact of the prefix greatly diminished. This tradeoff is a result of prefixes being included in the softmax normalization term in the attention head.
Motivated by this, we build on previous work (Chen et al., 2024) to propose PrefixMemory-Tume (PMT), which relocates the prefix outside the attention head by approximating it with an external module consisting of a trainable matrix M and representation function s()). Diagnostic experiments suggest that PMT is substantially more expressive than standard PT, serving as proof of concept for our decision to shift the prefix outside the attention head. We also provide a unified overview of the design choices involved in externalizing the prefix, discussing how future work can build upon our preliminary approach when developing more advanced context-based methods.
Empirically, we evaluate PMT in both few-shot adaptation and application-scale settings across diverse preference alignment and math reasoning benchmarks. Against strong baselines ,e.g., LoRA and full fine-tuning, PMT is consistently competitive. Regular PT flounders in comparison. Our work presents the following key contributions.
We demonstrate empirically that Prefix-Tuning Tuning performs poorly on modern LLMs because of an inherent tradeoff between input and prefix contribution within the attention head.
We introduce PrefixMemory-Tuniiee prefix module outside of the attention head. Prefix includes further modifications to improve the expressivity of the module. We provide a unified overview of our decision -making process in constructing PMT guide users when constructing future context-based methods.
a novel architecture based on Prefix-Tunee Tuming that isolates the
We perform extensive experiments to show the efficacy of PMT. Our experiments show that, the few-shot setting, PMT is competitive with ofate-of-the-art approaches such as LoRA-au Achhievingg an average absolute improvement of 8.1% % over LoRA and 29.4 % over Prefix- Tunnng across all six evaluated settings.
This serves as proof of concept that, when the prefix information is isolated from the attention head like in PMT, prefix-tuning methods can serve as a viable alternative to current modern methods and an exciting future area of research.
2 RELATED WORK
Weight- Baahd PEFT Methods. LoRA (Hu et al., 2021) one of the most widely adopted weight- based PEFT methods, injecting small trainable low-rank matrices into transformer layers while freezing the original weight matrices so that the effective updates lie in a low-dimensional subspace. QLoRA (Dettmers et al., 2024) further improves memory efficiency by applying low-rank adapters on top of a 4-bit quantized base model. LoRA+ (Hayou et al., 2024) does not change the low-rank parameterization itself; instead, it provides theoretical analysis showing that using identical learning rates for the LoRA matrices A and B can hinder efficient feature learning in wide networks, and proposes differentially scaled learning rates with an optimally derived ratio for these adapter matrices. These methods primarily modify linear layers within transformer blocks and thus only implicitly affect the attention mechanism through weight updates.
In contrast to weight-based methods, context-based PEFT methods directly alter the input context provided to LLMs while keeping the backbone weights frozen. Repre- sentative approaches include P- T-Tniing (Liu et al., 2021; 2024), Prompt Tuning (Lester et al., 2021). and Prefix-Tuning (Li & Liang, 2021), which prepend continuous vectors to the input embeddings or inject learnable prefixes into the key-value pairs of attention layers. Empirically, prefix- methods have been shown to achieve competitive performance with full fine-tuning in low-data or few-shot settings on a variety of generation tasks (Li & Liang, 2021), while maintaining strong parameter efficiency. More recent work such as DePT (Shi & Lipani, 2023) and ADePT (Tang et al., 2025) further improves the flexibility of context-based PEFT by decomposing soft prompts into short prefix plus additional low-rank or feed-fon forward components, and adaptively allocating capacity across them for different tokens. In the vision domain, VPV (Han al., 2023) proposes effective
Context-Based PEFT Methods.
2

---

Published as a conference paper at ICLR 2026
and efficient visual prompt tuning scheme that injects prompts inside attention heads, showing that modifying how prompts interact with attention can significantly improve efficiency and performance. However, several studies have also reported that the performance of Prefix-Tuning Tuning saturates degrades as the prefix length increases for large-scale models (Ouyang et al., 2023; Petrov et al., 2023), which limits its effectiveness in learning tasks that substantially differ from the pre-training distribution. Our work is motivated by these scalability issues and aims to improve how prefix-like signals interact with the underlying attention computation.
Feed-Forward Layers as Memory. A complementary line of work views the feed-forward networks (FFNs) in transformers as an implicit key-value memory that stores abstract knowledge in model parameters. Geva et al. (2021) show that FFN layers in Transformer language models can be formulated large key-value memory, where rows of the first linear layer act as "keys" that detect textual patterns and the corresponding rows of the second linear layer serve as values that induce output token distributions. Subsequent work further analyzes how FFN layers construct predictions by composing concept-level sub-updates in the vocabulary space (Geva et al., 2022), and how specific knowledge neurons inside FFNs express relational facts that can be localized and edited (Dai et al., 2022). More recently, Qiu et al. (2024) conduct empirical study on updating the key versus value matrices in FFNs for knowledge editing and fine-tuning, providing additional evidence that FFNs operate as memories that store high-level knowledge.
Despite these limitations, context-based methods exhibit several unique advantages beyond raw performance. They offer greater interpretability due the explicit and externalized nature of learned prompts (Lester et al., 2021; Le et al., 2025), enable lightweight test-time adaptation through prompt retrieval modification (Zhou al., 2022; Yi et al., 2025), and serve memory for storing task-specific information (Kossen et al., 2024; Dai et al., 2022). These properties highlight the broader potential of context- based PEFT methods and motivate the need for improved designs. Yet, for vanilla Prefix Prefix-Tuning its weaker performance in challenging settings makes it difficult fully exploit these advantages practice. This work aims modernize Prefix-Tuning so it remains relevant in the current L.M regime.
form of non-parametric
3 PRELIMINARIES
Transformer models were introduced to address sequence-to-seqquee tasks and primarily consist of attention layers, feed forward networks, and other task specific modules (Vaswani al., 2017). In this paper, we assume an input sequence = [ with token embeddings x; E Rdd for all i [n] such that X E Baxd
3.1 THE ATTENTION MECHANISM
Attention modules are key component of transformers which accepts the entire sequence as an input. Typically, attention layers consist of multiple heads, each with separate set parameters. For notational simplicity focus single headed attention. A single attention head takes the form:
Definition (Single-headed Attention) Given input X
Nxd and trainable matrices Wq. WK E Rddxd Wv E Rdddy, A single attention head takes the form:
0. T Ejsi sim(g: kz)u
Ejsr sim ki))
Based on Katharopoulos et al. (2020), where og the i-th output token whilst qi = Z.WQq k; I&WK k and ryWy and sim(q, k) exp(
gk" is similarity score.
Vax
3.2 PREFIX-TUNIN
Definition Prefix-Tuning, Deffx-Tuunng (PT) is a form of parameter-effiicee fine-tuning that prepends a sequence of vectors the inputs. Given prefix [$1.... sp). where 8, E Rd for all and input X, the new prompt becomes X [S].... Spy I.,..., xz The vectors {s,)1= are then trained based traditional gradient based methods while the rest of the model weights are frozen.
3

---

Published as a conference paper at ICLR 2026
Referring to Equation the inclusion of prefix [s1, ..., sp] yields the following output:
of T Ejs, sim(g; kz)uJ + Σ jsp sim WK (WVs)) Ejse sim ; k₂) + 2 sim(g: Wx8;)
(2)
Compared with full parameter fine-tuning and even most other PEFTs, extremely hight-weight training approach. Research shows that prefix-tuning excels in low-data or few-shot settings when guiding the model to leverage mix of its pretrained tasks (Li & Liang, 2021; Petrov et al., 2023).
offers an
4 LIMITATIONS OF PREFIX-TUNING IN LLMS
In the previous section, note that PT is partic- ularly effective when leveraging pretrained tasks. With the continual increase in the size and ca- pability of large language models (LLMs). sup- ported by an ever-expanding pretraining corpus, one might anticipate a corresponding rise the prominence and effectiveness of PT. However, contrary to expectations, the adoption of Prefix- Tuning has significantly declined in recent years, as evidenced by its sparse implementation on state-of oh-the-art models available in repositories such as Hugging Face. This diminished popu- larity primarily due to PT's underwhelming performance with larger and more complex mod- els, which manifests in reduced accuracy and instability. As depicted in Figure 1, Prefix-Truse Prefix-Tuning consistently underperforms compared to LoRA on three commonly used generative classification benchmarks, despite introducing a similar number of new parameters (see Section 6 for details). With LoRA and other PEFTs consistently outperforming Trefix-Tuning established benchmarks, the overall relevance and applicability of Prefix- Prefix-Tuning methods have been increasingly called into question.
LoRA vs Prefix-Tuning Performance
UUMAO CHAR22 Woaf'
Accurscy (Ng)
60
40
Goerratio
Dbpedia
Bigbench
Figure Performance comparison between Prefix-Tuning and LoRA.
4.1 DOES PREFIX- TUNING ALTER THE ATTENTION PATTERN?
So why doesn't Prefix-Tuning behave well on state-of-the-ar LLMs? The popular stance is that PT cannot alter the attention distribution in the attention heads. As demonstrated in (Petrov et al., 2023), prefix-tuning is only capable of biasing the attention layer activations, which forms a severe limitation. This is shown to be for single-layer transformers and shallow transformers in general. In this study, we argue that, while this analysis is indicative for shallow transformers, it does not capture how PT behaves on LLMs, which are deep multi-layer transformers. Our experiments in B.2 show that PT can modify the attention pattern of LLMs significantly, despite having bad performance. This leads us to believe that an inability affect the attention pattern is not why PT performs badly.
4.2 TRADEOFF BETWEEN PREFIX AND INPUT SIGNIFICANCE
In this section, we argue that the fundamental limitation of Prefix-Tuning is the inherent tradeoff between the contribution of the prefix and the input. This can be observed by rewriting Equation based on the work by (Petrov et al., 2023) as follows:
a + =Σ Dayu,
of
(3)
jSp
where Oūi Egge k, sim(q. 0i Ejs, oi and u3 Wvel.
sim sim.g. WKs:)
Equation 3 shows that the output with Prefix-Tuue Prefix-Tuning can be represented as a linear combination between the attention from the input o; and the attention from each prefix 1h with weights aıy- Prefix- Tuning mainly does two things: we-wesshts the original attention output and adds query-dependent bias vectors.
4

---

Published as a conference paper at ICLR 2026
When the prefix is long relative to input length: In this case, we expect the value of to be large, which results in a greater change in the attention pattern since the base model's 's attention pattern is mainly dependent this, we conducted experiments with different prefix lengths and measured the changes in attention patterns using the REEF framework (Zhang al., 2025). Our results in Table 10 confirm that as prefix length increases, the deviation from the base attention pattern grows. Details can be found in Appendix B.3. When is large, the contribution from the input itself is smaller. The model then has reduced specificity regarding each input and risks being dominated by the prefixes. Too little significance may be placed upon the input itself.
o; this explains
observations in Figure 10. To further verify
This is further exacerbated by the fact that, as the length of the prefix increases, unable to make full use of the space spanned by the vectors (Wr& This phenomenon is also noticed by (Petrov et al., 2023) and is attributed to the competing optimization goals for the prefix 8,. The prefix both needs to grab attention through W 8 and determine direction through Sys;-
is
When the input is long relative to prefix length: we can expect the value of a to be small. The opposite issue arises because when each a, is small, the contribution of the prefix term is diminished. As LLMs get more and more capable, relying more on long sequences arising from techniques such as chain-of- thain-of-thought reasoning (Wei et al., 2023), it is understandable for the effectiveness of prefix-tuning to be severely limited. Too little significance has been placed upon the prefix-tuning
PREFXMEMORY- METHOD AND FRAMEWORK
5.1
MOTIVATION AND CONSTRUCTION
key insight from Section 4.2 is that the trade-off between prefix and input contribution stems from the prefix's 's confinement within the softmax operator of the attention head. This motivates PrefixMemory-Tumiee
a novel extension of PT, which represents pilot attempt to bring the prefix information out of the attention head.
We first draw the terms containing the prefix information out of the attention head by splitting Equation into:
oft = Eisr sim ky)o + (1 Eosp sim(g, WK5, (WVs) Ejsi sim(g;.kg))
=xEss
(4)
Ejs, sim(g, WK8;)
where À E [0, 1] is a constant. This replaces the softmax regularization tradeoff, which is dependent on the length of the input and context, with a fixed convex linear combination similar to previous works (Munkhdalai et al., 2024; Wu et al., 2022). Then, we approximate the similarity metric sim(:.) with kernel feature map e such that sim(:,) ))= Φ6-) ((-) and Φ(-) R2d Rdd, We have
oe T = Eje, sim k2)u Ejgr simssim(i k;))
+ (1 A) p(q)) T Ejso 6 (WV ((q:)
(5)
similar approach is used in Chen al. (2024) to approximate in-context learning prompts, which has shown that the bias term bj = Σjsp <(W (Wvs)) is capable of capturing contextual prompt or prefix information. The natural generalization of this step is to replace the bias b by more expressive, trainable matrix M E Rd, and the analogous term by = ((Wss,) by trainable vector N e Rdo, which yields:
TTT Eysosimler Ko) + (1 - A) p(q.)
(6)
Ejer secssss(gg.. ky)
((g:)
In practice, due to the trainable nature of M and layer normalization, À can be absorbed into the trainable weights. Furthermore, ((q)) N is no longer meaningful for regularization, so it can be removed. Therefore, the final attention output of the PrefixMemory-Tuniie architecture found in figure has the following form:
Σ sim sim(g. gg kg)u
0
PMT T
E sim(gi,k)) + v(a)) M.
5

---

Published as a conference paper at ICLR 2026
5.2 CHOICE OF FEATURE MAP
Regarding the choice of e there are many options which represent tradeoff between expressivity and cost. The choice of & crucial for the performance of this method. A few from existing literature include (xx) elu(x) (Katharopoulos et al., 2020) and (w) = ReLU((Wx + b) (Mercat al., 2024). In this study, as a proof of concept, we conduct experiments with 6(x) elu(x) and Φ(z) gelu(z) due to of implementation. Experiments found in ta- ble show that the choice of representation function, even between elu and gelu has a mean- ingful impact performance. Other choices may offer more expressiveness and better per- formance, but would require significantly more detailed tuning and are left to future work.
Remark 1 (Trainable feature maps) By choosing ow (z) ReLU(Wx + b), the term (w ( becomes effectively single-layer MLP. Depending PrefixMemryy-ltning has the ability to be extremely expressive. As universal approximator used alongside standard attention, this component intuitively has the potential to capture parameter up-
Add
future choices for e(-),
MatMul
SoftMax
dates and contextual information. Prior work (Chen et al., 2024)
has shown that LoRA and SFT alone, when limited to parameter updates, is insufficient for capturing context information.
(opt.)
Linear
However, as PrefixMemory-Tee Tuning is designed for parameter- efficient fine-tuning, our initial goal was to avoid introducing many learnable parameters. Furthermore, exploring issues like MLP ini- tialization and training stability is significant undertaking and is left for future research.
Φ
MatMul
K
PrefixMemory in Scaled Dot-Product Attention.
Figure 2:
5.3 UNIFIED VIEW FOR CONTEXT BASED METHODS
This section outlines the design choices behind PT and PMT, offering the rationale for each guide future implementation decisions. To arrive at PMT, there are two following decisions to be made:
1. Shift the prefix module out of the attention head 2. Approximate Σjsp sim(., WxS)) by ((-) M
Choice 1: Shifting the prefix module out of the attention head is to avoid the limitations highlighted in Section 4.2. By doing so we avoid the scaling on both the input and prefixes so there is no longer the same tradeoff between input contribution and prefix significanc/ccontiiuttio.
Choice 2: Replacing the original similarity met-
ric by <b.) T M shifts the output from Equation to Equation 6. By doing so, lose some of the inherent structure of the attention mechanism. In return, have increase in model expressivity from the flexibility of a training matrix M. To verify this, since both PT and PMT can be viewed as adding guery-dependent d-dimensional bias terms the transformer, we calculate the covari- ate output matrices of the bias from each and find the respective eigenvalue decay. From Figure 3, we see that with PMT, the top eigenvalues corre- sponding to the main principal components are large and decay slowly compared to PT. This in- dicates that the output bias spans many principal components rather than collapsing onto handful of axes. In other words, PrefixMemory-TT Tonnig adds a bias from a more diverse, high-dimensional subspace.
Comparison of Top-50 Attention Output Feature Values
Prefix Tuning
PrefitMemary Tuning
Prefix Tuning Random Inil
Logf LagiFeeture Festure Value)
10
20
30
40
50
Top-50 Feature Index
Figure 3: Spectrum of prefix representations.
During the preparation of this paper, concurrent work (Meyer et al., 2025) was released, suggesting that prompt-tuning methods suffer from limited memorization capacity. In particular, there is an

---

Published as a conference paper at ICLR 2026
Table 1: Fine- Tuning Method Performance Comparison (Aceuracy %). Results across
datasets and models; best-performing results are in boldface, highlighting the effectiveness of PrefixMemory-Tuningg LoRA+ results are added for comparison.
L.LMA2-7B-Char
Qwen2.5-3B-Ing Instruct
Dataset
PMT
Full
LoRA
LoRA+
Prefix
PMT
Full
LoRA
LoRA+
Prefix
GoEmotions
45.2
32.7
36.2
39.4
5.6
37.3
37.8
26.8
32.7
21.2
DBpedia
92.7
92.6
90.1
91.9
61.3
96.9
94.4
89.5
94.4
82.0
BigBench
71.2
38.8
67.4
67.8
21.3
76.6
67.4
61.4
74.2
52.0
upper bound on the amount of new information that can be acquired through context prompts. This observation aligns with Choice 2 in our design, where we relax the previous architecture by replacing context prompts with M, thereby enhancing model expressivity and avoiding such constraints.
In future applications, users can choose to implement these choices in more sophisticated ways. For instance, approach to shift the prefix module out of the attention head by replacing softmax regularization with linear convex combinations can considered quite naive. We fully expect future work to build on this to achieve superior performance.
Remark (The Memory Perspective) We can view our method as explicitly treating the learnable matrix M as an internal memory store. Traditional context-based PEFTs, such as Prefix-Tuning, incorporate context memory by extending the KV inputs, rying the memory capacity to the prefix length. By linearizing attention and summing over the KV circuit, our approach decouples the memory capacity from sequence length and instead makes proportional to the dimensionality of M, enabling more flexible storage of attention patterns. In practice, M allows the model to record and retrieve token interactions without altering the core attention weights by acting as an external memory module. This memory interface is both more direct and more carameter-efficient than auxiliary MLP-bas -ls-b memory modules, which typically require deep architectural changes, incurring higher costs.
6 EXPERIMENTS
To validate both the mechanism and the practical utility of PrefixMemory-Tuning We structure our evaluation into two parts: (i) diagnostic experiments on well-instrumented datasets, enabling analyses of in-distribution (IID) accuracy and out-of-distribution (OOD) generalization that directly verify the previous analysis and mechanism advantages; and (ii) large-scale post-training that studies practical utility and efficiency at scale-treating PrefixMemory- Tuming as PEFT component in two widely used application domains, human preference alignment and math reasoning. Across both parts, we show that MeefixMeemry-TTnninnii outperform baselines while remaining training- and inference-efficien..
6.1 MECHANISM VALIDATION UNDER DIAGNOSTIC DATA
Setup. We evaluate on four generative classification Benchmars--BigBenchh (Suzgun et al., 2022), GoEmotions (Demszky et al., 2020), DBpedia (Kong al., 2024), and Banking77 (Casanueva et al., 2020)-using two instruction-tunee models, LLaMA2-7B-Chal (MHA) (Touvron et al., 2023) and Qwwnn2.5-3B-Instruct (GQA) (Yang et al., 2024). We compare PrefiXMemory-Tuning with full fine-tuning, LoRA (rank r-44) (Hu et al., 2021), Prefix-Tunin -unnnn (virtual tokens m=32) (Li & Liang. 2021), and training-free in-context learning (Brown et al., 2020). For robustness, each configuration is run across five independent trials; in every trial we sample one example per class from the source dataset, fine-tune, and report in-distribution (IID) accuracy on the standard test split, averaged over the five trials. To assess dittr---istiiutionn (OOD) generalization, models fine-tuned on DhgBeenchCoEEmotion//DBBeee full details in Appendix A.1.
are evaluated on Banking77 without additional tuning. We leave
Overall adaptation performance. Across three classification benchmarks, SeeinnMMmor--TTunnnn delivers superior or highly competitive accuracy while updating only small fraction of parameters (Table 1). On BigBench, it reaches 71.2% with LLaMA2-7B Instruct, outperforming LoRA, Prefix-Tunins and even full fine-tuning. On DBpedia, it attains top
and 76.6% with Qwen2.5-3B.
7

---

Published as a conference paper at ICLR 2026
LLMNA 18, Chat Bigfiensh 000 Actunacy (Rouns
Cwean 5, 38 Instivct Higllnch
000 Accuresy (Pound
Defrock Hereeene dou
siiry
PAR
6PC
274
204
90
Acsunacy (h)
ED BD
Figure 4: Pareto plots illustrating the trade-off between IID performance (on Bigbench) and OOD performance (on Banking77) for checkpoints of LLaMA2 and Qwen2.5 during training. Table 2: Impact of feature-map activations on PrefixMemory-TTme Tuning's 's accuracy (%)
PrefixMemory-Tuniee (ELU)
PrefixMemory- (GELU)
PrefieMemooy-TTuie (MLP Kerne)
Dalaset
LLaMA2-7B-CO Chat
Qwenn2 5-3B-lastruet
LLaMA2-7B-CMM Chat
Quen2 3-33-BBasstuuc
LLaMA2-7B-Cha
Quee2 Instrrct
GoEmotions
45.2
37.3
47,0
43.6
DBpedia
92.7
96.9
v3.2
96.4
v5.0
95.0
BigBench
71.2
76.5
72.0
76.2
61.5
77.1
results (92.7% for LLaMA2, 96.9% for Qwen2.5). On GoEmotions, it remains robust (45.2% with within 0.5 points of full fine-tuning). Overall,
LLaMA2-7B-Chat; 37.3% with Qwen2.5-3B- PwefiMMemory-TTniie consistently matches or exceeds strong baselines across models and tasks with far fewer updatable parameters. Details on parameter counts and performance across LoRA ranks are provided in the Appendix A.4.
Balanced IID accuracy and OOD generalization. Optimizing for in-distribution performance often degrades cut-of-distribution robustness; PrefixMemory- Tuning avoids this trade-off. Using Banking77 as OOD and BigBench as IID, Pareto plots over checkpoints show Tefix.Memory-Tuminggg consistently on (or near) the Pareto front for both LLaMA2- 22-2--B-hha -haat and Swen2.5-3B-Ing Instruct (Figure 4). This indicates that PrefixMeeory- Tunnng improves IID accuracy without sacrificing OOD resilience, yielding better eecuracy-robustnee balance than alternative fine-tuning strategies. Additional datasets show the same trend (Appendix A.2).
Stable across data sizes and attention types, with extra gains under GQA. On BigBench with five incremental data rounds, PrefixMemory-TTuing maintains strong, smooth gains across scales and attention mechanisms (Figure 5). It matches or surpasses LoRA and full aall-parameter tuning for both LLaMA2-7B. BhChat (standard attention) and Qwen2.5-3B- -Intrrcct (grouped- -ueer attention). with the largest improvements observed under GQA. These results suggest PrefixMemory-Tuniie centtecture-fienddy and data- scalable, making it practical for modern deployments where GQA is prevalent (Appendix A.3).
Kernel Feature Map Design. We study the kernel feature map o(-) in the prefix module by replacing ELU which is used in the previous setting with GELU As shown in Table 2, GELU yields small but consistent gains several tasks, suggesting & matters. Heavier parameterizations (e.g., MLPs) are left to future work to preserve parameter efficiency.
6.2 PREFIXMEMORY-TUNIIE AT SCALE: PREFERENCE ALIGNMENT AND MATH REASONING
Human Preference alignment. We apply SeeeixMMeory-TTniinee to align 3B 3B-pammmeter Qwen2.5 model with human preferences under different objectives, using 10k training samples for each. Specifically, we fine-tune the model with Supervised Fine-Tuning (SFT) on the Magpie- Ultra v0.1 instruction dataset (Xu et al., 2024), and with two preference optimization methods, Direct Preference Optimization (DPO) (Rafailov et al., 2023) and Simple Preference Optimization (SimPO) (Meng et al., 2024) on binarized UltrFeedback dataset (Cui et al., 2023). All experiments are run with the I.I Aa framework (Zheng al., 2024) and evaluated using AlpacaFval 2, auto-
8

---

Published as a conference paper at ICLR 2026
IID Accuracy on BigBench
Llama Qwen Full FT
ICL
PMT
LoRA
Prefix
LLaMA-2-7B-Chat
Qwon-225-3B-Instruet
60.3%
71.25
72.8%
77.0%
73.0%
76.0%
Accurecy (15
47.3%
87.0%
52.0%
52.1%
Accuracy (1%
00.215
18.9%
58.3%
47.0%
58.7%
$2.0%
48%
34.5%
41.1%
54.%
38.6%
48.5%
20.8%
41.8%
19.2%
24.15
20.0%
22.0%
21.35
21.1%
29.4%
26.3%
20%
32%
0%
3 Training Round
Training Round
Figure 5: Performance over five incremental rounds training data on BigBench. PweiixMemory-Tunigg consistently matches or exceeds baselines, with the largest gains observed on Qwen-2.5-3B-Imss. IInstruct.
Table 3: Accuracy (%) on open-ended math under CFT with different training sizes. For each training size and dataset, the higher of LoRA vs. SrfiiMMemmr--TTiinngg is bold; red (+A) after PrefiXMemoryTTunigg shows the gain over LoRA at the same training size.
AMC23
Minerva-Math
AIME24
Training samples
LoRA
PMT
LoRA
PMT
LoRA
PMT
4 10
40.0
42.5 (+2.5)
16.5
20.2 (+3.7)
10.0
13.3 (+3.3)
47.5
60.0
(+12.5)
14.3
25.7
(+11.4)
13.3
20.0 (+6.7)
50
47.5
60.0
(+12.5)
23.9
62.5 (+38.6)
16.7
23.3 (+6.6)
matic win-rate benchmark for mnstruction-followiin models (Li et al., 2023). Across all objectives, TrefxMemooryTTuingg delivers consistently higher win-rate improvements than LoRA as shown in Ta- ble 4. For example, under SFT the win-rate delta improves by +0.76 with TrefixMemory-Tumee +0.49 with LoRA; under DPO by +4.66 vs +3.52; and under SimPO by +1.74 vs +1.24. These results highlight the robustness and versatility of PrefixMemmor-TTniin the gains are strongest in the preference- baaee settings (DPO/SimPO), where PrefixMemorry-Tumee notably outperforms LoRA. We also observe a small but consistent DPO is better than SimPO in our setup, likely due to SimPO's higher hyperparameter sensitivity (schrieffer-z, 2024).
than
in preference alignment. Notably,
Math reasoning via CFT. We adopt Critique Fine-Tuning (CFT) (Wang et al., 2025), training the model to critique noisy solutions rather than imitate gold answers. We fine-tune Qwen2.5- Math-7B on WebInstruet-CFT with 4K/10K/50K critique pairs and evaluate on AMC'23 (He et al., 2024), AIME'24, and Minerva-Math (Lewkowycz et al., 2022). Across all data scales, Prefi outperforms strong LoRA baseline, with gains enlarging as data grows (Table 3); e.g., at 50K, Minerva-Math reaches 62.5% 23.9% and AMC'23 60.0% 47.5%. These results suggest PrefixMemory-Tumie scales reliably to advanced tasks.
6.3 PARAMETER EFFICIENCY AND COMPLEXITY ANALYSIS
We evaluate PrefixMemory- Tuning in terms of trainable parameters, memory footprint, training throughput, and inference latency. On BigBench, the peak fine-tuning memory usage comparable (e.g., for L.LMMA27B-- 7--chh,, LoRA with r=32 uses 16.5 GB vs. 16.7 GB for The training throughput, measured on BigBench, results in Table show that Prefix -Tuning improves over LoRA on both the 3B and 7B models, and over Prefix-Tunine on the 7B model. The inference latency results on Qwen2.5-Mathh indicate that on the medium-scale 7B model, latency remains comparable to the Base and Prefix- Tuning variants, while on the large-scale 72B model, PrefixMemory-Tumee -aaiiinnorrr-nnngnggg is slightly faster than Drefx--Tuning.
and Qwen2.5-722 Siven25--7B--Inttiie
reported in Table 6,

---

Published as a conference paper at ICLR 2026
Table AlpacaEval 2 win-rate deltas with 10K samples under SFT/DPO/SimPO.
Table 5: Training throughput. All numbers are
iterations per second (Iter/s; higher is better).
PrefiXMemory-Tuiing outperforms LoRA.
Model
PMT
LoRA (r=32) Prefix-Tuning
Method
SFT
DPO
SimPO
LLaMA2-7B
9.70
8.22
6.28
LoRA
+0.49
+3.52
+1.24
Qwen2.5-3B
7.57
6.54
7.94
PeeiixMemoryTTunuie
+0.76
+4.66
+1.74
Table 6: Inference latency on Qwen2.5-Math and Qwen2.5-777 I-strustt TTFT: time-to- to-tist--
token; TBT: time between tokens (lower is better); TPS: throughput (Token/s)
Model
Method Prefill (8)
Prefill TPS (tok/s)
Decode (s)
Decode TPS (tok/s)
TTFT (s)
(s)
Qwen2.5-Math-2 M-Math-BB
Base
0.1769
13558
12.83
39.89
0.0259
0.02506
Owen2.5-Mait M-MthhBBB
PMT
0.1803
13297
13.79
37.12
0.0279
0.02693
Swen2.5-Math-775
0.1788
13409
13.16
38.89
0.0258
0.02571
Qwen2.5-728
5-2.5------t-t-ue Instruct
Base
1.8378
1320
69.30
7.39
0.1314
0,1353
Dwven.5-77BBIimee Instruct
PMT
1.8396
1319
69.97
7.32
0,1327
0.16
Owen2.5-722BInnee Innstrcct
PT
1.8398
1319
70.89
7.22
0.1333
0.1384
7 CONCLUSION
In this work, revisit Prefix-Tuning for modern LLMs and diagnose core bottleneck: prefix signals are effectively trapped inside the attention head. We introduce PrefixMemory-Tumme which decouples the prefix from attention head via a lightweight, query-conditioned module, preserving PEFT efficiency while restoring expressivity and stability. Across instruction following, preference alignment, and CFT-based math reasoning, Prefix.eemory-Tume -Tmnigg consistently matches or surpasses strong adapters such as LoRA under small parameter budgets. This study positions context- methods as viable path future study and deployment. It is primarily a proof of concept to motivate further research into the inherent capabilities of context methods, such as how they tie into test-time scaling and training. We continue this discussion Appendix D.
ACKNOWLEDGMENTS
This material is based upon work supported by the Air Force Office of Scientific Research under award number FA2386-24-1-4011 and this research partially supported by the Singapore Ministry of Education Academic Research Fund Tier (Award No. TI 25 251RES2509).
REFERENCES
Joshua Ainslie, James Lee- Leorp. Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi cualii-query transformer models from multi-head checkpoints. arXiv preprint 3rXiv:2305.1333 13245, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient intent detection with dual sentence encoders, 2020. URL https: htttts://aatttte //arr///a.iv.////a/eaaeeaaeaia httpp////aawwww..//bbba/a// 04807.
org /200/
Brian K Chen, Tianyang Hu, Hui Jin, Hwee Kuan Lee, and Kenji Kawaguchi. Exact conversion of in-context learning model weights in linearized-attention transformers. International Conference on Machine Learning, 2024.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao. Wei Zhu, Yuan Ni. Guotong Xie, Zhiyuan Liu, and Maosong Sun. Utrafeedback: Boosting language models with high-quality feedback, 2023.
10

---

Published as a conference paper at ICLR 2026
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers, 2022. URL
wwg/ abs/ 2tt/2214..8880
Dorottya Demszky, Dana Movshovitz-Attas Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. Goemotions: A dataset fine-grained emotions. arXiv preprint arXiv:2005.00544 2020.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer key-value memories. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, 2021.
layers are
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 conference on empirical methods in natural language processing, pp. 30 2022.
Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang, Siyuan Qi, and Dongfang Liu. E- 2vpt: An effective and efficient approach for visual prompt tuning. arXiv preprint arXiv:2307.13770, 2023.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lorat: Efficient low rank adaptation of large models, 2024. URL https: http/aixvvv.////////////a org/ abs http://gggaaa..../1//// 240..1235.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu. Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang. Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0968: 09685, 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https: /arxiv..vvv. org.or//gaoo..oo abbs 220//00100080 08361.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference machine learning, pp. 5156-5165. PMLR, 2020.
Fanshuang Kong, Richong Zhang, Zhijie Nie, and Ziqiao Wang. Rethink the evaluation protocol of model merging on classification task. arXiv preprint arXiv:4412.13526 2024.
Jannik Kossen, Yarin Gal, and Tom Rainforth. In-context learning learns label relationships but is not conventional learning, 2024. URL https:ttpt.... /ttp://////aiv//// wwwg/bs.//....oooo. wwt.aa/.../.... 12375.
Minh Le, Chau Nguyen, Huy Nguyen, Quyen Tran, Trung Le, and Nhat Ho. Revisiting prefix-tuning: Statistical benefits reparameterization among prompts, 2025. URL htt.sttss://aaiii.ooom abs/2415 02200.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt 2021.
tuning. arXiv preprint
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857, 2022.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2 101.00/90. 2021.
11

---

Published as a conference paper at ICLR 2026
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction followiin models. https::/ggttiii wwwp///stmmmaaaaaaaaaaaaa
5 2023.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning V2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprini arXiv:21 2021.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. Al Open, 5:208 -25, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.051011 2017.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference- free reward. Advances in Neural Information Processing Systems, 37:124198-124235 2024.
Jean Mercat, Igor Vasiljevic, Sedrick Scott Keh, Kushal Arora, Achal Dave, Adrien Gaidon, and Thomas Kollar. Linearizing large language models. In First Conference on Language Modeling, 2024. URL htwps:ttp//oo. /ttps/////p/eevveeeeeeee ew net/ forum? www.mmmiddsoGssssHGoooe
Maxime Meyer, Mario Michelessa, Caroline Chaux, and Vincent Y. F. Tan. Memory limitations of prompt tuning in transformers, 2025. URL hww.t:tt://eette http:s:s/axxxv..rrt abss///////// 2509. 01559.00422 00421.
Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with antni-attention, 2024. URL https: www.a.ww.oog.o....o.ee.oo abs/
2404. 07143.
Yawen Ouyang. Yongchang Cao, Yuan Gao, Zhen Wu, Jianbing Zhang, and Xinyu Dai. On prefix- tuning for lightweight out-of-distribution detection. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1533-1545, 2023.
Aleksandar Petrov, Philip HS Torr, and Adel Bibi. When do prompting and prefix-tuning work? theory of capabilities and limitations. arXiv preprint arXiv:2310. 131:2230..1998,, 2023.
Zihan Qiu, Zeyu Huang, Youcheng Huang, and Jie Fu. Empirical study on updating key-value memories in transformer feed-forward layers. arXiv preprint 24t.::23222.222333. 2024.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadch, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' d'Autmee, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals. Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis. Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis and insights from training gopher, 2022. URL https: /aapavivvvooo org/ abs///b///// 2212.11112. 11446.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 06:53728-53744 2023.
schrieffer-z. can't 't reproduce AE-LC numbers in hf ckpt (Llama-3-88) 3-----3-------- SFT-DOO SimPO). GitHub issue #77,
December 2024, princeton-nlp/SimPPrre repos-
itory.
12

---

Published as a conference paper at ICLR 2026
Zhengxiang Shi and Aldo Lipani. Dept: Decomposed prompt tuning for parameter-effiient fine- tuning. arXiv preprint arXiv:2309.0517. 2023.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sehastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-though can solve them. arXiv preprint 2rXiv:2210.09261, 2022.
Pengwei Tang, Xiaolin Hu, and Yong Liu. Adept: Adaptive decomposed prompt tuning for parameter- efficient fine-tuning. arXiv preprint arXiv:2501.03291, 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307. 09288, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Yubo Wang, Xiang Yue, and Wenhu Chen. Critique fine-tuning Learning to critique is more effective than learning to imitate. arXiv preprint arXiv:2501.17703, 2025.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou, Chain-of-thought prompting elicits reasoning in large language models, 2023. URL httph::t::///ett http://axivv.oo ats///////// 2201. 11903.
Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers, 2022. URL htttps://aratet orgg /200..2222320 08913.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint 2rXvv2239.17455. 2023.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue. Pei Zhang, Qin Zhu, Rui Men. Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan. Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 2rXiv:2412.1511 2024.
Jaming Yi, Ruirui Pan. Jishen Yang, and Xiulong Yang. Mint Memory-infused prompt tuning at test-time for clip, 2025. URL https: / www.w.iw..g......eee....oooooo aabs 250 6. C3190.
Jie Zhang, Dongrui Liu, Chen Qian, Linfeng Zhang, Yong Liu, Yu Qiao, and Jing Shao. REEF: Representation encoding fingerprints for large language models. In The Thirteenth International Conference on Learning Representations, 2025.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient Ilm training by gradient low-rank projection, 2024. URL https:
//arxiv. org/ 2403 03507.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. 13372.
URL htttt:://aaaee org/
13

---

Published as a conference paper at ICLR 2026
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-lanuuage models, 2022. URL https: : /
2203. 05557.
14

---

Published as a conference paper at ICLR 2026
A DETAILS ABOUT CONTROLLABLE EXPERIMENT
A. EXPERIMENT SETUP
Datasets. We use four generative classification datasets:
(1) BigBench (Srivastava al., 2022; Suzgun et al., 2022): A comprehensive evaluation suite consisting of 23 challenging tasks. We focus on the Date Understanding task, formulated as 6-class QA problem in which the model must choose one of six answer categories. For simplicity, we refer to this setting as BigBench.
(2) GoEmotions (Demszky et al., 2020): A fine-grained emotion classification dataset containing 58K Reddit comments labeled with 27 emotion categories plus neutral (28 classes total). As the largest human-annotated English emotion dataset, GoEmotions covers broad taxonomy of emotions. We cast this as a generative QA task: the model reads comment and generates the corresponding emotion label.
(3) DBpedia (Kong al., 2024): A widely used ontology classification dataset consisting of Wikipedia abstracts assigned to 14 top-level classes. We formulate this as generative QA task where the model must output the correct class name given an abstract.
(4) Banking77 (Casanueva et al., 2020): A challenging intent classification dataset designed for conversational systems, consisting of 13,083 customer service queries annotated across 77 categories. We formulate this as a generative QA task where the model must generate the correct label given customer query.
Training and Evaluation Protocol. We assess each method's ability to quickly adapt to downstream tasks in few-shot setting by fine-tuning on up to five independent rounds of minimal data. In each round, we randomly sample one example per class (6 examples for BIG-bench, 28 for GoEmotions, and 14 for DBpedia) form the entire training set. After fine-tuning, we report in-distribution (IID) accuracy on each dataset's standard test split, averaging results over the five rounds to mitigate sampling variability. Since the ability to quickly adapt to new tasks often comes at the cost of generalization, we also evaluate out-of out-o--diitribution (OOD) performance using the Banking77 intent- classification dataset without additional fine-tuning. During inference, models receive a multiple- choice prompt listing all 77 Banking77 intents and must select the most appropriate label for each query. OOD accuracy computed as the proportion of test queries correctly classified, measuring how effectively learned features generalize to unseen domains. We perform this evaluation independently for each of the five models fine-tuned on different source datasets.
Models and Training Configuration. We experiment with two pre-trained language models to assess architectural effects: LLaMA2-7B- -hhat and Qwen2.5-3B-Instee models employ the multi-head attention (MHA) (Vaswani et al., 2017) and Qwen2.5 use grouped- query attention (GQA) (Ainslie et al., 2023). GQA ties together query heads by sharing key/value projections, offering faster inference and lower memory usage, which allows to examine if such architectural differences impact adaptation efficacy. Both models are used in their instruction version in order test the OOD performance. We fine-tune these models using the AdamW (Loshchilov & Hutter, 2017 7) optimizer with a small learning rate and fixed number of training steps (4000 steps). All methods use the same small batch size (batch size 2).
The LLaMA2 series
Baselines. We compare PrefixMemory- Tuning against several bascline approaches for adapting large language models, covering both parameter-efffcient and traditional full fine-tuning, as well as training-fres prompt-based baseline:
Full Fine-Tuning: All model parameters are fine-tuned on the minimal training set for each round.
This represents the conventional approach where all weights of models are updated.
Low-rank adaptation (LoRA (Hu al., 2021)): LoRA freezes original model parameters and introduces trainable low-rank update matrices into each Transformer layer. Only these small rank-r matrices are learned, substantially reducing the number of trainable parameters. We set
r = 64 to approximately match the parameter count introduced by PrefixMemory-Tunin
Prefix-Tunigg (PT (Li & Liang, 2021)): Standard prefix-tuning keeps all model weights fixed, learning only a continuous prefix vector that is prepended to the input at each Transformer layer. We follow the original implementation and set the prefix length m 32.
15

---

Published as a conference paper at ICLR 2026
In-Context Learning (ICL (Brown et al., 2020)): Unlike the previous methods, ICL involves no parameter updates. Instead, the training examples are directly provided as demonstrations in the context at inference.
A.2 MORE RESULTS: IN-DISTRIBUTION ACCURACYS AND OUT-OF-DISTRIBUTIN GENERALIZATION
In this appendix, we provide additional Pareto plots to complement the analysis presented in expri- ments section. Specifically, Figuresó and illustrate the trade-offs between in-distribution (IID) and out-of-distribution (OOD) performances for fine-tuned LLaMA2-7B- models across two additional datasets: GoEmotions and DBPedia.
and Qwen2.5-3B -
Each plot shows the IID accuracy (z-axis) evaluated directly on the respective dataset's held-out test set, and the OOD accuracy (y-axis) evaluated on the Banking77 dataset without further fine-tuning. Points within each plot represent model checkpoints captured at different training intervals, with colors indicating the respective fine-tuning methods used.
Consistent with our observations in the main text, the proposed method frequently occupies positions near the Pareto front. This indicates its effectiveness in maintaining balanced performance between achieving high accuracy on IID tasks and exhibiting strong generalization to OOD scenarios.
LI MMA 7a Chat Goffrotionn Acoursoy (Round
Quen2 5 30 Intouct GGEmmtiins
5)
Faeslting Deferns
Plectann
Deferno Du
PMT
PuRFT
PARFT
694
009
Acourac
Accuray PSI
Figure 6: Pareto plots illustrating the trade-off between IID performance (on GoEmotions) and OOD performance (on Banking77) for checkpoints of LLaMA2-7B-CCE and Qwen2.5-3B-Innst Instruct during training.
LIJMM3 78 Cher D8podia (Round
Oward & 38 instrct BRieda Aooirna $)
Faycuting
Peucaing
neterie PMT
Putkr
000
24
Figure 7: Pareto plots illustrating the trade-off between IID performance (on DBPedia) and OOD performance (on Banking77) for checkpoints of LLLMA2-7B-Chin and Qwen2.5-3B -n--srrutt during training.
16

---

Published as a conference paper at ICLR 2026
A.3 MORE RESULTS: PERFORMANCE ACROSS VARYING DATA SIZES AND ATTENTION MECHANISMS
To further validate the robustness and adaptability of PrefixMemory-TTme across different tasks and attention mechanisms, we provide additional experiment results on two more datasets: GoEmotions and DBpedia. Similar to the main experiment, we incrementally increased the training set size across five rounds, 3B 3B-Instruct (grouped-quer) attention, GQA) using PrefxxMemory-Tuiingg fall-parameter fine-tuning, and the In-context Learning (ICI.) baseline. Figure 8 and 9 illustrates the results across these additional datasets. Overall, these supplementary results reinforce our primary findings that PrefixMemorr--Tmieee scales effectively with data size and adapts particularly well to the grouped-query attention mechanism, outperforming existing parameter-efficient methods.
two models LLaMA-2-78 1-0-------hha Chat (multi-heat attention, MHA) and Qwen-2.5. 2.5- LoRA,
IID Accuracv on GoEmations
Liama & Qwen
ICL
PMT
LoRA
Full FT
Prefix
LLaMA-2-7B-Chat
Qwwen-2.53B-lnne
19%%
100%
87%
02%
Accuracy (1)
Accuracy (%)
60%
42.7%
42:19 42.1%
45.2%
28.3%
36.0%
31.3%
33.0%
78.7%
33.8%
3953%%
2852%
34.1%
37.2%
9.0%
17.4%
13.2%
12.0%
5.6%
8.4%
20.4%
12.5%
8.3%
8,5%
Training Round
Training Round
Figure 8: Performance comparison over five incremental rounds of training data on GoEmotions.
IID Accuracy on DBpedia
Liama & Qwen FT
PMT
LoRA
Prefix
LLaMA-2-7B-Chat
92.1% 1%
0%
92
13 3%
$8.5%
88.9%
8%
05.5% 01.5%
24.5%
09.0%
49%
06.32 58.2%
Accuracy (%))
01.3%
Accuracy (%5
09.0%
61.1%
63%
53.8%
60%
53.7%
44.0%
47%
42%
29.0%
17.1%
13.0%
12.9%
8.6%
0.0%
0.0%
Training Round
Training Round
Figure 9: Performance comparison over five incremental rounds of training data on DBpedia dataset.
A.4 MORE RESULTS: LORA RANK AND PARAMETER BUDGET
We sweep LoRA ranks r E (32, 16, 8} on GoEmotion, BigBench, and DBpedia under the same protocol. Lower ranks reduce parameters but hurt accuracy; higher ranks weaken parameter efficiency. With a budget comparable to LoRA r2=32, PrefixMemory-TTuing outperforms all ranks across datasets (Table 7).
17

---

Published as a conference paper at ICLR 2026
Table Accuracy (%) of LoRA at different ranks vs. PrefixMemory-Tuniig
Dataset
Model
LoRA r-32
LoRA r=
LoRA r=
GoEmotions
LLaMA2-78-CRL
36.24
37.54
37.04
Owen2.5-33-Instmuet
26.85
27.85
26.65
BigBench
LLaMA2-7B-Chl
67.44
65.74
66.94
Awen2.5-3B-Instin
61.45
60.85
62.05
Dbpedia
LLaMA2-7B-CHE
90.14
91.74
91.74
Qwen2.5-38 Bwen2.5-3B-Instuuee
89.55
88.05
92.05
Table 8: Trainable parameter counts and share of total for PrefixMemmor--Tuning LoRA.
Model
Method
Trainable parameters (M)
Percentage (%)
LLaMA2-7B-Chal
LoRA 64
33.55
0.50
LoRA 32
16.78
0.25
LoRA 16
8.39
0.12
LoRA 8
4.19
0.06
PrefixMemory-Uuee Tuning
16.91
0.25
LoRA = 64
14.75
0.48
LoRA 32
7.37
0.24
LoRA 16
3,69
0.12
LoRA
1.84
0.06
PrefixMemory-Tume Maniin
9.51
0.31
B VERIFICATION EXPERIMENT SETUP
To better understand how different methods affect model behavior, we design three comprekension- oriented experiments that focus analyzing attention patterns and internal representations. These experiments aim to shed light on the mechanisms and effects each approach. For consistency and comparability, use the GoEmotions dataset as the in-distribution (IID) dataset and the Banking77 dataset as the -iitt--disttibutionn (OOD) dataset across all experiments. The following subsections detail the setup of each experiment.
B. SPECTRUM ANALYSIS OF PREFIX REPRESENTATIONS
this experiment, we use Swve2255-3BIInsruue as the base model. fine-tune two variants-prefix- tuning (with prefix length of 32) and SeefixMemooy-TTningg--- the GoEmotions dataset using identical training configurations and a consistent sampling strategy (5 rounds).
Let Fb E gn denote the base model's final layer attention outputs for input tokens in total with representation dimension d, and F. E gaxd represent the corresponding fine-tuned model outputs. The representation effect (bias) matrix is computed as:
AF = F. Fbb
After normalization, we perform eigenvalue decomposition on the covariance matrix of representation effects:
AF AF -
where = diag(A), Xa) contains eigenvalues (A1 2 2 Ad), and the orthogonal eigenvector matrix.
We concatenate examples from the GoEmotions test split into the input sequences and extract the self attn.attn output from the final layer. We then compute the corresponding attention outputs bias from the two fine-tuned variants, analyze their eigenvalue spectra, and visualize the top 50 eigenvalues quantify how prefix tuning and method alters the representation space geometry.
18

---

Published as a conference paper at ICLR 2026
Attention: base layer32
Attention: lora laa_r22
Attention: prefix tuning, layer32
Figure 10: Attention Map of LLaMA2-7B-Cha, and its LoRA and Prefix-Tuning fine-tuned versions.
B.2 ATTENTION PATTERN VISUALIZATION
This experiment examines how different fine-tuning methods impact attention behavior. We use LLaMA2-7B BhChat and Qwen2.5-3B-Insttee Inntruntt base models, and fine-tune their respective prefix- tuning and PrefixMemory-Tume periment. We select one example each from the IID (GoEmotions) and OOD (Banking77) datasets test imputs. For each model, we extract the self. attn at weight from the final layer and visualize as a heatmap to reveal attention patterns. For the prefix-tuning variants, we isolate the attention weights corresponding only to real tokens (excluding prefix tokens), normalize them, and then produce the heatmap visualization. The results of the experiment are found in figure 10. systematic analysis of these attention patterns from the perspective of attention sinks (Xiao et al., 2023) would be an interesting direction for future work.
variants using the same data and settings in the previous ex-
B.3 REPRESENTATION SIMILARITY VIA CKA
Inspired by the REEF framework (Zhang et al., 2025), which utilizes centered kernel alignment (CKA) to quantify representation-levè differences, evaluate the similarity between base and fine-tuned models. The CKA similarity between two sets of representations X (base model) and Y (fine-tuned model) is computed as:
HSIC(X, Y)
CKA(X,Y) /HSIC(X, X) HSSC
where the Hilbert-SShmidt Independence Criterion (HSIC) is defined as:
HSIC(X, (m 1)2 K
Here, H = 1 11 is the centering matrix, and Kx, Ky Gram matrices with (Kx)s k(X,,)) and (Ky), )ig = k(Y, Yj), where l is a kernel function (we use linear kernel our experiments). X, denotes the i-th representation vector from layer outputs.
We use D.wen.55-BB-Inse
as the base model, and obtain its prefix- and PrefixMemory-Tuning variants using the used for evaluation. Following the sampling and CKA computation protocol from the REEF paper, we extract decoder representations from the 18 layer of each model and compute the CKA similarity with the base model. This allows us to quantitatively assess how each method alters the internal representations while controlling for computational variance.
training data and setup. The TruthfulQA dataset
Table 9; CKA Similarity Between Different Methods And Base Model
Method
CKA Similarity
Base Model
1,0000
LoRA PrefixMemory Tuning Prefix Tuning (32)
0,9978
0.9432
0.8242
19

---

Published as a conference paper at ICLR 2026
As shown in Table 9, present the CKA similarity between the base model and the models fine- tuned using three PEFT methods: LoRA, PrefiMMemoryTTning and prefix-tuning. is evident that Prefix.Memory-Tune resentations. Our proposed SeefxMeemryy-Tnniie method induces more substantial shifts in the model's 's representation space, indicating stronger impact on the model's 's expressive capacity. On the other hand, although prefix-tuning causes significant changes in the attention patterns, this also leads to much larger representation shifts, which may partly explain its relatively weaker downstream performance.
and LoRA exhibit notably different effects the model's internal rep-
Table 10: CKA Similarity Between Prefix Tuning And Base Model
Method
CKA Similarity
Base Model
1.0000
Prefix Tuning (16)
0.8802
Prefix Tuning (32)
0.8242
Prefix Tuning (64)
0.7957
As shown in Table 10, we further examine how the prefix length affects the representation similarity between the prefix fine-tuned model and the base model under the same dataset and training settings. It is clear that the prefix length increases from 16 to 64, the model's s internal representations deviate more significantly from those of the base model, indicating that longer prefixes introduce more substantial changes in representation space.
In our experiments, since both Prefix- Deeii-TTumng and PrefixMemory- Tuning only modify parameters within the self-attention mechanism-without affecting other components of the decoder layers-the resulting changes in representations can be regarded as a close approximation of changes in the attention pattern.
C IMPLEMENTATION DETAILS
We implemented our experiments using PyTorch and trained our models utilizing the DeepSpeed optimization library with ZeRO Stage 3 to efficiently manage memory usage during training. To further optimize memory and computational efficiency, offloaded both optimizer states and model parameters to CPU with pinned memory enabled, facilitating faster data transfers. Gradient communication and computation were overlapped, and contiguous gradients were enforced to enhance training throughput.
The AdamW optimizer was employed with weight decay of 0.1, momentum terms set as B1 0.9, B2 = 0.95 and epsilon of 1 x 10- Training was executed using automatic precision selection between FP16 and BF16 modes for optimal balance between performance and stability. The learning rate was held constant at 2 x 10-5 throughout the training process. Each GPU processed micro-batch size of one sample per step, while gradient accumulation was automatically managed to simulate larger batch sizes effectively. Gradient clipping was automatically controlled by DeepSpeed to maintain stable training dynamics.
For supervised fine-tuning (SFT) experiments, training was conducted using 2 GPUs, whereas human preference alignment experiments utilized GPUs.
D DISCUSSION
To conclude, this work we argue that the reason why Prefix-Tuning has been ineffective when applied to modern large language models is that prefixes are "trapped" within the attention head. To remedy this, we introduce a novel architecture that generalizes upon existing Prefix-Tunine Tuning methods by approximating the prefix module and shifting it out of the attention head. Surprisingly, even with this slightly naive implementation, our model is able to match of methods such as LoRA on popular benchmarks in a few-shot setting, far outpacing previous prefix-tuning methods. We treat this as proof concept that, if approached correctly, Prefix- Tuning methods can be competitive and exciting future avenue of research.
20

---

Published as a conference paper at ICLR 2026
We also acknowledge the existing limitations of our work. Rather than presenting clear alternative to existing PEFTs, PrefixMemory- Tuning is primarily a proof of concept. The design of our method has yet to be thoroughly ablated. For instance, this line of work can potentially be improved utilizing a more powerful choice of feature map ø such as a learnable one. Further studies are needed to test the limits of our method in more tasks and with more training objectives.
E LIMITATION
Despite the promising results demonstrated by PrefixMemory-Tunin several areas remain open for exploration. Firstly, our implementation utilizes the kernel approximation for simulating attention, specifically the exponential linear unit (ELU). While this choice enabled efficient experimentation and a clear proof-of-concep demonstration, other feature mappings or kernel functions could potentially yield improved performance. Exploring more sophisticated kernel approximations trainable kernel designs remains an exciting area for further enhancement of expressivity and effectiveness. Secondly, although Pemory- and input specificity within attention heads, our experiments did not extensively explore the effects of varying internal dimensionalities or architectures of the externalized prefix module. Further studies investigating these architectural choices and their optimization could unlock additional performance gains. Thirdly, our evaluations were primarily conducted in supervised fine-tuning (SFT) and human alignment scenarios. Extending evaluations to contexts involving abundant data would provide deeper insights into PrefixMeemrr-TTum to computational resource constraints at our institution, such comprehensive studies were beyond our current capabilities. We acknowledge ledge this limitation and leave extensive evaluations to future research. Lastly, evaluations conducted 3B-7B open-source models, which we treat as proxies for larger architectures. Extending the evaluation on larger models (c.g., more than 70B) would provide more complete picture of Merinrymmmrmiiiiii resource constraints our academic environment, such comprehensive large-scale studies are beyond our current capabilities. We acknowledge this limitation and leave extensive large-scale evaluations to future work.
effectively addresses the trade-off between prefix length
maximum capacity to acquire new knowledge. However, due
However, due computational
F BROADER IMPACTS
The introduction PrefixMemory-Tunee offers significant positive impacts by making large lan- guage model (LLM) adaptation more efficient and accessible, thus enabling broader participation in AI research and application, particularly for resource-constraing Additionally, by reducing computational requirements, PrefixMemory-TTee Tuming contributes positively to sustainability efforts in Al development. On the other hand, the enhanced ease of adapting powerful I.MSs also carries risks, such potential misuse in generating misinformation biased content. It is essential for researchers and practitioners to incorporate ethical practices, robust monitoring, and mit- igation strategies address these risks, ensuring that the societal benefits of PrefixMemory- significantly outweigh its potential negative impacts.
communities and organizations.
G LICENSES
We use standard licenses from the community. We include the following licenses for the codes, datasets and models we used in this paper.
Datasets Benchmarks:
BigBench (Srivastava et al., 2022): MIT
GoEmotions (Demszky et al., 2020): Apache License 2.0
DBPedia (Kong al., 2024): Creative Commons 3.0
Banking77 (Casanueva et al., 2020): MIT
Codes:
-LLMMA-Factory Zheng al. (2024): Apache License 2.0
21

---

Published as a conference paper at ICLR 2026
Alpaca-eval Zheng al. (2024): Apache License 2.0
Models:
Breen2.5-3-Instruuee (Yang et al., 2024): Apache License 2.0
LLLMA2-7B--HI (Touvron et al., 2023): LLaMA2 Community License
H
LLM USAGE
We used large language models (ChatGPT and Gemini) as writing and formatting assistants. In particular, it helped refine grammar and phrasing, improve clarity, and suggest edits to figure/table captions and layout (e.g., column alignment, caption length, placement). The L.M did not contribute to research ideation, experimental design, implementation, data analysis, technical content beyond surface-level edits. All outputs were reviewed and edited by the authors, who take full responsibility for the final text and visuals.
22


Reviews:
[{"review_id": "IDQJaHoWVt", "reviewer": "Reviewer_UiQ2", "summary": "In this work, prefix-finetuning in transformer is re-visited and a new prefix-memory module is added to the scaled dot product module in transformer. By revisiting the existing PEFT framework, it's pointed out that the main gap between prefix-tuning and other efficient finetuning approach like LoRa lies in the attention in-balance of learnable prefix and the input context. When the context is long, the contribution of learnable prefix diminished, and vice versa. To resolve this, the prefix-memory module is added directly to the scaled dot prod structure and used to dynamically adjust the learned attention to fix the problem above. Experiments are conducted to show the competitive results on multiple datasets when compared with SFT and other efficient tuning approach like LoRA.", "questions": "- The evaluation is a bit weak. In table 1 the comparisons are made between multiple finetuning approaches, like PMT(proposed), SFT, LoRA, and prefix tuning. However the results are confusing as the SFT results are weaker than other PEFT approaches. This indicates that the dataset used here might not be challenging enough. $\\rightarrow$ Is it possible to test on some more challenging tasks where the full SFT is needed and the usefulness of PMT can be better highlighted? \n- The module is called \"prefix memory\", but in the paper there is no analysis about what does this memory module learned. It would be better to qualitatively or quantitatively analyze about this to provide insights. $\\rightarrow$ Is it possible to get more intuitive understanding about what did this newly added memory module learn.\n- Did not mentioned or compare with some related work, like attention sink[1] and Aprompt [2]. For 1, seems the attention analysis in this work is similar, and for 2 it's a popular prompt based PEFT approach. \n\n[1] Xiao, Guangxuan, et al. \"Efficient streaming language models with attention sinks.\" arXiv preprint arXiv:2309.17453 (2023).\n[2] Wang, Qifan, et al. \"Aprompt: Attention prompt tuning for efficient adaptation of pre-trained language models.\" Proceedings of the 2023 conference on empirical methods in natural language processing. 2023.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The analysis on why prefix tuning failed under extreme cases is convincing and reasonable. E.g in section 4.2, the qualitative analysis makes sense to illustrate the attention balance is broken when either input context or prefix is relatively long.", "weaknesses": "- The evaluation is a bit weak. In table 1 the comparisons are made between multiple finetuning approaches, like PMT(proposed), SFT, LoRA, and prefix tuning. However the results are confusing as the SFT results are weaker than other PEFT approaches. This indicates that the dataset used here might not be challenging enough.\n- The module is called \"prefix memory\", but in the paper there is no analysis about what does this memory module learned. It would be better to qualitatively or quantitatively analyze about this to provide insights.\n- Did not mentioned or compare with some related work, like attention sink and Aprompt. Attention sink had some similar observation that the model's attention is usually overindexed to some specific tokens.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MEyaDYG1Tx", "reviewer": "Reviewer_JeiT", "summary": "This paper introduces **PMT (PrefixMemory-Tuning)**, a novel parameter-efficient fine-tuning (PEFT) method. The authors build on the evolution of Prefix-Tuning (PT), which was largely abandoned due to its poor scaling properties in large models, as well as more recent methods like LoRA and GaLore. While PT had been likened to prompt-based learning by introducing a set of trainable continuous vectors (prefixes) for each input, early analyses identified its underperformance as stemming from an inability to reshape attention patterns within attention heads. However, the authors argue that the true cause of PT’s degradation lies in the **trade-off between the prefix and the input representations**.\n\nLeveraging this insight, the authors improve upon conventional PT, proposing PMT as a more effective and efficient approach. Their experiments show that PMT surpasses both **LoRA** and even **full fine-tuning (FFT)** in terms of fine-tuning performance, providing a more scalable and efficient method for adapting large models.", "questions": "- The PMT architecture can be viewed as adding a linear transformation (similar to a mapping based on \\( q_i \\)) on top of the original model output. This raises an interesting question: could the model architecture itself be modified to inherently incorporate such a transformation, thereby achieving PMT-like behavior without the need for additional components? If this approach were viable, it might improve the model's generalization and efficiency.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper provides a strong theoretical analysis of how fine-tuning (FT) impacts the activation values of attention heads, reducing the relationship to a linear function of the original attention values. This simplification reveals the fundamental cause of **Prefix-Tuning (PT)**'s degradation in large-scale models. The authors effectively demonstrate how the trade-off between prefix and input representations negatively affects model performance.\n\n- The paper offers a clear and insightful analysis of the evolution from **Prefix-Tuning (PT)** to **PMT**, highlighting the **coupling problem** between the prefix and input representations in traditional PT. The authors address this issue by introducing effective approximation techniques that successfully decouple the components. Their experimental results consistently show that **PMT** outperforms **LoRA** and even surpasses **full fine-tuning (FFT)** on most tasks, demonstrating the effectiveness of their approach.", "weaknesses": "- The experimental models used in the paper, such as **LLaMA2-7B-Chat** and **Qwen2.5-3B-Instruct**, are relatively small in scale. Given that these models are not at the forefront of current large-scale models, the evaluation does not demonstrate PMT’s performance on truly large-scale architectures, where the method’s scalability and effectiveness may vary. Including experiments on larger models would strengthen the claims regarding PMT's applicability to cutting-edge architectures.\n\n- The comparison to other PEFT methods lacks more recent and advanced variants such as **QLoRA** and **LoRA+**, which are gaining traction in the field. This limits the strength of the empirical comparison, as it does not reflect the latest advancements in PEFT techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nPC4pMQIvk", "reviewer": "Reviewer_GSuw", "summary": "This paper identifies the performance limitations of Prefix-Tuning in modern large language models (LLMs) and proposes a method called PrefixMemory-Tuning to address these issues.", "questions": "See Weaknesses.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": ">s1: Shifting the prefix module out of the attention head itself is a reasonable start point.\n\n>s2: Experiments show that, in the **few-shot setting**, PrefixMemory-Tuning is competitive with state-of-the-art approaches (such as LoRA).\n\n>s3: The presentation is clear, and the figures are of high quality.", "weaknesses": "> w1: **The discussion of related work contains inaccuracies**. For instance, lines 87-90 state that \"LoRA+ refines this concept further, projecting the model’s weights onto low-dimensional subspaces to achieve efficiency comparable to full fine-tuning at significantly reduced computational cost.\" However, the actual contribution of LoRA+ lies in its theoretical analysis demonstrating that using identical learning rates for matrices A and B in standard LoRA prevents efficient feature learning in large-width networks. The method addresses this limitation by employing differentially scaled learning rates for the adapter matrices with an optimally determined ratio, rather than proposing weight projection onto low-dimensional subspaces.\n>\n>  LoRA+: Efficient Low Rank Adaptation of Large Models. ICML 2024.\n\n>w2: There is **no support** for the claim in Line 144-146 \"Research shows that prefix-tuning excels in low-data or few-shot settings\".\n\n> w3: **Insufficient literature review**: To name a few: (1) For context-based PEFT methods: [1] [2]; (2) Lines 307-315, a good work to show the memory perspective is [3].\n>\n>[1] DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning. ICLR 2024.\n>\n>[2] ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning. ICLR 2025.\n>\n> [3] Transformer Feed-Forward Layers Are Key-Value Memories. EMNLP 2021.\n\n>w4: The experiments were primarily conducted in a **few-shot setting**. Recent studies (such as [4]) have also found zero-shot approaches to be competitive or even superior in certain scenarios. What are your thoughts on this?\n>\n>[4] Revisiting Chain-of-Thought Prompting: Zero-shot Can Be Stronger than Few-shot. EMNLP Findings 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qv5aQFfIyk", "reviewer": "Reviewer_gihG", "summary": "The paper addresses the limitations of Prefix-Tuning (PT), a parameter-efficient fine-tuning (PEFT) method for large language models (LLMs), which struggles with modern LLMs due to a trade-off between input and prefix contributions within the attention head. The authors introduce PrefixMemory-Tuning (PMT), a novel approach that decouples the prefix from the attention head, enhancing expressiveness and performance. Empirical results show PMT outperforms PT and is competitive with state-of-the-art methods like LoRA, achieving an average improvement of 8.1% over LoRA and 29.4% over PT across six benchmarks.", "questions": "1. How would PMT perform with more sophisticated feature maps (e.g., trainable MLPs) compared to the current elu/gelu implementations?\n\n2. Some related work can be theoretically discussed. \"E^ 2vpt: An effective and efficient approach for visual prompt tuning\" ICCV, which adds some prompts in the attention head.\n\n3. What are the computational and memory overheads of PMT compared to LoRA and other PEFT methods at scale?\n\n4. How does PMT handle extremely long input sequences in real-world applications, given the trade-off issues identified in PT?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. PMT addresses a fundamental limitation of PT by relocating the prefix outside the attention head, improving its scalability and expressiveness.\n\n2. Extensive experiments across diverse benchmarks (e.g., preference alignment, math reasoning) demonstrate PMT's competitive performance.\n\n3. The paper provides a clear explanation of PT's underperformance and a unified framework for future context-based PEFT methods.", "weaknesses": "1. The study uses simple feature maps (elu, gelu) as a proof of concept, leaving more expressive options unexplored due to implementation complexity.\n\n2. The paper does not deeply address computational cost or scalability for very large LLMs, which is critical for practical deployment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)