Activated LoRA: Fine-tuned LLMs for Intrinsics
Authors: Kristjan Greenewald
Luis A. Lastras
Thomas Parnell
Vraj Shah
Lucian Popa
Giulio Zizzo
Chulaka Gunasekara
Ambrish Rawat
David Daniel Cox
Venue: NeurIPS 2025 poster ()
Keywords: low rank adapters
parameter efficient finetuning
inference-efficient adapter architectures
Primary area: deep_learning

Abstract: Low-Rank Adaptation (LoRA) has emerged as a highly efficient framework for finetuning the weights of large foundation models, and has become the go-to method for data-driven customization of LLMs. Despite the promise of highly customized behaviors and capabilities, switching between relevant LoRAs in a multiturn setting is inefficient, as the key-value (KV) cache of the entire turn history must be recomputed with the LoRA weights before generation can begin. To address this problem, we propose Activated LoRA (aLoRA), an adapter architecture which modifies the LoRA framework to only adapt weights for the tokens in the sequence after the aLoRA is invoked. This change crucially allows aLoRA to accept the base model's KV cache of the input string, meaning that aLoRA can be instantly activated whenever needed in a chain without recomputing the prior keys and values. This enables building what we call intrinsics, i.e. specialized models invoked to perform well-defined operations on portions of an input chain or conversation that otherwise uses the base model by default. We train a set of aLoRA-based intrinsics models, demonstrating competitive accuracy with standard LoRA while significantly improving inference efficiency. We contributed our Activated LoRA implementation to the Huggingface PEFT library.

Activated LoRA: Fine-tuned LLMs for Intrinsics
Kristjan Greenewald, Luis Lastras, Thomas Parnell, Vraj Shah, Lucian Popa, Giulio Zizzo, Chulaka Gunasekara, Ambrish Rawat, David Cox IBM Research
Abstract
Low-Rank Adaptation (LoRA) has emerged as a highly efficient framework for finetuning the weights of large foundation models, and has become the go-to method for data-driven customization of LLMs. Despite the promise of highly customized behaviors and capabilities, switching between relevant LoRAs in multiturn setting inefficient, as the key-value (KV) cache of the entire turn history must be recomputed with the LoRA weights before generation can begin. To address this problem. we propose Activated LoRA (aLoRA), an adapter architecture which modifies the LoRA framework to only adapt weights for the tokens the sequence after the aLoRA is invoked. This change crucially allows aLoRA to accept the base model's KV cache of the input string, meaning that aLoRA can be instantly activated whenever needed in a chain without recomputing the prior keys and values. This enables building what we call intrinsics, i.e. specialized models invoked to perform well-defined operations on portions of an input chain or conversation that otherwise uses the base model by default. We train a of aLoRA-based intrinsics models, demonstrating competitive accuracy with standard LoRA while significantly improving inference efficiency. We contributed our Activated LoRA implementation to the Huggingface PEFT library
Introduction
The rapid adoption of large language models (LLMs) has catalyzed significant advancements in natural language processing tasks, from text generation to knowledge extraction. However, adapting these models to specific tasks or domains often demands finetuning their immense parameter space, a process that computationally expensive and difficult scale. Low-Rank Adaptation (LoRA) has addressed these challenges by introducing enabling highly customized model behavior without the need to retrain modify the entire model. By optimizing small subset of low-rank matrices, LoRA has emerged as a widely-used lightweight and effective alternative for task-specific customization 225,, particularly for large foundation models such LLMs, and popular LoRA finetuning services are offered by both corporate and open-source LLM providers [26] 36 28 While large models perhaps have less need for finetuning, small models continue strong benefits from LoRA adapters 40 5).
serrameter-efficiene method for fine-tuning [12
Motivated by this, significant work has been done to enable highly efficient serving of and inference with LoRAs, e.g. R4 2) with vLLM [16] incorporating and further optimizing many of these algorithms in its popular, state-of-the-ard inference engine. Yet, while LoRA excels in static or single-task scenarios, in modern applications wide mix of skills is typically needed, e.g. in multiturn interactions and agentic applications 41 44 Crucially. LoRAs often must be carefully tuned to avoid forgetting/degradation of performance on the wide variety of tasks the base model already performs well on. In applications where dynamic switching between multiple specialized skills would be advantageous (e.g. agentic or reasoning pipelines), this strategy breaks down. Additionally, multi-task LoRA training is significantly more difficult than training a LoRA for a single task, and
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

this approach would be inherently non-modular, requiring retraining to incorporate further abilities later on.
In an ideal world, we would like to be able to seamlessly switch-within the same chat/agentic between arbitrary sets of pretrained LoRA configurations for specialized tasks, while keeping the base model as-is for most interactions in the sequence. Unfortunately, the LoRA architecture is not well-suited to this regime, creating major inference inefficiencies. Specifically, for each switch between LoRA adapters, it becomes necessary to recompute the representation of all context tokens prior to generation with the LoRA. In the case of the popular attention mechanism introduced by (37), such a representation takes the form of the key-value (KV) cache of such prior tokens but more generally, it is a problem that reoccurs even in different architectures, for example, in a state-space space model, the state representation of the tokens prior to generation would need to be recomputed if the matrices A and are updated by a low rank correction. This recalculation introduces significant latency. GPU memory, and computational overhead that all scale with the length of the context that must be prefilled, especially as LLMs increasingly work with documents, files, or reasoning/agentic chains in the many (sometimes hundreds of) thousands of tokens [24] 21 [2) (even millions for software engineering [14). This limits LoRA's usability in scenarios where rapid transitions between specialized behaviors are essential.
In this work, we address this shortcoming, presenting our Activated LoRA method, which activates adapted weights only on tokens corresponding to short intrinsic instruction and subsequent genera- tion, leaving the weights for other context tokens unchanged. We elaborate on the setting we envision here. Within the discipline of software engineering, intrinsics are generally useful functions that are built into a programming language whose implementation can be optimized by compiler. We define an LLM intrinsic to be a capability that can be invoked through well defined API that is reasonably stable and independent across model generations and families of how the intrinsic is implemented. Performance metrics such as accuracy, latency, and throughput may vary significantly across such implementations.
The concept of activated LoRA takes an opinionated view on how such differentiated performance could be attained. As inspiration, we note that instructions in LLMs can appear in many places in a prompt: Figure Iillstrates an example of two such places: an "early prompting" case where the instruction precedes the content, or a "late prompting" case where the content precedes the instruction. In the latter, the instruction does not have to be revealed ahead of time, making the representation of the content (KV cache) potentially reusable for or from other tasks. We note that these paradigms can take on many shapes; for example, prefix tuning methods such as [[8] explicitly use early prompting framework, tuning this early prompt. Recently, a set of real-world tasks (and trained adapters) conforming to the late-prompting intrinsics regime was presented in [5 in the context of building performant and robust RAG [17 pipelines. We believe that a vast array of intrinsics-style finetuning tasks can be created with applications throughout the agentic. chat, and reasoning spaces.
LoRA adapter, like the prefix tuning example above, shares the downsides of the early prompting concept; the Ladapedd LLM's internal representation of the content (KV cache) is fit-for-purpose, specific to the LoRA adapter and not reusable by or from the base model or other adapters. Note that while late-prompp tuning approaches do exist and may be viable in some cases, they tend to underperform prefix tuning (early-prompt tuning), which in turn significantly underperforms LoRA- based training [10] As a result, trainable late-prompting style adapter is needed to close this gap.
We therefore introduce Activated LoRA (aLoRA), a novel extension of the LoRA framework designed to only adapt the model's weights for tokens encountered after activation, allowing base model KV cache for prior context to be reused Fiigure2.. By decoupling the adaptation process from the need to recompute the input's representation, olRA facilitates instantaneous switching between specialized models (intrinsics) while maintaining seamless interaction with the base model. This innovation not only reduces computational costs, but also unlocks new possibilities for deploying highly modular, task-specific behaviors within complex workflows.
'In this work, we the term "(KV) cache" to denote the set of saved keys and values for prior tokens in the LLM context, noting that our approach is not limited to attention types that use keys and values. We use "(KV) cache reuse" to mean (re)use of stored keys and values from an LLM call (e.g. base model generation) in a subsequent LLM call (e.g. the aLoRA generation) whose context shares a prefix with the original context and/or generation.
2

---

early prompting
instruction
content
answer
late prompting
content
instruction
answer
Late early prompting framework for intrinsics. The aLoRA adapter architecture is designed to preserve the cache-reuse benefits late prompting by adapting weights only on red tokens, allowing it to reuse the base model cache for the blue input tokens.
Figure 1:
In our experiments, we demonstrate significant speedups for aLoRA vs LoRA on the state-of-the-ard inference engine vLLM [45), and show that aLoRA adapters do not lose accuracy versus LoRA on collection of benchmark finetuning tasks and a collection of real-world "intrinsics" tasks from (5).
Relationship to other LoRA variants. QLoRA 7 proposes quantizing the weights in the base model while keeping the adapter higher precision in training and inference, which dramatically reduces overall memory costs. This can be directly applied to Activated LoRA as well" with KV cache reuse still possible if the base model inference also uses the same or sufficiently similar quantization. Experiments with this are beyond the scope of the present work. DoRA [23] adapters are similar to LoRA with different decomposed low rank weight matrix, with magnitude and direction of the adaptation parameterized separately. In principle, extension to DoRA-style adapters (" ("Activated DoRA") is immediate along the same principles as aLoRA. That said, inference with DoRA is less efficient than LoRA, so it is typically recommended [25 that DoRA adapters be merged into the base model weights - a procedure that not possible for our "activated" approach due to the selective application of the adapter weights.
(a)
prompt
eval:
eval2
prefill (base)
gen (base)
prefill (LoRA1)
prefill (LoRA,)
gen (LoRAi)
prefill (LoRAg)
prefill (LoRA2)
gen (LoRA2)
(b)
prompt
eval:
evalz
prefill (base)
gen (base)
gen (aLoRA1)
gen (aLoRAz)
Figure 2: Computation and memory pattern of (a) LoRA (b) aLoRA used as evaluators of an answer given by base model. (1) prompt is input to the base model, which generates answer, (2) prompt + answer is input to both intrinsics in parallel, which generate eval and eval respectively. Narrow rectangles denote tokens and wide rectangles denote the KV cache.
1.1 Background
Modern decoder-only causal LLMSs are transformers containing a sequence of decoder layers, each of which typically contain MLP layers that operate on each token's 's representation independently, and causal attention mechanism that allows tokens to attend to representations of prior tokens in the same layer (see Figure in the appendix). We review the ubiquitous softmax attention mechanism introduced by (37] and its connection to the KV cache, well as LoRA adapters Further, note that LoRA and aLoRA adapters can be applied to MLP blocks as well, but since is typical practice to not adapt the MLP blocks focus the presentation the attention blocks only.
?Implementation already supported the Huggingface PEFT library. "Extension of this to other forms of attention is straightffowward, but leave this treatment to future work.

---

Attention: Recall that the softmax attention mechanism in each attention layer takes the form
QQ
Attention(Q, K, V) softmax
(1)
vdx
where dy is the dimension and Q, K, V are concatenated queries, keys, and values for the tokens:
Q xwe, K =
= =
where WO, W and WV are weight matrices.
LLM inference and the KV cache: LLMs, using causal attention mask [29]. generate tokens autoregressively one at time. Suppose we have already generated tokens 1,..., 1 with cor-
responding hidden-state embeddings 21,.... .t-1- I- during prefill generation) their keys and values were already computed:
Whenever these tokens were processed (either
K = [Ki; ; V
;V V.-1,, K; ==,, V. - x, w.WY
Due to causal attention, tokens can only attend to prior tokens, and hence are not affected by subsequent generations and can be stored for use by future tokens. At generation step we therefore only need to compute the new query, key, and value for the latest token
QO W. =,=z,Wk, V₁ = =
(3)
and then form the full cached matrices by appending K1: [K 1: Ki]. Vi:+ [Va-ii Vi]. Finally, the output hidden state for token i. is (dg is hidden dimension)
h4 Attention (Qt, Kis, Via) softmax
2+ KT
Via.
(4)
Vds
Because the entire matrix of past keys K1 and values Vi+-1 is precomputed and stored in the "KV cache," at each new step we only pay the cost of computing one new row for each of Q, K, and over the remaining cached K and V. As
V, and performing a single-row softmax and weighted shown in 29), this reduces what would have been an (t2dg) full-matrix multiply down to only Ot d4) operations at each step, yielding massive speedups.
LoRA: LoRA adapts the attention weights WQº, W and WV by replacing them with W + AQ, W + A and WV + +VV, where A 4 A A are rank matrices. This yields
Q = x=X
= x (WK++KA)
V= X + AA).
(5)
This lowers the number of parameters, making finetuning significantly more efficient 13 the LoRA is active for the entire chain, then the KV cache-based inference applies and generation is efficient. If, on the other hand, any part of the input was generated or prefilled by the base model or another LoRA, then K and V for the LoRA are different than the corresponding K and V for the base model, hence any existing base model KV cache cannot be used by the LoRA, meaning the KV cache for the entire (potentially very long) input must be recomputed. To clarify, this involves passing the input through all layers of the transformer in sequence-not just the adapted attention layers-since modifications attention blocks modify the inputs to all downstream layers and tokens.
2 Activated LoRA
Just as in LoRA, our aLoRA architecture adapts the attention weights W.º WK and W by replacing them with Wº A W + A and W + AV, where matrices. The difference lies in how these adapted weights are used. We assume that the default generation model for the chat is the base model, and that intrinsics only operate on these base model generations. As a result, we can assume that the base model has precomputed a KV cache for the input context (the blue region in Figure IT.
K A are rank
The aLoRA architecture designed to match the base model keys and values on context tokens, allowing the adapter to reuse base model KV cache for those tokens, vice versa, the base model to reuse KV cache that the alooR adapter creates for its context inputs. Specifically, in the attention mechanism ( aLoRA only adapts the Q, K, V matrices for tokens occurring after the start of the

---

invocation sequence. Let the token index for adapter activation be Kinvoke, and let the current token have index invake. Instead of (5) we then have
Q=( where X Kistunoola and X ot it are the portions of X coming before and after the aLoRA model invoked. If Xf 1::taookk--1 is associated with tokens generated or prefilled by the base model, then X Xithnwoke -1 WK and Xittwwote- WVV are already in the KV cache and do not need be recomputed. Similarly, any tokens that have been prefilled or generated by the aLoRA model have keys and values processed by the adapted weights, so X i K A and X finvoko. (WV + A)) are also available (except for the current token being generated). As a result, the aLoRA architecture can seamlessly reuse the existing base model KV cache as well as continue to maintain its own KV cache as it generates. Note that adaptations for the MLP blocks that preserve this KV cache property can be done via directly corresponding equations for the Q, K, V blocks above.
X Iistiaroko -1 WQ X (iinncho (WQ +
X Xistmookee W
X Xistimook 1 WV
KK
X it (WK A AK)
V=
X it (WV +AV)
(6)
We can formalize this cache reuse claim as follows (proved in the appendix):
Proposition 1 (KV equivalence and aLoRA inference). For the causal decoder-only transformers we consider, the keys and values (actually, all internal states) prior to linvoke are identical for the base model and any aLoRA adapter model using (6) Specifically, Khase Kiteeooo-- = 1:finwoton and
aaaapter
bbase
yadapter Litwweoke Litnneokee Inference with the al.oRA adapted model can be done causally one token at a time (by simply increasing in (6) Heratively) with KV cache reuse.
Furthermore, we can quantify the computation and memory savings:
Proposition 2 (aLoRA vs. LoRA inference costs). Consider invoking an adapter with Teache tokens of input for which a base model KV cache exists and Tnww < Teache input tokens without cache? S The first token generated by the aLoRA adapter requires (TTachee new operations, while LoRA requires ([Teache + +Tnew )). Furthermore, aLoRA must maintain only ([Twww) additional KV cache memory, while LoRA requires additional (Teache + -new). If N distinct adapters share the first Teache tokens as input, aLoRA costs become CoNNTaahheTare and nNTTeww), while LoRA has (NN (Teache Tnew> )²) and (NN + Tlew]).
As the context length and number of concurrent adapters rise, the computation and memory advantages of aLoRA scale linearly. Note that while methods exist for KV cache compression that can reduce overall memory footprint, e.g. [19), and various methods exist increasing the practical scalability of attention, KV cache reuse is still a major factor in practice context length increases.
Cache reuse patterns: Importantly, while we typically emphasize the aLoRA adapter reusing cache from the base model, the fact that input cache is interchangeable between the base model and every aLoRA adapter presents multiple opportunities for reuse. Consider three regimes:
1. The adapter reuses cache from the base model, e.g. checking a base model generation.
2. The base model reuses cache created when the aLoRA adapter invoked, e.g. when an adapter checks an input prior to sending to the base model for full generation.
3. Multiple adapters reuse the same cache, e.g. when checking base model outputs multiple axes. This regime creates the most dramatic advantage for a over LoRA.
In the results, we explore concrete intrinsics which follow these patterns. Note that any KV cache created by aLoRA adapter for their own generations are not reusable by other adapters the base model, as these tokens come after the adapter weights are turned on. If these strings needs to be input into other adapters or the base model, they should be prefilled again. This is a fairly minor concern, however, as intrinsics generations are often short, and per token, prefill is much faster than generation.
Invocation: We found it useful to demarcate the activation of the ooRA adapter via short invocation token sequence. Advantages are elaborated on in Appendix A and include allowing more space for the adapted weights to process the input. Typically, this sequence will be appended to the prompt prior to generation, just as "generation prompt" is typically appended to prompts for instruct-tuned model (e.g. assis st ant >> or some other string). While the aLoRA will often be invoked programmatically, this design in principle allows for the base model (or other intrinsics) to call the aLoRA model themselves (we do not explore this behavior the current work).
We presume that Thew is the invocation sequence and following.
5

---

2.1 Training
The activated LoRA framework is explicitly designed for tuning the LLM to modify its output conditioned on an input. This regime matches (but is not limited to) instruction finetuning tasks, where the context tokens are excluded from the loss while finetuning the adapter to produce a specified output sequence (supervised finetuning, or SFT). aLoRA can in principle also be used in RI. training pipelines, though we do not explore these in the current work.
Our implementation of aLoRA is in the supplementary material. It seamlessly supports standard Huggingface training libraries [4] as well as inference/generation methods (with without using available base model KV cache) if needed (e.g. for testing or proofs of concept when the efficiencies of vLLM 45 are not needed). Later in this section, we will also show inference experiments using our modification of the more efficient (SOTA) inference package vLLM to support aLoRA.
For aLoRA SFT, training data is specified as set of (possibly multiturn) inputs and completions, typically with the base model's ' chat template applied. An invocation token sequence for the aLoRA model is optionally specified as described in the previous section. This invocation sequence is appended to the input sequence, and the model is finetuned to produce the output given the input. To train the aLoRA adapter, base model is specified, and following standard LoRA practice. apply low-rank adapters to any (or all) of the query, key, and value blocks in the attention layers 6 In training, the aLoRA is aware of which tokens occur before the invocation sequence, and does not adapt the weights for those tokens (as in (6).
Adapter Model Capacity and Increased Rank: Empirically, observe that aLoRA adapters sometimes require higher rank than LoRA adapters to get good performance (no experiments in this paper use rank higher than 32). We offer the following intuition for this. LoRA adapters free to adapt the weights for the keys and values for tokens prior to activation, so they are able to "compress" information needed for generation into the low-rank signal captured by the adaptation. This signal can then be more easily "picked up" by the adapted query weights for generated tokens. See Appendix D for further exploration. This also provides the key motivation for starting the adapted weights at the intrinsic instruction tokens, rather than waiting to activate the weights only at generation time-this choice boosts model capacity.
We now compare aLoRA to standard LoRA both in terms of inference compute costs as well as generation quality and accuracy. Recall that our goal is to achieve significant gains in inference cost, while preserving (not losing) accuracy relative to what is achievable with LoRA.
Inference timing results on vLLM
In Figure 2 we illustrate how computation and memory differ in LoRAs versus aLoRA in simple agent pattern. A prompt (blue) is passed to a model which then prefills the corresponding KV cache, and then generates an answer (green). The task is now to evaluate the answer, in this example, two different hypothetical evaluators are used. An example of such evaluations may be to determine if the answer is faithful to the content provided, or how certain the model is of the answer, given the content. If these evaluators are implemented using LoRAs (Figure 24),, then to generate the evaluations, new KV cache computations need to be performed, independently for each evaluator. In the case of aLoRAs (Figure p). the KV cache of the underlying model is reused, resulting in significant savings.
To prove this experimentally, we modified vLLM [45] to be able to perform inference on aLoRAs with base model prefix cache reuse, and compared LoRA and aLoRA inference on 7 small << 14B) LLMs in the setting where the initial base model prompt length is varied, the base model generated answer is comprised of 256 tokens, and each adapter evaluation is given by 16 lokenssRResee Results are in Figure 3 showing timing for both I evaluator and 5 evaluators g In this experiment, we set the LoRA to have rank 8 and the aLoRA to be rank 32, illustrating that any need for rank increase with aLoRA has negligible inference time effect since the adapter parameters still typically much less than I 1% of the base model parameters.
"Adapting MLP layers is possible but less common in LoRA SFT.
"Code for vLLM implementation can be found at https: http::://gghhbbbb htwp://ww...ommtttuubeeeeeeeeeee
tree/aloze alora
"While 5 evaluators may initially seem a lot, this is stand-in for more complex pipelines where the input, retrieved documents, and generation are all being checked, as in (5).

---

(LoRAin.oRA)
(LoRA/sioRA)
larmataaa
35
Panto//omatttmaa...... 3.168
3.2.8
Iom
25
15-
Intaal prompe lenghh
30
107
102
148
Evaluation time compartson lone evaluator)
Evaluation time comparison liive evaluatorn)
mcta laaallaaao 3.1 68 (ler))
mcto lamallara 3. 68 talor!
104
Owen/Ower25 5: 4iS-rrrruuc-1 (orrl
10-
Owan/Ower25 2- aib-trttruuc--M (olora)
tmmgrommp (bec))
10
10-1
10"
10-2
10"2
10-3
10-0
10P
304
103
10
(tokenst
Figure 3: Comparison of aLoRA and LoRA when used as evaluators in a simple agentic pattern. Top left: Multiplicative speedup of aLoRA evaluator LoRA, showing up 35x improvement depending on base model and prompt length. Top right: Multiplicative speedup for the end-to- end pipeline including the base model generation (256 tokens) and 5 parallel eval (adapter) generations (16 tokens each). Despite the large fixed cost of the base model call, end-to-end aLoRA speedups still significant, highlighting LoRA inefficiency. Bottom row: Log-log plots for the wall clock evaluation, showing that even for small models, the delay for LoRA becomes significant in absolute terms the prompt and number of evaluations in the agentic pipeline scale.
The speedup factor of aLoRA increases as prompt length increases, adapter generation time improving over 2 to 7x even for 250-token prompts, and over 20x for most models on the longest prompts. Despite most of the compute advantages coming from the prefill savings (time to 1st token of the adapter), we still see highly meaningful speedups in the end-to-end pipeline which also includes the large fixed cost of generating 256 tokens with the base model. Finally, in the last row, despite adapters only generating tokens, we see that the LoRA delays are noticeable wall clock time. Note these wall clock delays will be further magnified for more complex agentic or reasoning pipelines where many rounds input and generation happen before the overall pipeline completes.
Finetuning Accuracy Results
Having demonstrated significant inference time speedups for aLoRA, remains to ascertain whether we lose generation accuracy or quality. We first train LoRA and aLoRA adapters on 4 LLMs on a set benchmark SFT tasks, and then consider a set more challenging "intrinsics"-type tasks for which well-engineered, very recent LoRA adapters already exist against which we can compare. Our code implementing Activated LoRA has been contributed to the Huggingface PEFT library httt::///gggta....omo/ggggggggee www/huuggigfaaee/eeeeeeetoo
[25), which also use for LoRA g

---

1ask
+2 benchmark collection of Llama 3.2 11
Liama 3.2 3B
Liama 3T
Mistral 7B
LoRA
aLaKA
LoKA
aLoRA
LoRA
al.oRA
LoRA
aLOKA
Rengali Hate
Speech Clasification
79.30%
81,94%
86.348
89.43%
70.04%
85.02%
72.25%
85,46%
WIQA:
Classification
68.92%
71.38%
76.15%
76.00%
74,92%
78.00%
61.08%
79,08%
MMLU Concegraal
Playsies MOQA
33.339
38.89%
72.2066
66.67%
55.56%
53.5698
55.56%
55.56%
MMLJ College Computer
Science MCQA
58.33%
58.30%
75.00%
75.00%
SocialIQA Question
Generation
86.00%
88.77%
89.85%
90.15%
52.00%
88.92%
97.22%
90.92%
Hindi Sentence
Pertu
69.00%
74.69%
98.309
63.89%
99.23%
96.30%
SuperGi LIE
Generation
98.42%
95.79%
95.26%
96.845
98.95%
92.1
99.47%
92.11%
Figure 4: LoRA aLoRA accuracy (%) each task across base models after hyperparameter grid search guided by the validation set. While individual task performance is noisy due to the size of the datasets etc., there is no consistent accuracy loss from using aLoRA over LoRA.
4.1 Benchmark SFT tasks
In (2). a collection of 1000 instruction SFT tasks were curated and LoRA adapters were trained for each. These tasks were drawn from the Super-Natural Instructions datasets, which in turn drew from sources such as MMLU [1] etc.
From this list, we selected tasks at random, roughly split between multiple-choiee and freeform outputs, We excluded from consideration any tasks with (a) too small datasets, (b) overly open-ended instructions, or (c) difficulty such that trained adapters did no better than random chance. The selected tasks and datasets are detailed in Appendix [F] For each task, we trained both LoRA and aLoRA adapters for 4 instruct-tune -unne models of various sizes. To ensure fair comparison, for each task-model pair, using the validation set performance we performed grid search over 4 rank values and learning rate v and followed typical LoRA SFT best practices for all parameters. Note that we did not observe any pattern to the performance-maxinii particular there was not a clear pattern of LoRA or aloR needing higher or lower rank. The invocation sequence was set to equal the model's chat template generation prompt. See Appendix for further details.
rank value in these experiments,
Results are shown in Figure While there is decent variability due to the generally small size of the datasets and non- non-exhaustivenes of the hyperparameter search, overall can be seen that neither aLoRA or LoRA has a consistent accuracy advantage. The median performance difference is 0.0%, and the mean performance difference is 0.6% in favor of LoRA (3.06% in favor of activated LoRA when restricted to MC tasks). Using 2 sided t-test, the p-value for rejecting the null hypothesis of equal means 0.8, failing to reject at level 0.05. This supports our thesis that aloRA can achieve comparable accuracy to LoRA while significantly outperforming in inference costs.
4.2 Intrinsics Tasks
In this subsection, we consider real-world tasks that fit the framework of intrinsics, specifically, settings where the task requires a potentially long input that either (a) already has base model KV cache available from either being input to or generated by the base model, or (b) does not have (complete) base model KV cache available, but base model KV cache created through an aLoRA call will be used in subsequent base model call. We draw from those proposed in (5) for a RAG pipeline, though these intrinsics are not limited to RAG. See [ for deeper motivation for each intrinsic and extensive experimental results comparing their LoRA adapters to strong baselines.
In (5), LoRA adapters were trained for the intrinsics using the Granite 3.2 8b Instruct model. For direct comparison to their released adapters, we train corresponding aLoRA adapters (using the same datasets and chat-template-hare -baed formatting) and compare their performance in terms of generation
'PEF contains the model implementation and handles inference. Training in our experiments done using a standard Huggingface TRI. 38 trainer (SFTTrainer).
Note that this experiment represents 1120 separate adapter training
8

---

quality and/or accuracy Note that since all the below models use the same base model, they can be swapped in and out as needed the same flow. The reader can envision the wide range of possibilities enabled by these intrinsics, e.g. for RAG. See the Appendix G for additional details for this section.
T
This intrinsic provides a Certainty score for model responses to user questions. The model will respond with a number from 0 to 9, corresponding to 5%, 15%, 25%, ..... 95% confidence respectively. Training data for these confidence scores are obtained by applying the UQ calibration method of [32] to a large, diverse set of benchmark question answering datasets, quantizing the predicted confidences and using these predicted confidences as SFT targets for the adapter. Following the chat template, the invocation sequence is 4 tokens: >>, Note that the model is evaluating re- sponses from its base model in other words, it cannot be applied to generations from other models. The al LoRA architecture is thus particularly well-suited to this use case. In practice, the goal is to provide highly efficient uncertainty score without having to resort to expensive larger judge models.
Uncertainty Quantification:
< starrt_os of _f____o_rrlll__ role >certainty
The Uncertainty Quantification intrinsic returns an ordinal score, so we compute the mean absolute error between the predicted integer and the target integer in the SFT data, Results are shown in Figure 5 Note that performance is largely unchanged using aLoRA instead of standard LoRA.
Certainty Score
LoRA
al.oRA
MAE
0.50
0.49
Figure S: Test error for the Un- certainty Quantification intrin- sic. Note that aLoRA does not lose meaningful performance.
Answerability Determination: This intrinsic assesses whether user's final query in a multi-turn conversation can be answered given the retrieved documents. In RAG other settings, this decides whether to proceed with generation or abstain with an "I don't know" response. The invocation sequence is set to sttar_of_roooe Panswerabillee lity sfarndooffrrooooo >>. We tested on binary answer- ability classification on the single-turn SQUADRun Benchmark [30] with the user query and the supporting documents, and the multi-turn MT-RAG Benchmark [15] using full multi-curn conversa- tion history along with the supporting documents. Figure 6 shows the results. Overall, the aLoRA model does not lose performance relative the LoRA model.
Query Rewrite: This in- trinsic is generally applica- ble for multi-turn conver- sational use cases, and its role is to perform rewrites of user queries for better performance for the down- stream tasks. It is especially useful in RAG settings, see the metrics and evaluation results below. The query rewrite task is as follows: given a multi-turn conversa- tion between a user and an Al assistant, de-contextualize the last user utterance (query) by rewriting it (whenever necessary) into an equivalent version that is standalone and can be understood by itself. The rewritten query can be sent to downstream tasks (c.g., to retriever in a RAG setting) as a better replacement for the original user query, and without the need for (a potentially very long) context.
Dataset
Adapter
Unans,
Ans.
Weighted F1
R
F1
P
R
FI
LoRA
84.2
68.0
75.2
73.1
87.2
79.5
77.4
SQUADRUN Dev
aLoRA
83.0
81.1
82.0
8144
83.3
82.4
82.2
LoRA
85.4
89,3
87.3
87.0
82.4
84.6
86.1
RAG Benchmark
al.oRA
85.8
89.1
87.4
86.8
83.0
84.9
86.2
Comparison of classification performance of LoRA vs. al A across the SQUADRUN Dev set and MT-RAG benchmark. Metrics are broken down by class (Unanswerable vs. Answerable) and include precision (P), recall (R), Fl score, and weighted Fl.
Figure 6:
Retrieval recall evaluation (Recall@k) with different query rewrite strategies, evaluated on full, mon-standalone and standalone subsets of MT-RAG dataset (15) are shown in Figure All retrieved passages are obtained using the Elser retriever with the same settings as in the above paper. We evaluate on three different subsets of MT-RAG detailed in the appendix.
Answer quality evaluation using RAGAS Faithfulness (RAGAS-F) and RAD-Bench on full, non- standalone and standalone subsets of MT-RAG dataset shown in Table 7b (see appendix for details). Note that throughout, performance numbers for aLoRA and LoRA are within a point or two. See [5] for additional comparisons showing that these approaches outperform benchmarks and are very close to the performance with gold rewrites.
As they released LoRA adapters for only base model, limit ourselves their choice of base model.

---

Fall MI-RAG
Non-standalone
Standalone
Sirategy
RG5
R0:10
R855
R3:10
R2:220
R355
R3:100
R0::20
0.42
0.63
LoRA
0,56
0.68
0.76
0.44
0.57
0.66
0.63
0.75
0.83
(a) Retrieval (Recall @55, @ 10, and (0)
Non-standalone
Standalone
Strategy
RAGAS-F
RAD-Beuch
aloRA
0.77
0.69
0.83
LoRA
0,81
0,70
0.79
0.69
0.83
0.71
(b) Answer generation quality (RAGAS-F, RAD-Bench
Figure 7: Impact of query rewrite strategies on both retrieval and generation tasks across subsets of MT-RAG.
Jailbreak Detection: This intrinsic is designed for detecting jail- break risk within user prompts. Prompts with jailbreak risk vary across a wide range of attack styles from direct instructions, to encoding-style, social-hace hocial- based attacks and even ones that ex- ploit special token or context overload [В In our experiments focused training intrinsics for detecting social hacking style of adversarial prompts. As with prior intrinsics, the alooR//LoRA detectors trained in the same conditions with rank of 32. The intrinsic is trained to return binary label - "Y" indicating jailbreak risk present and "N" indicating no risk.
Acc
TPR
FPR
aLoRA
LoRA
0.943
0.898
0.011
Figure 8: Performance of jail- break risk detectors.
After training, we evaluate the jailbreak intrinsic on new out-of-distribution datasets not used in training to robustly assess generalizability [46]. The out-o--distribution datasets comprise of 3,282 samples containing mixture of samples with jailbreak risk 333 20 39] and benign samples [4]. As with the other intrinsics, we see very little performance difference between LoRA and aLoRA.
Conclusion
We presented Activated LoRA (aLoRA), novel modification of the LoRA framework enabling efficient and dynamic adaptation of large language models without requiring recomputation of the key-value (KV) cache. By modifying LoRA adapt weights only for tokens generated after activa- tion, aLoRA facilitates seamless integration into multiturn settings, enabling the use of specialized "intrinsics" for well-defined operations within broader conversational or processing pipeline. While aLoRA adapters sometimes require higher rank than LoRA, this is not meaningful downside at inference time since the rank matrix multiplications are a very small part of overall inference costs (although training costs do increase).
Our experiments demonstrate that aLoRA maintains competitive accuracy compared to standard LoRA while significantly reducing inference costs. This capability was showcased through appli- cations in uncertainty quantification, answerability determination, hallucination detection, query rewrite, and jailbreak detection. The flexibility and efficiency of aLoRA highlight its potential to streamline the deployment of modular, task-specific models complex workflows, paving the way for more adaptive and responsive LLM--diven systems.
We anticipate that aLoRA be profitably applied to vast array of intrinsics-style finetuning tasks that can be created for applications throughout the agentic, chat, and reasoning spaces. Future work will explore developing and proving these expanded use cases.
10

---

References
[1] Awesome chatgpt prompts.
[2] Rickard Gabiis-abri,,ssnn, Jiacheng Zhu, Onkar Bhardwaj. Leshem Choshen, Kristjan Gree- newald, Mikhail Yurochkin, and Justin Solomon. Compress then serve: Serving thousands of lora adapters with little overhead. arXiv preprint arXiv:2407.00066, 2024.
[3] Christopher Clark, Kenton Lee, Ming-Wee Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 10044, 2019.
[4]
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly:
Introducing the world's first truly open instruction-tuned LLM.
2023.
URL https: / wwww www.daaaaabas..a httpaa/rw......cooooo/oo////////
de wwr.-f-ere-tt-----eeee
[5] Marina Danilevsky, Kristjan Greenewald, Chulaka Gunasekara, Maeda Hanafi, Lihong He, Yannis Katsis, Krishnateja Killamsetty, Yatin Nandwani, Lucian Popa, Dinesh Raghu, Frederick Reiss, Vraj Shah, Khoi-Nguyen Tran, Huaiyu Zhu, and Luis I Lastras. A library L.LM intrinsics for retrieval-augmmentd generation. arXiv preprint arXiv:2504.11000 2025.
[6] Boyi Deng, Wenjie Wang, Fuli Feng, Yang Deng, Qifan Wang, and Xiangnan He. Attack prompt generation for red teaming and defending large language models. Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2176-2189, Singapore, December 2023, Association for Computational Linguistics. doi: 10. 10.18863/v 1/2023 findings-emmlp.144 URL
org/2023. findings emn 1433
.43
[7] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. Advances in Neural Information Processing Systems, 36: 10088-10115, 2023.
[8] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high- instructional conversations. arXiv preprint 2305. 14233, 2023.
[9] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, and Alex Vaughan. The llama 3 herd of models. arXiv preprint arXiv:2407:21788, 2024.
[10] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatren and Graham Neubig. To- wards unified view of parameter-efffiient transfer learning, arXiv preprint arXiv:2110.04366, 2021.
[11] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[12] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Marco Morrone, and Quentin de Larous- silhe. Parameter-effictte transfer learning for NLP. In International Conference on Machine Learning, pages 2790-2799, 2019.
[13] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
[14] Mingjian Jiang, Yangjun Ruan, Luis Lastras, Pavan Kapanipathi, and Tatsunori Hashimoto. Putting it all into context: Simplifying agents with LCLMs, 2025. URL htttps://aras
orq/ab abs 2505. 08120
[15] Yannis Katsis, Sara Rosenthal, Kshitij Fadnis, Chulaka Gunasekara, Young-Suk Lee, Lucian Popa, Vraj Shah, Huaiyu Zhu, Danish Contractor, and Marina Danilevsky. MTRAG: A multi- turn comversational benchmark for evaluating retrieval- -uggeente generation systems, 2025.
URL ttt:: www.iiiooooroooooo...o
2501, 03468
[16] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model
11

---

serving with paged attention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 626, 2023.
[17] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, and Tim Rocktäschel. oetrieval-augmennt generation for cnowledge-intent Systems, 33:9459-94444 9474, 2020.
NLP tasks. Advances in Neural Information Processing
[18] Xi Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In International Conference on Learning Representations, 2021.
[19] Yuhong Li, Yingbing Huang, Bowen Yang. Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. Advances in Neural Information Processing Systems, 37:22947-229700 2024.
[20] Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. ToxioChat: Unveiling hidden challenges of toxicity detection in real-world user-Al conversation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 4694-4702. Association for Computational Linguistics, 2023, doi: 10. 1001866/1/22223IIDIIIN SEEDNNGS-EEMLP. EMNLP. 311. URL https: / /toi. /ttp:///oo..../o.//////o 1111111..66335 18653/6633/33 2023. Eindings--mminnn. 311
[21] Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, and Daya Guo. Deepseek-v2: A strong, economical, and efficient mixture-of-expert language model. arXiv preprint 2tt.:2400.044334. 2024.
[22] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03112, 2023.
[23] Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.
[24] Xiang Liu, Zhenheng Tang, Hong Chen, Peijie Dong, Zeyu Li, Xiuze Zhou, Bo Li, Xuming Hu, and Xiaowen Chu. Can LLMs maintain fundamental abilities under KV cache compression? arXiv preprint arXiv:2502.01944, 2025.
[25] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. PEFT: Sfate-of-the-art parameter-effiiiet fine-tuning methods. https
//github. hwt.:/c/./.hgggggggggggg peft 2022.
[26]
OpenAL. OpenAl fine-tuning API. https:://platform.peeen/ wwt. cetom//ocs/guiii guwdesssss//ge/////////// htt.pp/pa.....ooooooo/u/...eese.
Fine-tuning 2024.
[27] Niklas Pfister, Václav Volhejn, Manuel Knott, Santiago Arias, Julia Bazińska, Mykhailo Bichurin, Alan Commike, Janet Darling, Peter Dienes, and Matthew Fiedler. Gandalf the red: Adaptive security for LLMs. arXiv preprint arXiv:2501.00 07997, 2025.
[28]
Predibase. Multi-lora inference server that scales to 1000s of fine-tuned LLMs. https //loraexchange.. ai 2024.
[29] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
[30] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789, 2018
[31] Ambrish Rawat, Stefan Schoepf, Giulio Zizzo, Giandomenico Cornacchia, Muhammad Zaid Hameed, Kieran Fraser, Erik Michling, Beat Buesser, Elizabeth M. Daly, Mark Purcell, Prasanna Sattigeri, Pin-Yu Chen, and Kush R. Varshney. Attack atlas: A practitioner's perspective on challenges and pitfalls in red teaming GenAl. CoRR, abs/2409. 15398, 2024.
[32] Maohao Shen, Subhro Das, Kristjan Greenewald, Prasanna Sattigeri, Gregory Wornel,, and Soumya Ghosh. Thermometer: Towards universal calibration for large language models. In International Conference on Machine Learning, 2024.
12

---

[33] Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "Do anything now": Characterizing and evaluating in mn-the-wild jailbreak prompts on large language models. In Bo Luo, Xiaojing Liao, Jun Xu, Engin Kirda, and David Lie, editors, Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS 2024, Salt Lake City, UT, USA, October 14-18, 2024, pages 1671-1685. ACM, 2024.
[34] Ying Sheng, Shiyi Cao. Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, and Kurt Keutzer. S-LoRA: Serving thousands of concurrent LoRA adapters. arXiv preprint arXiv:2311.0322
2023.
[35] Arthur Mensch Chris Bamford Devendra Singh, Chaplot Diego de las Casas Florian, Bressand Gianna Lengyel Guillaume Lample Lucile, Saulnier Lélio Renard Lavaud Marie-Anne, Lachaux Pierre Stock Teven Le Scao, Thibaut Lavril Thomas Wang Timothée Lacroix, William EI Sayed Albert Q Jiang, and Alexandre Sablayrolles. Mistral 7b, arXiv preprint arXiv:2310.06825, 2023.
[36]
TogetherAL. Together fine-tuning.
Eine-tuning] 2024.
[37]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems, pages 5998-6008 2017.
[38] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. TRL: Transformer reinforce- ment learning. https: htt//ttthubb thub. wwt.coom//oo.ooo http://hhggnnfccececcc
2020.
[39] Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding. Vlad Ionescu, Yue Li. and Joshua Saxe. CYBERSECEVAL 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models. CoRR, hts/2408.00100.. 2024.
[40] Shangshang Wang, Julian Asilis, Ömer Faruk Akgül, Enes Burak Bilgin, Ollie Liu, and Willie Neiswanger. Tina: Tiny reasoning models via lora. arXiv preprint arXiv:2504. 15777, 2025.
[41] Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingehen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, and Jaskirat Singh. Openhands: An open platform for ai software developers as generalist agents. In The Thirteenth International Conference on Learning Representations, 2024.
[42] Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, and David Stap. Super-naturalinsttuutios:: Generalization via declarative instructions on 1600+ NLP tasks. In 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, 2022.
[43] Thomas Wolf et al. Huggingface's 's transformers: Sfate-of-the-ar natural language processing. co cottrannsorrmeeeeeeecceoeoooo.ee
https::/huggingg:
2019.
[44] John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528-500 50652,, 2024.
[45] Xu Zhu et al. vLLM: Fast inference of large language models. https://github. htt.gggtthhhhcooo com/ 2023.
viIm-projjec///e
[46] Giulio Zizzo, Giandomenico Cornacchia, Kieran Fraser, Muhammad Zaid Hameed, Ambrish Rawat, Beat Buesser, Mark Purcell, Pin-Yu Chen, Prasanna Sattigeri, and Kush Varshney. Adversarial prompt evaluation: Systematic benchmarking of guardrails against prompt input attacks on LLMs. arXiv preprint arXiv:2502.154277 2025.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
13

---

Justification:
We clearly state and support our claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss aLoRA requiring higher rank, being limited to non- non-pretraining type tuning tasks, and focusing on intrinsics-style tasks.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer:
[Yes]
Justification: Proofs are found in the appendix.
Guidelines:
The answer NA means that the paper does include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
14

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: While we cannot provide code for every experiment, we provide a user-friendly codebase that implements our method.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Partial Yes. We provide code, but not all datasets. Some datasets are public and can be downloaded.
Guidelines:
15

---

The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cr
www.ssssioioyynyoooo for more details. pub public/ae guwdes /tooliic/uidess/oeeuubii
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https) //nips.cc/publ httpppppllllgggg gu des Cooe/ooodSSSSnoee bubm ss onPol onPolicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental details are specified. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yss]
Justification: We mention p-value where appropriate, and throughout indicate how many samples were used. vLLM experiments were repeated until error bounds became too small to show.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We conform.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: While LLMs will certainly have societal impacts, we do not see any new or additional societal impacts to our innovation.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not release new data, and models trained specialized tasks that should pose no more risk than the base LLMs (which themselves put significant safeguards in place).
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite prior assets as requested by the owners.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, baaasettsppperssseeee paperssws thoode. com / datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
Our code is documented.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer our LLM policy (https: http///eerrppppppe cc/ Cotc///neeeeesesss...... 2025/ http://eeerrssseeeeeeeeeeeeeeeee LLM) for what should or should not be described.
20

---

A Invocation sequences
In our implementation, the aLoRA weights are activated one token after the start of the invocation sequence. For the benchmark dataset experiments, simply set the invocation sequence to be the "generation prompt" specified by the base model's chat template. For the various intrinsics experiments, most often used 3-4 token length sequence which included one-word description of the task.
The invocation sequence has several benefits. The invocation sequence can be designed to
Conform to the chat template (for instance by making the aLoRA response its own turn with specialized role).
Provide (optional) short prompt to aid the learning process.
Give the adapter weights few more tokens to process the input prior generating the output, often improving performance in practice.
In our current implementation, the invocation sequence denoting the starting point of adaptation is fixed, with the option to have variable prompt follow it prior actual generation. In principle, there no need for any consistency of input sequences so long as the point at which the adapter weights should be turned can be indicated by some means.
B LLM Transformer architecture
generic architecture is shown in Figure 9
Output Tohen
Vectore
FFNN
Decacer Block
Dacacor
Layer Norm
Fosilion Embooding
Moeked Aelf--ittnioor
Input Taken Vectors
Figure 9: generic LLM architecture.
C Proofs
We repeat the propositions here from the main text and prove them.
Proposition 3 (Proposition I (KV equivalence and aLoRA inference)). For the causal decoder-only transformers we consider, the keys and values (actually, all internal states) prior to Linvoke are identical for the base model and any aLoRA adapter model using (6). Specifically, K khase Lifawokke
Lifnnghe -1, and V bbase Litiavoke Vadapter 1sfingoke -1" Inference with the al.RA adapted model can be done causally one token time (by simply increasing in 66) iteratively) with KV cache reuse.
Radapter
Proof of Proposition KV equivalence and aLoRA inference). In the proof, for notational simplic- ity we focus on adapters to the attention blocks, but the treatment for adapters to the MLP blocks the same. By construction (6). for every token index i < favooe the adapter's weight matrices coincide with the base model's: s:
2-2 2 l, W.
W addapter W
W. Wadapter V.V.
21

---

Hence for each << fuvoke and starting with the first attention layer,
Q Qd
= 24 (W +0) - x. x.Wr = Quaan
and similarly
Ki Radapter x. W K V.adapter x. R.WV V.base
Since the transformer's subsequent layer outputs (including all MLP and layer-norm states) are deterministic functions of these Q, K, V.V.n up to i, it follows by induction on layer depth that all agree between the two models. In particular the cached
internal states for tokens I.... ....,,iivokke key- and value-matrices satisfy
K Lifinoole
x-adpterr
K-base
base
Vadapter I:fjavoke -1
yhbas 1:hhwokke -1-
L:buwoke -1
During generation at time > Linvoke both models proceed token- token-by-toker via exactly the same causal-attention mechanism ① merely appending the new (Qt, K,, V2) row and reusing the previ- ously cached rows. Since up to - 1 those rows coincide, the adapter may reuse the base model's KV cache for the first linvoke 1 tokens, and then continue to grow its own cache for tokens linvoke:-.. .., - 1. This establishes that aLoRA inference can indeed be performed causally, one token at time, with full KV cache reuse.
Proposition 4 (Proposition) 2 (aLoRA vs. LoRA inference costs)). Consider invoking an adapter with Teache tokens of input for which base model KV cache exists and Tnew < Teache input tokens without cache 12 The first token generated by the aL.RA adapter requires ( new)
operations, while LoRA requires O((Teache +T ) Furthermore, aLoRA must maintain only (TTnww) additional KV cache memory, while LoRA requires additional O(Teache + I Tnow). ). If N distinct adapters share the first Teache tokens as input, aLoRA costs become ONN new) and O(N while LoRA has O(N(TTacha + Tnew>> )2) and O (N((Tacce +T Tnew) )).
Proof of Proposition 2 (aLoRA vs. LoRA inference costs). Let the input be partitioned into Teache tokens whose base-model cache is already available, and Tnew tokens on which the adapter must act without precomputed cache. We compare the cost of generating the first new token:
aLoRA. To generate token = T Tecche + Tnew + 1, we do a forward pass of the transformer, where we have keys and values for the first Teache tokens (and therefore do not need to recompute these). The MLP blocks do not interact between tokens, so scale linearly only with Tnew + I. hidden dimension, and number of layers. For each attention layer, we
1. compute Tnow + new rows each of Q,K, V in O(Tnowd)))
2. form the attention scores by multiplying these new queries against the cached-plus-ney key-matrix of size (Teache + new ) de, costing ((Teache + (ew) d2),
apply the Tnew + 1 masked softmaxes and weighted sums over (Teache + new) values in O(Tnew (Teache + nnew) du),
form the adapter's own new cache entries, yielding at most ( extra storage.
Hence, considering hidden dimension and number of layers as constant, the dominant cost is OTTacche + Tnew (Tnew + T) O ). and extra memory O(Tnew).
LoRA. Since LoRA's rank-r updates apply to all iisens-iicluddngg those in the original context-no no prefill cache may be reused. Generating the first token thus requires recomput- ing attention over (Teache + T weww) tokens, incurring (( + Tuuw )2) O (T2 ( OT compute, and storing (T + T Tnew) key/value rows.
N distinct adapters each process the same Teache + Tnew context but disjoint Tnew segments, the
aLoRA per-adapter cost scales as ( + N T Tnow Teache)) time and O(N Tnew) memory, whereas LoRA's cost scales on the full quadratic context for each adapter (totaling O(N (Teache + Tnew )2) time and O(N (Teache + T )) memory).
We presume that Thew is the invocation sequence and following.
22

---

D Adapter Model Capacity and Increased Rank
Figure T0 illustrates this observation (described in the main text). In our experiments, rank of 32 seems to be sufficient in most cases, which is still vastly smaller than the size of the base model. Whenever comparing LoRA models, we chose LoRA rank that achieved top performance for LoRA, rather than attempting to match the ranks between LoRA and aLoRA.
context
Invocation
answer
(a) LoRA able to modify the keys (K) of task-relevant values (V) the context attention layers for the modified queries (Q) recover via the attention mechanism.
context
Invocation
answer
(b) For the context, aLoRA is not able to modify context KVs, must rely on learning modified Qs alone to extract information from existing context KVs.
Figure 10: Intuition for why aLoRA adapters often require increased rank r. ALoRA is not able to modify the keys and values of input tokens prior to the invocation sequence to send new "messages" forward to the generation step, and a too small rank constraint limits its capacity to do this.
E Inference Timing Performance Evaluation Details
In this section we document the methodology followed in the experiments leading to Figure These experiments are modeled, particularly for the smaller prompt lengths (10k tokens) after multi-turn RAG situations where collection of documents concatenated with multiple conversation turns are followed by answer of total length few hundred tokens, and then where judges are used to determine various qualities of the answer.
For each model considered, 5 low rank adapters with random weights were created for ranks 8 and 32, and independent vLLM instances were launched for each model and collection of adapters in two modalities: server where the adapters are regarded as regular LoRAs (where we used rank 8) and server where they are regarded as activated LoRAs (where we used rank 32). We note that while in our experiments such high rank discrepancies are the exception rather than the rule, adopted this setup so as to give an additional potential advantage to LoRA A in the evaluation, as lower rank update is in principle cheaper compute; we also remark that we saw very little change in the experiments compared to settign where both ranks match. For each model, the KV cache size in tokens calculated by vLLM is used to compute the batch size to by dividing the KV cache size by the length of the sequence that the LLM will be tested on (including initial prompt, generation and evaluation). A total of three batches are tested consecutively for each combination of model, number of evaluators and prompt length. Each batch is processed in sequential stages: answer generation for the entire batch is then followed by one more evaluations, also done for the entire batch.
Batches are created by choosing prompts random while ensuring that their length, after tokenization using any given model's tokenizer, is precisely the desired length. The length of the answers (256 tokens) and evaluations (16 tokens) are enforced by passing to vL..M's generate method matching minimum and maximum number of tokens to generate.
23

---

Benchmark SFT experiment details
F1
Models
Llama
3.2
IB
Instruct
Inst on Hug-
gingface), Llama
(meta-1lama/LLaaa-3 3.--amm---------trr-ee-- mmistralai/Mistrll- -
3.2
3B
Instruct
Llama3.1
8B
[9] and Mistral 7B
335
F.2 Tasks and Datasets
Here provide additional information the SFT tasks in Figure 4 each dataset on the size of the vrin/valiiatiiontesst splits, as well as the number of multiple- responses for the multiple-choice tasks. Freeform tasks have unstructured natural language strings as target output.
T information for
Table 2 provides the Huggingface path for each dataset. The URL can be recovered as https: innugiingtin face. otto/datssets/tttsooooooooo -oooosss----RRRsPPPP where PATH the name indicated in T
We next provide the task definition prompt for each of the datasets. Note that the actual datasets include this task definition, in-context-leariing (ICL) examples, and the test input as part of the full prompt to the LLM. Examples can be seen on the Huggingface pages for each dataset.
Bengali Hate Speech Classification "In this task, you are given hateful post in Bengali that ex- presses hate encourages violence towards person or a group based on the protected characteristics such as race, religion. sex, and sexual orientation. You are expected to classify the post into four classes: Religious, Political, Geopolitical and Personal depending on the topic."
WIQA: Effect Classification "In this task you will be given process, and question. The process contains a sequence of steps that happen in order. The question asks about the effect of certain event on another event. If the first event has a positive effect on the second event, answer with "for", it has negative effect, answer with "against". If there's no causal relationship between the two, answer with "none"."
MMLU Conceptual Physies MCQA "You are given question on conceptual physics. You are also given 4 answer options (associated with "A", "B", "C", "D"). out of which only one is correct. You need answer the question by selecting the correct option. You should only answer with the choice letter, not the whole answer."
MMLU College Computer Science MCQA "You are given a question on college computer science. You are also given answer options (associated with "A", "B", "C", "D"), out of which only one is correct. You need to answer the question by selecting the correct option. You should only answer with the choice letter, not the whole answer."
SocialIQA Question Generation "In this task, you'te given context and an answer. Your task is to generate the question for this answer based on the given context with commonsense reasoning about social situations."
Hindi Sentence Perturbation "Given sentence in Hindi, generate a new Hindi sentence by performing small changes on the sentence, Here, make sure that changes are semantically related and syntactically similar to the input. And the generated sentence should have high commonsense plausibility, that is to have reasonable probability of it being true."
SuperGLUE. Question Generation "In this task, you are given Wikipedia articles on range of topics, we you write a question based on the content of the articles that can be answered in binary manner i.c. True or False."

---

Type
Task
#Options
Train
Valid
Test
Multiple Choice
Bengali Hate Speech Classification
3
18. lk
227
227
WIQA: Effect Classification
3
5.2k
650
650
MMLU Conceptual Physics MCQA
142
18
18
MMLU College Computer Science MCQA
4
90
12
11
Freeform
SocialIQA Question Generation
5.2k
650
650
Hindi Sentence Perturbation
5.18k
648
647
SuperGLUE Question Generation
1.51k
190
189
Table 1:
Dataset sizes and option counts for selected tasks.
Task
Name on Lots-of-LoRAs (Huggingface)
Bengali Hate Speech Classification
WIQA: Effect Classification
MMLU Conceptual Physics
task693
MMLU College Computer Science
SocialIQA Question Generation
Hindi Sentence Perturbation
SuperGLUE Question Generation
Table 2: Task types and their Hugging Face dataset paths on Lots-of-LcRts
F.3 Training and evaluation details
Following typical LoRA SFT best practices, for both LoRA and aLoRA we used training epochs, alpha of 32, dropout of 0.05, adapted the K, Q, and V modules in all layers, and searched over ranks [6, 8, 16, 32) and learning rates [3 10 10-5,03 5,3 x 10-5, 10-1.3 10-1). Batch size of was used, with 16-bit arithmetic precision. Hyperparameters
selected by taking the configuration that performed best on the validation set, and reported performance was computed on the test set for those selected models. All training runs were done on single H100 GPUs.
All models were evaluated by comparing the generated answers golden answers provided in the " and percent correctness was used as the metric. For the multiple-choice tasks, agreement was simple evaluate. For the freeform tasks, an LLM judge prompt compared the generated answers to the golden answers and output binary decision.
G Additional details for intrinsics experiments
For the intrinsics tasks, attention weights (keys, queries, values) were adapted in all layers, using rank 32 adapters. The learning rate and number of epochs were tuned to achieve the best validation performance (as was the case for the LoRA adapters of (5). Intrinsics training tasks each used an GPU H100 node.
G.1 Uncertainty Quantification
Certainty score interpretation The returned percentages are calibrated the following sense: given set of answers assigned a certainty score of X%, approximately X% of these answers should be correct. Here "approximately" can be quantified via the expected calibration error, or ECE, Essentially what happens is teaching the adapter model what the base model knows and doesn't 't know. This inherently requires generalization
questions of wildly varying difficulty (some of which may be trick questions!) and to settings not in training. Intuitively. it does this by extrapolating based on related questions it has been evaluated on training this an inherently inexact process and leads to some hedging. Training pipeline First, probe-basee model was trained to produce calibrated certainty scores, using large diverse collection of QA datasets detailed in Appendix H.I Note that throughout, the chat template used. The procedure for this was as follows:
Note that some datasets may contain some label noise.
25

---

1. A "meta-dataset" was created containing User inputs, Answer generations (from the base model), and correctness labels for those generations.
2. For each row in the meta-datset, the base model was prompted with input of the form (User inputs, Answer generations, meta prompt), where the meta prompt was
Is the above answer correct?\n <A> Yes,
\n<B> No, \nAnswer:
and one token was generated.
3. The hidden state from the last layer of the model was saved off for the generated token. This was then combined with the correctness labels from step (1) to create dataset of (hidden states, correctness labels).
4. A 3 layer MLP was trained on the dataset of the previous step. This is known in the literature as a probe.
5. The logits of the output of this MLP held-out validation datasets were converted into probabilities, and the ECE was computed.
6. Temperature scaling was applied here to minimize the ECE, resulting in test dataset ECE of 0.02.
The above follows the procedure of [32 for freeform responses, and was applied to both the multiple choice and freeform data for consistency.
Having a calibrated probe model, we then created a teacher dataset, where all datasets were processed by the probe model and the computed probabilities were recorded and quantized in steps of 10% (05% to 95%). This teacher dataset served as the training data for the aoRR model, which was trained use the invocation sequence
< cestart_cf_rooe >ocelliittt lnd__ff_olle
and to generate the quantized percentage values.
G.2 Answerability determination
The input to the model is list of conversational turns and list of documents converted to string using apply_chat __mmmp l ate function. These turns can alternate between the user and assistant roles. The last turn is from the user. The list of documents is dictionary with text field, which contains the text of the corresponding document. To prompt the aLoRA adapter to determine answerability, a special answerability role is used to trigger this capability of the model. The role includes the keyword "answera abi lillltyyl iity:: <I ssstrt__off oo_oooorrle_e Panswer canswerabl iioiiittt y len__ff of of_roolleeeeoo >> When prompted with the above input, the model generates the answerable unanswerable output. See S for more details.
Training Details The al .oRA and LoRA adapters were fine-tuned under the following regime: rank =32, learning rate = 5e-6, number of epochs = 25, with early stopping based on validation set, and 90/10 split between training and validation.
G.3 Query Rewrite
Usage The input to the model is list of conversational turns converted to a string using apply satll_yhhtttteee seeattteppllate ate function. These turns can alternate between the user and assistant roles, and the last turm is assumed to be from the user. To prompt the aLoRA adapter to rewrite the last user turn, special rewrite role is used to trigger the rewrite capability of the model. The role includes the keyword "rewrite" followed by a short description the query rewrite task. Even though one main application for query rewrite is in RAG settings, this intrinsic can be used to rewrite user questions for other conversational use cases (e.g., to access a database, or other APIs, or tools). As such, the adapter does not need any RAG documents (that may be present in the context, in a RAG setting) and uses only the dialog turns with what is being said between the user and assistant.
Training Both the aLoRA and LoRA adapters were fine-tuned under the following regime: rank = 32, number of epochs = 25, with early stopping based on validation set, and 90/10 split between training and validation.
26

---

Evaluation data We evaluate on three different subsets of MT-RAG: a) full MT-RAG dataset (842 data points with last user turns); the non-standalone subset of MT-RAG dataset, which is subset of 260 (out of 842) last user turns that were annotated by humans as non-standalone (i.e., they are dependent on the prior context); e) the standalone subset of MT-RAG dataset, which is the complementary subset, with all the last user turns that were annotated by humans as standalone.
Answer generation quality We also evaluate answer generation quality, with top-k passages retrieved under the various query rewrite strategies for the retriever. We choose here & 20, but similar trends take place for other values of k. We used Granite Frranite3.22-81 instruct the answer generator, and RAGAS Faithfulness (RAGAS-F) and RAD-Bence score as metrics for answer quality. We use the same three testsets above.
G.4 Jailbreak Detection
Usage The input to the model is single prompt to assess for harmful content. Cur- rently, the intrinsics operate on single turn queries. To prompt the aLoRA/LoRA adapters < start_of_role >>jaibbreee len__of_olo___ >> is used.
Training Both the aLoRA and LoRA adapters were fine-tuned under the following regime: rank 32, fixed 5,000 optimization steps, 6e-5 learning rate with the Adam optimizer.
H Training datasets for intrinsics
H.1
QA datasets for Uncertainty Quantification Intrinsic
The following datasets used for calibration and/or finetuning.
BigBench
MRQA
newsqa
trivia_qa
search_qg __aa|_q______
openbookqa
web _oob_queetinss
smiles-qa
orca-math
ARC-Fasy
commonsense qa
social qa
super glue
figqa
riddle_sens
news
medmcqa
dream
codah
piqa
27

---

H.2 Training data for Query Rewrite Intrinsic
The training data contains both: 1) standalone examples, which teach the adapter to refrain from rewriting user questions that are already standalone, and 2) non-standalone examples containing diversity of patterns that are used to teach the adapter to expand the user turn so that it becomes standalone.
The training data used the publicly available Cloud corpus of technical documentation pages from MT-RAG 14 Based on this corpus of documents, a dataset was created consisting of high-quality, conversations, where the last turn of the conversation comes into versions: non- standalone version, and corresponding standalone version.
H.3 Training data for Answerability Determination Intrinsic
The training data uses the publicly available Government corpus from MT-RAG [5] as the source of documents. Based on this corpus, the dataset consists of a mix of human-created and synthetically generated multi-turn conversations. It includes the final user question can be answered based on the provided documents. These examples teach the adapter to recognize when sufficient information is present support answer. (2) Unanswerable queries, where the documents lack the necessary information to answer the final user query. Mixtral was used automatic judge validate the answerability labels and filter noisy samples.
types of examples: (1) Answerable queries, where
H.4 Training data for Jailbreak Intrinsic
The Jailbreak Intrinsic was trained on 40,000 harmful and benign samples. This is composed of several sub-sampled open source datasets (Gandalf Ignore Instructions [27]. Awesome ChatGPT
Prompts [1. BoolQ [3]. SAP [6) UltraChat [8] super natural instructions 42 as well as non-public
prompt datasets of harmful/benign content.
28


Reviews:
[{"review_id": "9x7972Po8b", "reviewer": "Reviewer_jjBF", "summary": "This paper introduces Activated LoRA (aLoRA), a novel variant of Low-Rank Adaptation (LoRA) designed for efficient multi-turn usage of large language models (LLMs).  LoRA fine-tuning updated the model weight with a low rank matrix. However, when switching between multiple LoRA adapters in a conversation or agent pipeline, one must currently recompute the model’s key-value (KV) cache for the entire context each time a new LoRA is applied during inference, which is inefficient for long conversations. To address this, the authors propose aLoRA, an adapter architecture that activates the LoRA weights only for tokens after a designated “intrinsic” invocation point. In effect, the base model processes the earlier context normally (caching its representations), and the LoRA adapter is turned on mid-stream to handle a specific sub-task. This design allows reusing the base model’s existing KV cache for all tokens before the adapter’s activation, avoiding any recomputation of the earlier context. Experiments and simple analysis are provided to corroborate the idea.", "questions": "How do you compare this aLoRA method with prompt tuning, which also seems to support KV cache reuse. In the current agent settings, a huge amount of the pipeline/workflow construction is done by prompt designing (and possibly prompt tuning). Can you think about some cases where it is hard to use prompt tuning (e.g., just append a specific prompt after the content to use a specific feature). Do you think your method can support smaller models than prompt tuning (which may mainly focus on large models)?\n\nIn a complex pipeline, one might invoke several different intrinsics in sequence or even in parallel (as evaluators). Can you elaborate on how aLoRA handles back-to-back or concurrent intrinsic calls? For example, if intrinsic A generates some text that is then used as input to intrinsic B (or back to the base model), do we simply treat that output as new input (incurring a base model pass for B’s input since base didn’t see those tokens)? In Figure 2 and your description, it seems base-model-to-intrinsic reuse is excellent, but intrinsic-to-base reuse is not supported (since aLoRA’s own generated tokens aren’t in the base’s cache).", "limitations": "I do not see a limitation section in the main paper. Maybe it is possible to include sth related to the model size for the experiments.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper tackles the real-world inefficiency of using multiple LoRA fine-tunes in one session. This dynamic adapter activation is a novel solution that directly targets the latency overhead in multi-skill LLM applications (e.g. agent systems, tool-augmented chats).\n\nActivated LoRA is, to the best of my knowledge (I am not an expert in this specific field about KV cache optimization and inference speedup), a new extension to the LoRA paradigm. Instead of static adapters that always apply to the whole input, aLoRA’s “activate on trigger” approach is original. It essentially treats adapters like callable functions at inference time, which is an innovative perspective on model modularity.\n\nThe paper’s experiments are comprehensive. It evaluates aLoRA on two fronts: (1) Inference speed in a realistic multi-step pipeline, and (2) Task performance on both standard benchmarks and new intrinsic tasks.\n\nWeakness:\n\nThe solution is somewhat tailored to scenarios where distinct task segments can be isolated in the prompt (the “intrinsic” is clearly delineated from the main conversation). This covers many use cases (the paper focuses on QA/RAG pipelines, agentic evaluators, etc.), but there are cases where a task might require full-context adaptation. For instance, if one wanted to use a LoRA to improve how the model interprets the entire user query (say for better understanding of domain-specific language), aLoRA’s approach of not adapting earlier tokens could be suboptimal.\n\nAlthough the overall trend shows parity between aLoRA and LoRA, some individual results exhibited noticeable difference. The authors claim that certain tasks had noisy outcomes due to small data sizes or non-exhaustive hyperparameter search. While they found no systematic disadvantage to aLoRA (median difference ~0%), these variations raise questions. It suggests that aLoRA may occasionally require more careful tuning (or higher rank) to match LoRA on certain tasks.\n\nThe experiments are conducted on models up to ~8–14 billion parameters (with many on 1B–7B). It is understandable due to computational constraints, but it means we do not see results on very large-scale LLMs. It is unclear if the method works well on larger scale. (Note: this is a minor issue, and should not be viewed as a concern for rejection. Even if there is no exp for larger model, the small model exps are already interesting)", "was_revised": true, "final_justification": "The authors wrote long responses, and lots of them address my concern, and make me understand better about their contribution. However, I don't feel like it is a score 5 paper at this point, but I would also like to keep the positive score. Thus, I will maintain my score unchanged, but increase my confidence slightly.", "rebuttal": "We thank the reviewer for their insightful review, appreciation of our proposed setting, and positive evaluation of the method and its originality. We respond to questions below. If anything remains unclear, let us know!\n\n> “The solution is somewhat tailored to scenarios where distinct task segments can be isolated in the prompt (the “intrinsic” is clearly delineated from the main conversation). This covers many use cases (the paper focuses on QA/RAG pipelines, agentic evaluators, etc.), but there are cases where a task might require full-context adaptation. For instance, if one wanted to use a LoRA to improve how the model interprets the entire user query (say for better understanding of domain-specific language), aLoRA’s approach of not adapting earlier tokens could be suboptimal.”\n\nRegarding interpreting an entire user query in domain-specific language – note that our benchmark experiments already involve this scenario, specifically the Hindi and Bengali based benchmarks, for which Mistral had little to no training (see relevant online materials). aLoRA matches (Hindi) and exceeds (Bengali) LoRA. We therefore do not see “unfamiliar prompt” to be a particularly difficult case for aLoRA, though more difficult regimes could exist. We also include an intrinsic (e.g. Query Rewrite) where careful understanding of the entire prior context is important to create a self-contained rewritten query. A hypothesis is that aLoRA is exploiting the fact that LLMs have evolved to provide powerful, if not task specific, representations of contexts that can be tweaked to new unseen tasks at runtime.\n\nThat said, we agree that aLoRA is likely not universally superior to LoRA in all scenarios, indeed in inference pipelines where one adapter is used exclusively and the base model never invoked, LoRA will naturally nearly always be preferred, since there is no need for cross-model cache reuse. That said, we believe we have identified a wide range of highly interesting applications where the approach will make a very meaningful impact in runtime efficiency, especially as agentic systems move towards becoming ubiquitous.\n\n> ”some individual results exhibited noticeable difference [between aLoRA and LoRA]”\n\nNote that this was only observed in the case of the small benchmarks, where (a) some of the datasets are very small and (b) we had to limit ourselves to pre-defined hyperparameter grid search. Note that for all “real-world” tuning applications where we were able to invest more in tuning the model, we do not see these differences. That said, we have already called out in the paper that aLoRA does often require higher rank and slightly more careful tuning, but we have found these downsides to be surmountable without too much effort. In the revision, we can revisit the handful of failure cases here and do a denser hyperparameter grid search to present a fuller picture.\n\n> ”It is understandable due to computational constraints, but it means we do not see results on very large-scale LLMs…this is a minor issue”\n\nThanks for raising this point (and observing that the answer is indeed our computational constraints). We tested the largest models that we could fit on our H100 GPU. We attempted multi-GPU inference with larger models, but at time of submission, vLLM v1 with both LoRA and aLoRA was failing on our system (note that vLLM v1 is in very active development). That said, we will revisit this and hope to get larger multi-GPU vLLM experiments in time for the revision. Based on everything we have seen, we have very strong reasons to believe all observed trends will continue, as prefill costs remain roughly similar in a relative sense between small and large models.\n\n> Comparison with prompt tuning\n\nThere are several prompt tuning methods in the literature. As we mention in the paper, while the most performant prompt tuning methods are generally _not_ supportive of cache reuse since they tune the prefix, tuning the suffix will preserve KV reuse in a meaningful way and may be appropriate for certain intrinsics. We note, however, that it is widely established in the literature (e.g. He, Junxian, et al. \"Towards a unified view of parameter-efficient transfer learning.\" arXiv preprint arXiv:2110.04366 (2021)) that prompt tuning methods underperform LoRA-based methods, often significantly. As a result, the innovation of aLoRA remains important to achieve the best performance for these specialized tasks in general. In the revision, we will more clearly make this point and re-emphasize that aLoRA is not the only solution that enables KV cache reuse, but rather a way to unify the accuracy advantages of weight tuning with the runtime advantages of late prompting (whether fixed or optimized). We will expand our discussion of this point in the revision.\n\n> Intrinsics in sequence and prefill\n\nYou are correct that _generated_ aLoRA tokens will have KV cache not usable by the base model or other adapters. We believe this fairly unavoidable, and is a fairly minimal downside since the aLoRA generated tokens can be simply prefilled into the base model cache after generation, if needed. We say this is a minor downside because (a) for many intrinsics, the output of the aLoRA is not needed for subsequent base model calls, (b) the generation of the aLoRA will usually be short, or at least much shorter than say an input document or base model generation, (c) prefilling short strings is very fast, much faster than generating that same string. As a result, the overall cost of generating with the aLoRA and then prefilling the result into the base model is a small extra cost proportionally to the cost of the aLoRA generation (and is often not needed).\n\nFinally, recall that _prefilled_ tokens done by an aLoRA adapter _are_ reusable by the base model and any other aLoRA. As a result, the main gains of reuse are in fact achieved in the reverse direction (intrinsic-to-base and intrinsic-to-intrinsic).\n\n> Limitations\n\nThe primary limitation that we called out in the paper is that aLoRA often needs higher rank $r$ than LoRA. As we noted in the paper, while this may expand (already small) training costs, this has no measurable downside in terms of inference costs or model accuracy. We will add this to the conclusion.\n\n**Summary**\n\nThanks again for the interesting suggestions, we hope that these clarifications and forthcoming additions will help address your concerns and merit an \"accept\" rating."}, {"review_id": "Hb4LWlITiV", "reviewer": "Reviewer_AjC2", "summary": "This paper proposes Activated LoRA (aLoRA), a method designed to significantly improve inference efficiency in scenarios where multiple LoRA adapters are dynamically invoked, such as in multi-turn agentic pipelines. The key contribution is a mechanism that allows selective application of adapter weights to only a portion of the input tokens, thereby enabling KV cache reuse and reducing redundant computation.", "questions": "1. What are the specific ranks used for LoRA and aLoRA in the supervised fine-tuning experiments? Since the rank directly impacts parameter count and memory usage, it would be useful to report this explicitly in the result tables.\n\n2. How does aLoRA integrate with other widely-used LoRA extensions such as DoRA and QLoRA? Since these methods address different aspects (e.g., quantization, dynamic scaling), understanding how they interact with aLoRA would be valuable to practitioners.", "limitations": "The limitations and potential negative societal impact are not provided.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper is clearly written and well-organized, making it easy to follow both the motivation and the technical details of the method.\n\n2. The experimental results effectively demonstrate that aLoRA substantially reduces inference time compared to standard LoRA, especially in settings with long contexts and multiple concurrent adapters.\n\nWeaknesses:\n1. Limited Novelty. While the cache reuse mechanism is practically important, the idea is conceptually straightforward. The novelty lies more in the engineering and integration than in theoretical advancement.\n\n2. The datasets used for SFT are relatively small. Demonstrating similar gains on larger-scale or more diverse instruction tuning datasets would make the findings more compelling.", "was_revised": true, "final_justification": "The paper defines a scenario that may be practical for specific use cases and presents promising results within that scope. However, the methodological innovation appears modest, and it is debatable whether the proposed use cases are too constrained for some meaningful tasks.", "rebuttal": "We thank the reviewer for their review and for acknowledging the effectiveness of our presentation and experiments. The reviewer's comments have helped us notice possible areas of improvement in our messaging, which we will address in the revision.\n\n> ”Limited Novelty. While the cache reuse mechanism is practically important, the idea is conceptually straightforward.”\n\nWe strongly emphasize that \"straightforwardness\" should not be seen as a counterargument to \"novelty.\" Indeed, the beauty of our innovation is that a straightforward, non-disruptive change can achieve such major practical runtime improvements. This is in fact an advantage that _magnifies_ the potential for impact in the field, as the \"straightforwardness\" of our approach is key to being able to integrate tightly with vLLM and other open source packages used by researchers.\n\n> ”The datasets used for SFT are relatively small.”\n\nThis is not quite true. While some of the initial benchmark datasets are small, the datasets used for the intrinsics finetuning tasks are actually large. For instance, the certainty intrinsic uses hundreds of thousands of data points, which is an extremely large dataset in the context of LoRA finetuning.\n\n> ”What are the specific ranks used for LoRA and aLoRA in the supervised fine-tuning experiments? Since the rank directly impacts parameter count and memory usage, it would be useful to report this explicitly in the result tables.”\n\nWe will add a table showing the ranks selected by grid search in the initial SFT benchmark experiment (the ranks used for the intrinsics SFT experiments are already shown). The ranks selected by grid search did not show any consistent patterns, in particular, there was no increase in the chosen rank for aLoRA overall. Even though the rank differences are not substantial, we decided to be conservative in our computational complexity experiments and chose, for the vLLM inference experiments we showed, to compare aLoRA with $r=32$ to LoRA with $r=8$. In other words, the very large speedups we show for aLoRA are with 4x the adapter parameters than the LoRA it is compared to. Our experiments strongly suggest that the adaptation rank is not contributing to the cost of generation. Indeed, it can be intuitively seen that the rank of the adapter will have almost no impact on computation and memory costs, as the adapters in all cases count for <1% of total model parameters.\n\n> ALoRA integration with QLoRA and DoRA\n\nThanks for this observation! We will add a statement that QLoRA, being simply a quantization of the base model, is easily extended to the aLoRA setting (it is however beyond the scope of the present work to include experiments). The caveat is that for KV cache reuse to apply, the base model should also be running in the same quantization.\n\nSimilarly, we can address DoRA. While in principle the same idea could be applied to DoRA, DoRA is not efficient to run as an adapter, and the standard packages recommend merging the learned weights to the base model weights before doing inference. As merging is not possible with aLoRA due to selective weight application, we don’t anticipate this to be a useful direction unless the inefficiencies of DoRA can be addressed or mitigated.\n\n> Limitations\n\nWe will add a more explicit statement in the revision.\n\n**Summary**\n\nWe hope that these clarifications and forthcoming additions will help address your concerns and merit a reconsideration of your evaluation. If concerns remain, please let us know during the discussion period!"}, {"review_id": "xV6NgS6sZr", "reviewer": "Reviewer_Sg2F", "summary": "The authors propose aLoRA, a method for reusing the key-value (KV) cache from the base model when generating tokens with LoRA adapters for specific tasks. In this approach, the KV cache is computed once using the base model for the input context and then reused across multiple LoRA adapters during generation. The paper presents both computational savings and performance evaluations to support the effectiveness of aLoRA.", "questions": "* If the aLoRA generation diverges substantially from the base model — for instance, adapting an English generation model to generate text in French — is this adaptation achievable?\n* Section 4.1 may be misleading, as it does not directly assess the performance implications of reusing the base model’s KV cache when aLoRA is applied multiple times. It seems obvious that aLoRA can be fine-tuned for a single task without sharing, simply leveraging the base model’s KV cache — similar to how prefix-LM methods work. Hence, this result alone does not fully demonstrate the claimed benefit.", "limitations": "Yes, although it doesn’t go into enough depth.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n* Significance: For certain applications, reusing the base model’s KV cache can have clear practical benefits.\n\n**Weaknesses**\n* Quality: The analysis of LoRA adapters lacks depth, leaving gaps in understanding how adapter differences (such as to the different tasks) affect the overall performance. Also, some propositions are fairly obvious and offer limited practical insight.\n* Significance: The scenario presented in the paper — using aLoRA to evaluate multiple aspects of a prompt and answer — seems rather limited. It raises the question of why the same model couldn’t generate multiple evaluations without needing separate adapters.\n* Originality: The extension of KV cache reuse to multiple LoRA adapters is an interesting addition; however sharing the KV cache between models or layers is not entirely new.", "was_revised": true, "final_justification": "The authors’ discussion did not resolve my concerns regarding the novelty of this work. I am therefore maintaining my original borderline-reject score.", "rebuttal": "We thank the reviewer for their review, and address questions below.\n\n> ”Quality: The analysis of LoRA adapters lacks depth”\n\nUnfortunately, we don’t see anything concrete in this point that we can respond to. Please be specific regarding any gaps in analysis that you would like us to address.\n\n> ”…some propositions are fairly obvious and offer limited practical insight.”\n\nWe include our propositions to formalize our claims. Their inclusion gives key practical insight into the quantitative nature of aLoRA’s advantages. We make no claims that they were “difficult” to derive and nowhere in the paper do we claim these are major contributions or even “theorems”.\n\n> ”The scenario presented in the paper — using aLoRA to evaluate multiple aspects of a prompt and answer — seems rather limited.”\n\nThere seems to be some misunderstandings here which we clarify. In particular, the “evaluation” use case is by no means a limitation, but instead a strong motivating example. Specifically,\n\n1. We note that the article includes uses for aLoRA that extend beyond evaluation; for example, the query_rewrite intrinsic is focused on full query generation, rather than only evaluation, and many of the benchmark SFT tasks are freeform. Indeed, a common use case in agents would be families of “refinement” steps where initial generations are improved and made to satisfy requirements.  In general, any well-defined task which makes sense to finetune for and will be used in concert with the base model is a potential use case for aLoRA. We will add the above discussion to the manuscript. \n\n2. The choice of the “evaluation” use case is motivated precisely because this regime is not niche but rather, the bedrock of the entire burgeoning field of LLM agents, and to a lesser extent, certain test-time-scaling reasoning pipelines. For the importance of “evaluation” and “refinement” steps in agents, see Huang, X., Liu, W., Chen, X., Wang, X., Wang, H., Lian, D., ... & Chen, E. (2024). Understanding the planning of LLM agents: A survey. arXiv preprint arXiv:2402.02716 and many others.\n\n3. We do not require multiple evaluators/intrinsics before we see meaningful gains. Note that for single evaluation settings, we do demonstrate double digit relative improvements in the cost of the evaluation task. \n\n> ”why the same model couldn’t generate multiple evaluations”\n\nWe definitely don’t require each aLoRA adapter to only implement one evaluation – the reason to limit the number of evaluations each adapter does is not a technical limitation (though multi-task adapters are harder to train), but actually modularity for user flexibility allowing for mixing and matching of skills. Either way, the significant runtime advantages of aLoRA over LoRA remain. We view this modularity-with-almost-no-downside as a selling point of aLoRA.\n\n> ”Originality: The extension of KV cache reuse to multiple LoRA adapters is an interesting addition; however sharing the KV cache between models or layers is not entirely new.”\n\nIt is unclear to us what your concern is here. If you have other references we should cite, please let us know and we will add them to the revision.\n\n> ”adapting an English generation model to generate text in French — is this adaptation achievable?”\n\nIndeed, see the SFT task results for Hindi and Bengali. Mistral 7B was not finetuned on these languages, yet aLoRA does very well.\n\n> ”Section 4.1 may be misleading, as it does not directly assess the performance implications of reusing the base model’s KV cache when aLoRA is applied multiple times.”\n\nSection 4.1 is not misleading, it does indeed directly assess the performance (accuracy) implications of reusing the base model's KV cache when aLoRA is applied multiple times. Importantly, the aLoRA architecture is extremely well-positioned for modular finetuning, i.e. multiple adapters can be tuned independently yet used together. In particular, in Section 4 we provide a collection of adapters that are meant to work together in agentic RAG scenarios, as these address pre-search, post-search and post-generation stages in RAG. Please clarify any other concerns you have here.\n\n> Limitations\n\nWe will add a more explicit statement in the revision.\n\n**Summary**\n\nWe hope that these clarifications and forthcoming additions will help address your concerns and merit a reconsideration of your evaluation. If concerns remain, please let us know during the discussion period!"}, {"review_id": "qrFkqG09Ys", "reviewer": "Reviewer_mmG2", "summary": "This paper introduces Activated LoRA (aLoRA), a novel modification of the LoRA framework designed to improve inference efficiency in dynamic, multi-turn settings. The key problem addressed is the significant computational overhead incurred when switching between standard LoRA adapters, which requires a full recomputation of the KV cache for the entire context history . aLoRA solves this by adapting model weights only for tokens that appear after a specific invocation point in the sequence. This allows the aLoRA adapter to directly reuse the KV cache generated by the base model for all preceding context, thereby eliminating the need for recalculation and enabling instantaneous switching. The authors frame these specialized, on-demand adapters as \"intrinsics\". Through extensive experiments, the paper demonstrates that aLoRA achieves substantial inference speedups while maintaining competitive accuracy compared to standard LoRA across a variety of tasks.", "questions": "The paper effectively demonstrates the utility of aLoRA within a \"late prompting\" framework. However, it is unclear how this method would handle more complex prompts where instructions and content are interleaved. For example, consider the following prompt:\n\n    `Here is my article, and I need you to polish it... [Article content]... Please use the following evaluation criteria when polishing... [Criteria content]... Now, based on the criteria above, please optimize my article sentence by sentence and provide your reasoning.`\n\n\nIn a scenario like this, which does not fit a simple content -> instruction model, where would the single invocation_sequence be placed to activate the desired aLoRA? Could the authors provide an example or elaborate on the strategy for applying aLoRA in these more complex, mixed-prompt situations?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strengths\n\n1.  Focus on real-world problems: This paper explores a key bottleneck in real-world deployments of LLM. The inefficiency of switching LoRA adapters in a agentic pipeline or multi-turn chat is an important real-world problem.\n2.  Effective and well-designed method: The proposed aLoRA method provides an effective and elegant solution to the above problems. The core idea is to reuse the key-value cache of the base model by delaying the activation of the adapter weights, which is intuitive and can directly improve efficiency.\n3.  Comprehensive experimental validation: The paper's claims are supported by comprehensive experiments. The authors provide convincing evidence of inference speed improvements for different models, demonstrating real-world advantages over  vLLM engines. The experiments show that the efficiency gains do not come at the expense of performance, and aLoRA remains competitive with standard LoRA.\n\n### Weaknesses\n\n1.  Reliance on an external judge for activation: The aLoRA framework relies on a \"invocation token sequence\" that is \"often invoked programmatically\" to signal the activation point. This implies the existence of an external controller or “judge” that knows precisely when to call a specific intrinsic function. In many real-world unstructured conversational environments, determining the right timing and appropriate intrinsic function call is a complex task in itself. This paper assumes that such external control logic is readily available, which may limit its applicability in scenarios that lack predefined operational flows.\n2.  Oversimplified Assumption of Prompt Structure: The paper's motivation heavily relies on a simplified \"late prompting\" (content -> instruction) paradigm to highlight the benefits of cache reuse. However, practical user prompts are often more complex, with interleaved instructions and content.  This oversimplification may pose challenges when applying the method outside of well-defined, structured tasks.", "was_revised": true, "final_justification": "The authors' responses addressed my concerns regarding the external controller and prompt format. I believe their responses have deepened my understanding of the article. I have no further concerns. I believe this article has practical application value, the experiments are solide, and it can be accepted by the main conference.", "rebuttal": "We thank the reviewer for their insightful questions and positive evaluation of our work, highlighting the real-world relevance and design of our method and convincingness of our experiments. We respond to questions below.\n\n> ”The aLoRA framework relies on a \"invocation token sequence\"…which may limit its applicability in scenarios that lack predefined operational flows.”\n\nThis actually has a very nice solution for the case where there is no external controller. While beyond the scope of the present work, this invocation is not hard to add to the tool-calling ability of modern LLMs by listing it to the LLM as an available tool it can call.\n\nThat said, agentic settings can always have a controller available in principle, as by definition the turns of generation are being driven by the system, not the user. Further, the existence of a controller does not mean that the flow is locked-in, any relevant information for the controller can be created bespoke by an LLM. We are intensely motivated by this agentic setting as we believe it will explode in relevance over the coming years and become more important than the classic unstructured conversational setting.\n\n> ”Assumption of Prompt Structure: The paper's motivation heavily relies on a simplified \"late prompting\" (content -> instruction) paradigm to highlight the benefits of cache reuse. However, practical user prompts are often more complex, with interleaved instructions and content.”\n\nFortunately, this imagined scenario won’t be a limitation. See our answer to your question below.\n\n> ”consider the following prompt: Here is my article, and I need you to polish it... [Article content]... Please use the following evaluation criteria when polishing... [Criteria content]... Now, based on the criteria above, please optimize my article sentence by sentence and provide your reasoning.”\n\nThank you for this question! We would like to clarify that our concept of “late prompting” is one abstraction level up from this example, we are completely agnostic to how the user chooses to present their request. Indeed, the scenario you have written exactly follows the “late prompting” framework we propose! Firstly, note the above is a single prompt. For this case, as-is there is only one model call, so no redundant prefilling, and no issues – this is neither late nor early prompting in our framework. You could imagine a follow up check to make sure the criteria are followed (either programmatic, or by the user). Here, in fact, typical user behavior is late prompting! Specifically, the user will have a follow up question “Did you follow all criteria carefully” (which is late prompting as the original generation is in context prior to this statement), followed by \"given the criteria that were missed, can you rewrite the content\". They will almost never follow the early-prompting scenario, which would be to copy-paste the prior generated article back into the context window following the “Did you follow all criteria carefully”, despite the fact that the generated context is already there on the screen as a prior generation. This showcases the point of our early/late prompting framework – it is essentially a call for agentic systems to adhere more to the human-like multi-turn chat, and abandon an observed tendency to format machine-created prompts without regard for cache reuse.\n\n**Summary** \n\nWe hope the above addresses your questions! We are happy to continue the discussion in the next phase."}]

Author rebuttal:

Meta-review: This paper introduces  Activated LoRA (aLoRA), effectively (to my understanding) allowing for "hot swapping" LoRA adapters mid processing, allowing these adapters to reuse some of the base model's work.

During the review phase, there was generally disagreement about this paper, with two reviewers highlighting empirical validation, significance, and good novelty and two reviewers suggesting "low novelty." I am recommending obvious acceptance here. I generally think this is a case where the negative reviews just don't really make any compelling arguments for rejection at all. To me, "low novelty" needs to mean "extremely similar / increlemental relative to existing prior work," to be grounds for rejection by itself, not "conceptually straightforward" as Reviewer AjC2 explicitly said their concern was.

In general, conceptually straightforward methods in machine learning that work are strictly better than the alternative, and I don't think conceptual complexity is the thing we should be hill climbing for at NeurIPS. Since there weren't really any non-superficial concerns raised by the rejecting reviewers, I'm aligning with the other two.
Decision: Accept (poster)