Joint Localization and Activation Editing for Low-Resource Fine-Tuning
Authors: Wen Lai
Alexander Fraser
Ivan Titov
Venue: ICML 2025 poster (2025)
Keywords: activation editing
low-resource
localization
Primary area: deep_learning->large_language_models
TL;DR: Activation editing approach that jointly optimizes the selection of intervention components and the intervention strategy in low-resource setting

Abstract: Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, are commonly used to adapt LLMs. However, the effectiveness of standard PEFT methods is limited in low-resource scenarios with only a few hundred examples. Recent advances in interpretability research have inspired the emergence of activation editing (or steering) techniques, which modify the activations of specific model components. Due to their extremely small parameter counts, these methods show promise for small datasets. However, their performance is highly dependent on identifying the correct modules to edit and often lacks stability across different datasets. In this paper, we propose Joint Localization and Activation Editing (JoLA), a method that jointly learns (1) which heads in the Transformer to edit (2) whether the intervention should be additive, multiplicative, or both and (3) the intervention parameters themselves - the vectors applied as additive offsets or multiplicative scalings to the head output. Through evaluations on three benchmarks spanning commonsense reasoning, natural language understanding, and natural language generation, we demonstrate that JoLA consistently outperforms existing methods.

Parameter-efffcient fine-tuning (PEFT; Han et al., 2024) methods are widely used to adapt large language models (LLMs). However, popular PEFT approaches like LoRA (Hu et al., 2021) often struggle in low-resource settings with Joint Localization and Activation Editing for Low-Resource Fine- Line-Tuning
Wen Lai Alexander Fraser 12 Ivan Titor
Abstract
only a few hundred examples. Inspired by advances in inter- pretability research (Vig et al., 2020; Zhang & Nanda, 2023), activation editing techniques (Wu al., 2024a; Yin et al.. 2024) have emerged as alternative. These methods mod- ify model activations to adapt LLMs to new tasks, leveraging the intuition that L.MMs encode many semantically meaning- ful properties in coordinate-aligned (or even disentangled) manner. The activations can then be adjusted with simple operations such as pruning, rescaling or addition. Activation editing method avoid more complex transformations, such as the MLPs used in the original Adapters (Houlsby et al., 2019). Editable components in activation editing include bias terms (Ben Zaken et al., 2022), MLP layer outputs (Wu et al., 2024a), hidden states within MLP layers (Wu et al., 2024b), and attention head outputs (Yin et al., 2024).
Parameter-efficien fine-tuning (PEFT) methods, such LoRA, are commonly used to adapt LLMs. However, the effectiveness of standard PEFT methods is limited in low-resource scenarios with only few hundred examples. Recent advances in interpretability research have inspired the emer- gence of activation editing (or steering) tech- niques, which modify the activations of specific model components. Due their extremely small parameter counts, these methods show promise for small datasets. However, their performance is highly dependent on identifying the correct mod- ules to edit and often lacks stability across differ- ent datasets. this paper, we propose Joint Local- ization and Activation Editing (JoLA), method that jointly learns (1) which heads in the Trans- former edit (2) whether the intervention should be additive, multiplicative, both and (3) the intervention parameters themselves- the vectors applied as additive offsets or multiplicative scal- ings to the head output. Through evaluations on three benchmarks spanning commonsense reason- ing, natural language understanding, and natural language generation, we demonstrate that JOLA consistently outperforms existing methods.' !
Compared to standard PEFT methods like LoRA (Hu et al., 2021), activation editing modifies significantly fewer param- eters. For example, in our experiments, the optimal LoRA configuration altered 0.826% of LLaMA-3-8B's (Dubey et al., 2024) parameters, whereas LoFIT (Yin et al., 2024) updated only 0.002%.² This drastic reduction makes acti- vation editing particularly appealing for low-resource sce- narios, where only a few hundred training examples are available.
However, activation editing's effectiveness is highly sensi- tive to the choice of modules. This selection is typically determined either by fixed hyperparameters - specifying which layers and component types to modify (Ben Zaken et al., 2022; Wu et al., 2024b) - or by additional methods that estimate the importance of different model components for a given task (Yin et al., 2024). Furthermore, existing ap- proaches vary in their intervention strategies (e.g., additive multiplicative modifications), with clear consensus which method is most effective across tasks and mod- els. As a result, performance tends to be inconsistent across different datasets and models (see Table 1 and Figure 4).
1. Introduction
"Work done during Wen Lai's visit to The University of Edin- burgh. 'Technical University of Munich Munich Center for Ma- chine Learning 'ILLC. University of Edinburgh "ILLC, University of Amsterdam. Correspondence to: Wen Lai wen.lai te", Ivan Titov <ititov www.eeff...ccc..ammmemo der.fraser @ tum.de>.
To address these limitations, we propose Joint Localiza- tion and Activation Editing (JoLA), a method that, for given task, jointly learns (1) which components edit, (2) whether to apply additive, multiplicative, or combined inter- ventions, and (3) the optimal intervention parameters the vectors applied as additive offsets or multiplicative scalings
Alexander Fraser <alexan-
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'The code for the method is released https: ///::////t// /ggghtb........ com/ wen a lavine, jola.
"Deaaledd comparisons are provided in Appendix A.

---

Joint Localization and Activation Editing for Low- Now-Resource Fine- Fine-Tuning
RED
Output
ReFT 2nd a gio & R' 2n - R-2
Joint Localization and Editing [JJLL)
glur wflnn, getin gells
Edit to Subspace
Gate S
a
Edit Vector
MLP
Additive
W
R
Multiplicative
Layer (UMig
LoFiT
g
BitFIT
A &
Wr
Bloss
6 0 gf a #0 2/00
Step 1: Solect important Heads
Layer Norm (LN)
Heatt-HHad Self Attention
Multi-Head Self Attention
A! 20
2000
100
Q
Layar Norm g. (.M
Step 2: Learning Offiet Vectirns Selected Heads
Bias
Wi
Bios
Blas
200
(1 a w()h)) ) 0 th th
Input
(a) Different Activation Editing Approaches
(b) JoLA
Figure 1. Comparison of previous representative activation editing methods with proposed JoLA. (a) includes BitFIT (Ben Zaken et al., 2022), which fine-tunes only the bias term; RED (Wu et al., 2024a) introduces scaling and bias vectors in the MLP layer; ReFT (Wu al.. 2024b), which fine-tunes the hidden layer representations; and LoFIT (Yin et al., 2024) intervenes with attention heads in two steps. (b) JoLA introduces a gating mechanism that dynamically selects and locates attention heads to modify the activation outputs. We compare activation changes (2(5.))) across modules under two interventions (additive a (f.5)) and multiplicative m ((.5), relative to the initial activation value (t()))
to modules' outputs. Rather than relying on fixed heuristics or manual selection, JoLA dynamically identifies the most relevant components and applies targeted modifications to their activations.
In summary, our contributions are as follows: (i) We intro- duce JoLA, a novel activation editing approach that jointly optimizes the selection intervention components and the intervention strategy, specifically tailored for low-resource scenarios. (ii) We demonstrate that JoLA achieves stable and consistent performance across diverse tasks, addressing key limitations of existing methods. We further validate its effectiveness across different data scales and model sizes. (iii) We provide new insights into the role of attention heads in activation editing, showing that they are the most impact- ful components for fine-tuning.
To achieve this, JoLA uses HardConerete gates with expected-LO regularization, technique previously em- ployed for parameter (Louizos et al., 2018) and component pruning (Voita et al., 2019). This method encourages spar- sity, ensuring that only small subset of components is selected for editing, thereby reducing the number of inter- ventions and, thus, the method's effective parameter count. We also observe that appears sufficient focus on heads' outputs rather than other component types, further reducing the parameter counts and enhancing the simplicity of the method. By combining additive offsets and multiplicative scalings, JoLA provides a flexible, data-efficient adaptation strategy.
2. Background
Activation editing in LLMs modifies intermediate activation outputs to steer model behavior. We categorize existing approaches into three types based on the transformation function applied to activations. Given an activation out- put (1.i) Rd for i-th component layer I, the general transformation is:
We evaluate JoLA across three benehmark categories: com- monsense reasoning, natural language understanding, and natural language generation. Experimental results on 26 tasks from the benchmarks (Hu et al., 2023; Wang et al., 2024b; Gehrmann et al., 2022) demonstrate that JoLA con- sistently outperforms existing methods in low-resource set- tings (as shown in Figure 4), delivering robust performance across various data scales and model sizes.
24 (1.1)
=(((()))))))))
(1)
where f(-) determines the intervention type:
Additive methods apply a bias vector of E Rd:
(1.1 (1.4)
2, + (1.1))
2

---

Joint Localization and Activation Editing for Low- Now-Resource Fine- Fine-Tuning
Multiplicative methods scale activations as 2
Blas Term
Attention
ml 0z where (l.+)) E R4 and @ is element- wise product.
(1.i),
60-
MLP
Attention MLP
Fliden States
Attention + Bias Term
Hybrid methods combine both transformations:
(l.i)' ml)) 24 + (!..).
(1.))
0.000
36
Existing methods follow these paradigms but often rely fixed selections of components for modification, limiting adaptability. For example, BitFit (Ben Zaken et al., 2022) updates bias terms, while RED (Wu et al., 2024a) employs per-dimension scaling vectors and bias vectors. ReFT (Wu et al., 2024b) applies fine-tuned low-rank hidden states with MLP layers, and LoFIT (Yin et al., 2024) intervenes in se- lected attention heads with additive bias vectors but requires manual selection. JOLA also modifies attention heads but unifies the processes of localization and intervention within a single framework, in contrast to LoFIT's rigid two-stage pipeline. A detailed comparative analysis between JoLA and LoFIT is provided in Appendix
30
00
25
20.0
20-
0
SIGA
WinoGrande
Law
Physics
EZE, 22G
WEB NLG
Figure 2. Performance comparison of activation editing across different Transformer modules: bias terms, MLP layers, hidden states, and attention heads.
Q1: Component Selection. We evaluate activation edit- ing across four Transformer components: bias terms, MLP layers, hidden states, and attention heads Figure 2 shows that attention heads are the most impactful component to target. Unlike other modules, which primarily refine in- termediate representations, attention heads encode key se- mantic relationships and are critical for reasoning and task adaptation (Ren et al., 2024). Interestingly, combining in- terventions across multiple components tends to degrade performance, which we interpret as a form of overfitting. These findings highlight the importance of carefully select- ing where we intervene (i.e., the choice and location of components) and having fewer interventions. Further dis- cussion is provided in Appendix C.
3. Method
In this section, we introduce JoLA, novel approach for fine-tuning LLMs in low-resource settings. We first identify two key challenges in existing approaches and present an analysis to better motivate our method (Section 3.1). We then propose a gated dynamic attention head selection mech- anism address these limitations (Section 3.2). Figure illustrates the comparison of previous activation editing ap- proaches and JOLA.
3.1. Motivation
Activation editing methods have demonstrated success in modifying Transformer components such as bias terms (Ben Zaken et al., 2022), MLP layers (Wu et al., 2024a), low-rank hidden state subspaces (Wu et al., 2024b), and specific attention heads (Yin et al., 2024). However, two critical questions remain underexplored: Q1: Which Transformer components are most crucial for effective ac- tivation editing? Q2: What combination of multiplicative and additive operations yields the best performance for in- tervention? Existing approaches predefine the components to edit and rely on fixed intervention strategies, such as sim- ple multiplicative scaling, which limits adaptability and can lead to inconsistent performance across tasks, especially in low-resource scenarios. To address these questions, we con- duct controlled experiments to compare the effectiveness of editing different Transformer components and analyze the relative contributions of multiplicative and additive opera- tions.3
Q2: Scaling vs. Offset Operations. Activation editing typically involves two operations: scaling (multiplicative) and offset (additive) adjustments. To evaluate their rela- tive importance, we conduct ablation studies isolating each operation. As shown in Figure 3, bias offsets consistently contribute more to performance improvements than scal- ing. We hypothesize that this behavior arises because the bias directly adjusts the latent representations, facilitating fine-grained task-specific adaptation while retaining the fea- tures of the pre-trained model. In contrast, scaling modifies the activations uniformly, which may introduce unintended distortions. These findings motivate our approach: JoLA incorporates both operations but prioritizes offset interven-
'Intervention on "hidden states" follows ReFT, which applies learned modifications directly to the output of the MLP sub- layer-i.e, after the nonlinearity- witthn within the transformer block. In contrast, intervention on "bias terms" follows BitFit, which fine-tunes only the existing bias parameters in the model, such as those in linear projections and layer normalization. Notably, BitFit modifies only the biases in the linear layers surrounding these mechanisms and does not introduce new parameters.
3
Detaill on experimental setup and datascts are provided in Scction 4.
3

---

Joint Localization and Activation Editing for Low-Resource Fine-Tuning
2019) and extracting reasoning paths in graph neural net- works (Schlichtkrull et al., 2021).
60
55.33
w/o Scaling Offset
$1.38
50
Both
During training, each gate is modeled as a scalar stochastic variable drawn from a Hard-Concrete distribution (Louizos et al., 2018),
50
40
10.00
21
ga (13) P(ge (..))
(1.1))) 0a
Im
(..)
P(g
(0.0). (3)
30
18:22
To clarify, these gates do not take any input each gate is simply instance of the Hard- CandrCCneeeee distribution with single learnable parameter.
20
1427
1924
1128
5.50
10
The Hard-Concrete distribution is a mixed discrete- continuous distribution over [0, 1], with point masses at O and 1 and continuous density over (0, 1). The closed-form probability of gate being non-zero (e.g.. 1 P(g (1.))
SIQA
WinoGrande
Law
Physics
E2E_ NLG
WEB_NLG
Figure 3. Comparison of the performance impact of scaling factors versus bias offsets in activation editing.
| @a (())))) )), is used define a sparsity-inducing regularizer:
Lo(o)
E(1 P(g =0 o!l.1))
tions for more effective adaptation.
(4)
3.2. Joint Localization and Editing
+1 1 P(g
(1.3) = 0 (2(l.))))
Based on insights from Section 3.1, JOLA focuses adaptive attention head interventions to maximize activation editing effectiveness. Existing methods like LoFIT (Yin et al., 2024) require manual hyperparameter tuning to select the number of attention heads and cannot adjust the chosen heads during training. Moreover, their head selection crite- ria do not necessarily align with interventions. For instance, LoFIT employs multiplicative variables to determine head selection before restarting training with additive-only inter- ventions. To address these limitations, we propose method that jointly learns which heads to modify while optimizing intervention parameters (i.e., vectors m (1.)) and al(.))).
The overall training objective balances task-specific perfor- mance with sparsity:
L(m, a, の) = Lzent (m, a) + (Lc(
(5)
where Lzent is the standard cross-entropy loss, and con- trols the trade-off between performance and sparsity. Opti- mization is performed over all intervention parameters: ø. and
As with parameter pruning (Louizos et al., 2018), the ex- pected value of each gate can be computed. The interven- tions with very low expected gate value (i.e., (Egg! < can be disregarded with no effect on JoLA performance. For the remaining heads, the gate is set during inference to its expected value, E (ig((())] and Elght+*). removing any randomness and ensuring stability in the inference stage.
extend the hybrid intervention method from Section by introducing two scalar gates, gi (I.i) and Jm (L.i) both in [0, 1].
This results in the transformation:
(1.1) = (1 11.0)) m ml.))) o 2t (L.) (1.)) a (1.1)
(2)
By dynamically selecting and adapting attention head in- terventions. JoLA achieves efficient and effective activa- tion editing, overcoming the limitations of previous meth- ods. Our approach ensures robust, data-efficient adaptation across diverse tasks, making it well-suited for low-resource settings.
where 1 E ddd is vector of ones. The transformation is de- signed so that when both gates are closed (ga (I.1) (1.3)
gm
0), it reduces to the identity map, effectively disabling the in- tervention for that head. By using separate gates, the model can learn to apply additive and multiplicative modifications independently.
4. Experiments
Since our goal apply activation editing to a small, adap- tively chosen subset of heads, we encourage the gates to be exactly zero where intervention is unnecessary. To achieve this, we use expected-Lo regularization, a technique origi- nally introduced by Louizos et al. (2018) for pruning neural network weights. This approach has since been success- fully applied to tasks such head pruning (Voita et al.,
4.1. Datasets and Tasks
We evaluate JoLA on three diverse tasks: commonsense reasoning, natural language understanding, and natural lan- guage generation. Additional details regarding the datasets can be found Appendix D.

---

Joint Localization and Activation Editing for Low- new-Resource Fine-Tuning
(a) LLaMA-3 on Reasoning Tasks
(b) LLaMA-3 on Understanding Tasks
(c) LLaMA-3 on Generation Tasks
BoolQ
Evenogy'
Cumputer Science
E2E
Heilehhing
8D
ARC-
Eeggnnering
Chematry
60
Hedtth
do
Bieiness
dBoA
MC-
Histor
idogy
WEB_V
Commenn Gen
Law
Paycholagy
PI0A
Wiroe
3004 (d) Qwen-2.5 on Reasoning Tasks
Phllesophy
Xeum (f) Qwen-2.5 on Generation Tasks
(e) Qwen-2.5 on Understanding Tasks
Bookd
Eumegis Cumputer
E2E_NLG
HellSwag
80
A
.00
Chematry
30
Hoalth
Business
20
20
GBQR
MC-+
Histor
ddog
WEB va
Camman Gen
Law
P104
VinoGranze
Math
Plysics
3104
Other Philesopyy
Xeam
LoRA
BHHFR
RED
RAPE
RAFT
LoffT
-
JALA
Figure 4. Performance comparison of JOLA and baseline methods across commonsense reasoning, natural language understanding, and natural language generation tasks for LLaMA-3 3 (Dubey et al., 2024) and Qwen-2.5 (Yang et al., 2024).
Commonsense Reasoning. For commonsense reasoning, we utilize widely adopted benchmark (Hu et al., 2023; Wu et al., 2024b) containing 8 datasets: ARC-c and ARC- e (Clark et al., 2018), BoolQ (Clark et al., 2019), Hel- laSwag (Zellers et al., 2019), OBQA (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), and Wino- Grande (Sakaguchi al., 2021). These datasets consist of multiple-choice questions, where the model must directly generate the correct option without providing explanations.
and XSum (Narayan et al., 2018) for abstractive summariza- tion of long documents. This selection ensures diverse evaluation of generation tasks, including coherence, infor- mativeness, and abstraction.
4.2. Baselines
We compare JoLA against a range of state-of- of-the-art base- lines: (1) Zero-Shot: Direct evaluation of pre-trained large language models (LLMs) without fine-tuning, including LLaMA-3 (Dubey et al., 2024) and Qwen-2.5 (Yang et al., 2024). (2) Parameter-Efficien Fine-Tuning: LoRA (Hu et al., 2021), method for efficient fine-tuning by injecting trainable low-rank updates into the model's weights. (3) Activation editing during training: BitFiT (Ben Zaken et al., 2022), a method that fine-tunes only the bias terms of the model; RED (Wu et al., 2024a), which adds scaling and bias vectors to the outputs of MLP layer; ReFT (Wu et al., 2024b), which directly intervenes on task-specific hidden states with MLP Layers, and LoFIT (Yin et al., 2024), two-stage method that selects task-relevant attention heads and applies bias tuning. (4) Activation editing during inference: RePE (Zou et al., 2023), which modifies repre-
Natural Language Understanding. We evaluate on the MMLU-Pro benchmark (Wang et al., 2024b), covering 14 domains: Biology, Business, Chemistry, Computer Science, Economics, Engineering, Health, History, Law, Math, Phi- losophy, Physics, Psychology, and Others. Each task re- quires selecting the correct answer from ten options, testing the model's 's broad knowledge and reasoning capabilities.
Natural Language Generation. For generation tasks, we select 4 datasets from GEM benchmark (Gehrmann et al., 2022), including CommonGen (Lin et al., 2020) for concept- to-sentence generation, E2E (Novikova et al., 2017) and WebNLG (Gardent et al., 2017) for data-to-text generation,
5

---

Joint Localization and Activation Editing for Low-Resource Fine-Tuning
Table Main Results: Average performance comparison (Accuracy, BLEU, ROUGE-L, BERTS of different activation editing methods across reasoning, understanding, and generation tasks for LLaMA-3.1 and Qwen-2.5 models. The best results in each category are highlighted in bold.
Llama-3.1-SBe -nStttnsrttt
Qwen2.5-78-Instruue
Reusoning
Understanding
Generation
Reusoning
Understanding
Generation
(C)
(CC
BLEU (1)
ROUGE-L(T)
BERTScore (1)
ACC (?)
(T)
BLEU (t)
ROUGE-L ())
BERTScore (1)
zena shot
53,70
40.00
12.56
77.23
78.65
37.21
14:03
34.29
78.52
LoRA
66.58
42.07
13:27
36.97
77.74
78.28
46,22
19:46
45.34
82.40
BitFa
63.05
35.02
9.25
28.81
74,83
69.25
28.72
13:47
33.10
77,89
46.19
37.33
11.24
76.24
38.76
34.75
77.52
RePE
63.61
35.54
8.49
27.61
74.30
69.85
29.15
12.19
33.07
76.98
ReFT
65,9
40.89
12.60
36.89
77.21
72.69
47.74
16.02
37.40
79.74
LoFTT
56.19
27.76
76.71
43.13
34.68
77.16
JoLA
70.55
47.00
17.07
40.65
80.54
82.40
51.57
24.00
50.23
85.90
sentations derived from contrastive prompts.
5. Results
This section evaluates the performance of our proposed method, JoLA, in comparison with various basclines. Ta- ble 1 presents the average performance for all methods across the three tasks, while Figure 4 illustrates the results for individual subtasks. More detailed numerical results can be found in Appendix G.
4.3. Implementation
We conduct experiments using the Llama-3. 1--iat....-BB-mmm 1-8B-Instruct (8B) and Qwen2.5-7B-Instie base models. Both are publicly available via the Hugging- face r To study the impact of model size, we also experiment with smaller (Llama-3.2 1.-i--33--B---ntrrece B-Innstuct, Llama- 3.2 22-3B-Instruct and larger (Llama-3.1-708 11.1--0B--nssrcc model variants. For all datasets, we sample 200 examples to sim- ulate low-resource scenarios, with further analysis of data size effects provided in Section 6. The prompt templates used in our method are also included in the Appendix E. The Hard-Concrete distribution has two associated scalar parameters: a scale parameter and a temperature parameter. Following prior work on sparsification (Voita et al., 2019; Louizos et al., 2018), we train only the scale parameter and fix the temperature to 0.33. In all bascline experiments, we observe that the choice of hyperparameters significantly affected performance across different tasks. To address this, we conduct a hyperparameter search for each method, selecting five hyperparameters and averaging the results. The final outcomes are presented in Table and Figure 4. More details on the training setup, computational resources, and hyperparameter selection process are provided in Ap- pendix F.
(7B) models as the primary
Performance of Activation-Bass Baselines. Activation editing basclines exhibit varying levels of success across tasks, but their sensitivity to hyperparameter selection and layer intervention limits their consistency. For example, BitFit (Ben Zaken et al., 2022) is quite sensitive to the place- ment of bias terms within the model. Adjusting bias terms in dropout layers or attention mechanisms results in per- formance fluctuations, particularly in low-data scenarios. Similarly, RED (Wu et al., 2024a) depends on the specific positions where scaling and bias vectors are introduced, leading to inconsistent results. RePE (Zou et al., 2023) is highly sensitive to the quality of activation representa- tions across tasks, making it challenging to generalize its performance. ReFT (Wu et al., 2024b) achieves moderate success by intervening on selected layers but faces chal- lenges in determining the optimal number and choice of layers. LoFIT (Yin et al., 2024), while effective in lever- aging task-relevant attention heads, struggles to maintain consistency across tasks.
4.4. Evaluation Metrics
Performance of LoRA. LoRA achieves noticeable improve- ments over zero-shot baselines and, somewhat surprisingly, outperforms previous activation editing methods across all tasks when its rank hyperparameter is appropriately tuned. In tasks such as natural language generation, LoRA achieves higher BLEU and ROUGE-L scores, highlighting its ability to generate coherent outputs.
We employ exact match accuracy the evaluation metric for commonsense reasoning and natural language under- standing tasks. For natural language generation, we use BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004) and BERTScore (Zhang et al., 2019) scores as implemented in the GEM benchmark (Gehrmann et al., 2022).
Performance of JOLA. Our proposed method, JoLA, con- sistently outperforms all baselines across the three tasks by significant margin. This can be attributed to JoLA's 's dy- namic gated selection mechanism. Unlike LoFIT (Yin et al.,
hhttps://huggingffen
6

---

Joint Localization and Activation Editing for Low-Resouree Fine- Tine-Tnnin
2024), which requires manual selection of attention heads, JoLA's mechanism enables the modifications to less rele- vant heads to gradually "die off" during training reducing to the heads of the base model, improving robustness and adaptability. In commonsense reasoning, JoLA achieves an average improvement of 3.97% over the best-performing baseline (LoRA) in LLaMA-3, as shown in Table 1. For natural language understanding, JoLA demonstrates consis- tent performance across diverse domains in the MMLU-Pro benchmark (Wang et al., 2024b) across all 14 subtasks as illustrated in Figure 4. In natural language generation tasks, JoLA achieves higher BLEU, ROUGE-L and BERTScore scores compared to activation- based baselines and LoRA.
(5) 0.ee
(01 LLNNA 3.1
zar shot
1eo Gate
2010 abso Gate
40
32
15
Lowe
Plyes
BLRG
aapt
mindrrndee
ESE RE
wes R.C
Figure 5. Ablation 2: Performance comparison of models with separate gating units for scaling and offset vectors versus shared gating unit.
6. Analysis
In this section, we present detailed analysis of JoLA through ablation studies (Section 6.1, Section 6.2, and Sec- tion 6.3), an exploration of gate status during training (Sec- tion 6.4), and evaluations across varying data and model sizes (Section 6.5). Unless otherwise specified, the anal- yses in this section are conducted on selected tasks, in- cluding SIQA, WinoGrande, Law, Physics, E2E_NLG, and WEB NIGG. In addition, provide a case study to better visualize the advantages of JoLA in Appendix H.
6.2. Ablation 2: Number of Gates
In Equation (2), we employ separate gating units for the scaling vector and the bias vector. To investigate the impact of this design, we compare configurations where each vec- tor has its own gate with contigurations where both vectors share a single gate. In the latter case, the heads, if selected, is always updated both in the additive and multiplicative fashion. As illustrated in Figure 5, although the shared gating configuration achieves performance improvement over the zero-shot baseline, it underperforms compared to the configuration with separate gates. This suggests that the choice of intervention should depend on what role the head plays in given task. Using independent gating units enables fine- graiegra control over each vector's contribu- tion, facilitating more precise task task-specific adjustments and preventing over-modine wer-mooification of the activation outputs.
Reasoning
Understanding
Generation
SIQA
WinoGrande
Law
Physics
ENC.NLG
WEB.NLG
MLP wif gste MLP with
50,10
$1.62
34.00
20.00
10.31
14.45
23,00
16.25
Attention weo gate
18.12
Attention with gate
66.22
58.33
40,00
46.00
15.84
24.39
Attention + MLP wlo gate
57.17
48.74
23.00
13:00
8.23
12.36
Attention - ML JP with gate
53.28
$2.07
27.00
16.00
10.42
14.83
Table Ablation 1: Impact of ML.P and Attention interventions with/without gate mechnism on model performance across tasks.
20ro eret
PA5S
75
200
567
550
JOLA
Porfomance
6.1. Ablation 1: Gate Mechanism
Dynamic gating attention head selection is central to the performance of JoLA, detailed in Section 3.2. To eval- uate its necessity, we compare models with and without the gating mechanism. As illustrated in Table 2, the gating mechanism substantially improves task performance, both when intervening in attention heads and MLP layers. We speculate that this improvement arises because certain atten- tion heads can be modified more effectively to achieve the desired behavior in a generalizable way, whereas modifying others may disrupt the model. The gating mechanism can selectively adjust the activation outputs of relevant attention heads, avoiding excessive or unnecessary edits that could harm performance. In contrast, models without this gating mechanism fail to differentiate between "editable" and less "editable" heads, resulting in performance instability.
504
www.tandd
Lae
1859 st.
56AA
Law
Phynce
Figure 6. Ablation 3: Comparison of different head selection strategies: SMP, DSP, PASS, and JoLA.
6.3. Ablation 3: Different Head Selection Strategies
Head selection is critical component of JoLAAss 's design. To evaluate whether alternative selection strategies could achieve similar outcomes, we compare JoLA with three established methods: (1) SMP (Zhang et al., 2021), which trains separate pruner to rank and identify attention heads
7

---

Joint Localization and Activation Editing for Low-Resource Fine-Tuning
that are less important for the task; (2) DSP (Li et al., 2021), which employs Gumbel-Softmax (Jang et al., 2017) to iter- atively select the top-K heads; and (3) PASS (Ding et al., 2024), which uses robust optimization to enforce determin- istic sparsity.
ity. At intermediate scales (5,000-10,,00 samples), JOLA remains competitive with slightly outperforms strong parameter-efficient fine-tuning methods such as LoRA. When scaling further to 20,000 and 100,000 examples, JoLA shows a modest performance gap relative to LoRA, which is expected given that JoLA updates significantly fewer parameters. Nonetheless, it continues to be on par with LoRA's performance. These results demonstrate that JoLA not only effective in low-resource scenarios but also scales effectively to large datasets, making it practical solution for both data-scarce and data-rich real- appli- cations.
As shown in Figure 6, JOLA outperforms these methods, es- pecially in low-resource scenarios. SMP's reliance on large datasets for training the pruner makes it ill-suited for sparse data. DSP's iterative selection process is highly sensitive to noisy gradients from small datasets, leading to unstable or incorrect selection decisions. While PASS achieves de- terministic sparsity, its regularization objective overfits to limited data distributions, resulting in suboptimal gate de- cisions. By contrast, JoLAA's 's stochastic gating mechanism effectively balances exploration and exploitation, allowing it adaptively identify important heads even in low-data settings.
Different Model Size To evaluate the scalability of JoLA with respect to model size, we test three variants: Llama- 3.2-IB-Instruct, Llama-3.2 3.2-3.B-B--Bsuueee and Alama-3.1-708 Instruct. As shown in Figure 9, JoLA consistently deliv- ers significant performance improvements across all model sizes. Notably, larger models benefit more substantially from JoLA's 's dynamic selection mechanism, as they inher- ently possess greater redundancy in attention heads. This finding highlights JoLA's 'ss scalability and effectiveness in optimizing large-scale models while maintaining robust per- formance in low-data scenarios.
6.4. Gate Status during Traning
To further investigate the behavior of the dynamic gating mechanism, we analyzed the probability of the multiplica- tive gate (gm.) and additive gate (g.) being "closed" (i.e., set to 0) during training on the OBQA dataset (Mihaylov et al., 2018). As shown in Figure 7, both gates are initially "open" " at the beginning of training (batch 1), allowing all attention heads to be editable. As training progresses, the probability of gates being "closed" increases, in that way the approach decides that these heads do not need to be modified. Inter- estingly, the multiplicative gate is more frequently "turned off' in the later stages of training. This observation supports our conclusion in Section 3.1 (Q2) that the additive gate Ja has a greater impact on final model performance. To reiter- ate, we do not deactivate any attention heads. Instead, we selectively determine where to apply interventions. When both ga and Im set to 0, the computation for that head remains identical to the original model, effectively bypass- ing intervention. By the end of training on OBQA, 86% of heads have ga- = O, and 94% have Jm = 0, reflecting strong sparsity in applied edits.
7. Related Work
Low-Resource Fine-tuning. Recent advancements in LLMs have transformed a wide range of NLP tasks (Zhao et al., 2023). However, efficiently adapting these models to diverse applications remains challenging, especially in low- resource settings. Parameter-efficient fine-tuning (PEFT) methods (Hu et al., 2021; Dettmers et al., 2024), which update small subset of parameters integrate new mod- ules (Houlsby et al., 2019), achieve performance comparable to full fine-tuning across various tasks ( (Wang et al., 2024a). Yet, mitigating overfitting in low-resource scenarios remains a key challenge. Activation editing techniques (Ben Zaken et al., 2022; Wu et al., 2024a;b; Yin et al., 2024) offer a lightweight approach model adaptation, often argued to be more data-efficient than standard PEFT methods.
6.5. Further Analysis
Pruning. Neural network pruning (Cheng et al., 2024) aims to reduce model complexity and computational demands by removing less important or redundant components. Our ap- proach builds on pruning techniques, specifically expected- Lo regularization (Louizos et al., 2018). However, rather than pruning heads, our goal is to modify a selected subset of heads while keeping the rest intact. Subnetwork pruning techniques (e.g., Frantar & Alistarh, 2023;Sun et al., 2023) seek to identify an effective subnetwork, often tailored to a specific task, domain, or language. However, their primary objective is typically to match the performance of the full model rather than to specialize it for a particular task.
Different Data Size To evaluate JoLA's robustness across different data scales, we conduct experiments us- ing both small (100-1,000) and large 1.000-100,000) training examples sampled from the SIQA (Sap et al., 2019) and WinoGrande (Sakaguchi et al., 2021) datasets. As shown in Figure 8, JOLA consistently outperforms all baselines evenn even with as few as 100 training exam- ples-
strong effectiveness in extreme low- resource settings. Performance improves steadily as the number of training examples increases from 100 to 10,000, highlighting JoLA's adaptability to varying data availabil-
8

---

Joint Localization and Activation Editing for Low-Resource Fine- Tuning
Booth
(a) Additive gate (ga)
Baith
Basch
Bassh
Batch 20
(a) Multiplicative gate (gm)
Figure 7. Gate pruning probabilities for the additive gate (ga) and multiplicative gate (gm) during training on the OBQA dataset. probability of indicates a fully closed gate for the corresponding attention head.
SIOA (Small Datal
Wino Grande (Small
1.0 Model
Model
75
JELA
66-
68
Bari-tonn 62.78
XLA
35-
15-
210 200 302 420 5ee 553
000
1000
100
201
360
400
501
668
802
2000
530 Meeerond Lan Mhiks E36 ALo Will
50A WorDeed: Lon
Molls 12, MLO vit
Bats)
WitoGranda
8B Model
7O8 Model
LodA
Zafo-bon: 62.78
Zere 11.
15-
Wozchente
Piyaa MLG
Figure Performance of JoLA across different data sizes, evalu- ated on the SIQA and WinoGrande datasets.
Figure 9. Performance comparison of JoLA different model
sizes: Llama-3,22 3.2- Instruct, Llama-3.2 Instruct,, and Llama-
Sparse Fine-tuning. Sparse finetuning (Dao et al., 2022: Thangarasa al., 2023) is technique for adapting L.LMs to specific tasks datasets while only updating small subset of the model's parameters. Our approach shares similarities with sparse fine-tuning, technique commonly used multilingual modeling (Nooralahzadeh & Sennrich, 2023; Choenni al., 2024), where languages are typically assumed be encoded modularly. Sparse fine-tuning iden- tifies specific components (e.g.. heads), fine-tunes all their parameters, and discards the others. In contrast, JoLA ad- justs the activations of selected components while keeping the rest intact. While the goal of sparse fine-tuning is often to match the performance of the full model using a smaller version, our aim not only to reduce model size but to enhance performance over the full model.
8. Conclusions
In this paper, we introduce JoLA, a novel approach to low-resource fine-tuning that jointly learns to dynamically localize the attention heads for targeted intervention and determine effective editing strategies using multiplicative scaling and/or additive bias vectors. We observe that atten- tion heads more effective than other model components in actívation editing, offering a novel perspective for fu- ture research. Extensive experiments and ablation studies demonstrate the robustness of our method in low-data set- tings and across model scales, highlighting the importance of joint component selection and activation editing.

---

Joint Localization and Activation Editing for Low-Resource Fine-Tuning
Acknowledgments
Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human I Language Technologies, Volume 1 (Long and Short Pupers), pp. 2924-2936, Minneapolis, Minnesota, June 2019. Associ- ation for Computational Linguisties. doi: 10. 18 N39- URL https: /ttttst///aaaaaattiiii /tt////aaaantotooooroooeaa org/ N19 -330/.
The work is partially supported by the Dutch National Sci- ence Foundation (NWO Vici VI.C.212.053). The authors thank Rochelle Choenni for the suggestions. The work was supported by the European Research Council (ERC) under the European Union's 's Horizon Europe research and innovation programme (grant agreement No. 101113091) and by the German Research Foundation (DFG; grant FR 2829/7-1).
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 054577, 2018.
Impact Statement
Dao, T., Chen, B., Sohoni, N. S., Desai, A., Poli, M., Gro- gan, J., Liu, A., Rao, A., Rudra, A., and Ré, C. Monarch: Expressive structured matrices for efficient and accurate training. In International Conference on Machine Learn- ing. pp. 4690 4721. PMLR, 2022.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
References
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized 1lms. Advances in Neural Information Processing Systems, 36, 2024.
Ben Zaken, E., Goldberg, Y., and Ravfogel, S. BitFit: Sim- ple parameter-efficient fine-tuning for transformer-bas based masked language-mode models. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th An- nual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers). pp. 1-9, Dublin, Ire- land, May 2022. Association for Computational Linguis- URL https: acl short.1/.
Ding, D., Jawahar, G., and Lakshmanan, L. V. S. PASS: Pruning attention heads with almost-sure sparsity tar- gets. Transactions on Machine Learning Research, 2024. ISSN 2835-88555 URL https: hatps:///penreeeee /ppereview. net/ fort/forumlae wwt.oorrmmmmsssusstttt.oo.
tics. doi: 10..8665//1120022.al-shhho www.llanttoooy...rgggooo.o
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Bisk, Y., Zellers, R., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Inter- national Conference on Machine Learning, pp. 10323- 10337. PMLR, 2023.
Cheng, H., Zhang, M., and Shi, J. Q. A survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46 2024. doi: 10. 10.110/TPAM.222444447000
Gardent, C., Shimorina, A., Narayan, S., and Perez- Beltrachini. L. Creating training corpora for NLG micro- planners. In Barzilay, R. and Kan, M.-Y. (eds.). Pro- ceedings the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pp. 179-188, Vancouver, Canada, July 2017. Associa- tion for Computational Linguistics. doi: 10.18653/v1/ 1017 URL https: //apaa/too//laathhooooyo.o/o/o/ orgg/ P P17-1017/.
Choenni, R. Shutova, E., and Garrette, D. Examining modularity in multilingual LMs via language-specialize subnetworks. In Duh, K., Gomez, H., and Bethard, S. (eds.), Findings of the Association for Computa- tional Linguistics: NAACL 2024, pp. 287-301, Mex- ico City, Mexico, June 2024. Association for Computa- tional Linguistics. doi: 10. 10.18653/v1/2224.nnmii www.://22..iiinn--naaaa naacl.
Gehrmann. S., Bhattacharjee, A., Mahendiran, A., Wang, A., Papangelis, A., Madaan, A., Memillan-majol A.. Shvets, A., Upadhyay, A., Bohnet, B., Yao, B., Wilie, B., Bhagavatula, C., You, C., Thomson, C., Garbacea, C., Wang, D., Deutsch, D., Xiong, D., Jin, D.,, Gkatzia, D., Radev, D., Clark, E.,, Durmus, E., Ladhak, F., Ginter, F., Winata, G. I., Strobelt, H., Hayashi, H., Novikova, J., Kanerva, J., Chim, J., Zhou, J., Clive, J., Maynez, J.,
21. URL https: / www.anooooogyy.o.oooooo org/2024. Sindings-naace 21/.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Burstein, J., Do- ran, C., and Solorio, T. (eds.), Proceedings of the 2019
10

---

Joint Localization and Activation Editing for Low-Resource Fine-Tuning
Sedoe, J., Juraska, J., Dhole, K., Chandu, K. R., Bel- trachini, L., P., Ribeiro, L.. F. R., Tunstall, L.,, Zhang. L.. Pushkarna, M., Creutz, M., White, M., Kale, M. S., Eddine, M. K., Daheim, N., Subramani, N., Dusek, O., Liang, P. P., Ammanamanchi, P. S., Zhu, Q. Puduppully, R., Kriz, R., Shahriyar, R., Cardenas, R., Mahamood, S., Osei, S., Cahyawijaya, S., Stajner, S., Montella, S., Jolly, S., Mille, S., Hasan, T.. Shen, T., Adewumi, T., Raunak, V., Raheja, V., Nikolaev, V., Tsai, V., Jernite, Y., Xu, Y. Sang, Y., Liu, Y., and Hou, Y. GEMv2: Mul- tilingual NLG henchmarking in single line of code. In Che, W. and Shutova, E. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing: System Demonstrations, pp. 266- 281, Abu Dhabi, UAE December 2022. Association for Computational Linguistics. doi: 10. 10.18653/1/2022. monlp-demos..e URL https: htsss///alaatthhoco org/2022. emn 1 p demos.2m222 27/.
Jang, E., Gu, S., and Poole, B. Categorical reparameter- ization with gumbel-softmax. In International Confer- ence Learning Representations, 2017. URL https: / //openreview.ct net/
Lai, W. Mesgar, M., and Fraser, A. LLMs beyond En- glish: Scaling the multilingual capability of LLMs with cross-lingual feedback. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 8186- 8213, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: (0.18653/v1/2024 findings-acle ac1.488. URL https: / accaantooooyy.. o org/ 2024. fi ndi indiiniisss nas- acl. 4888/
Li, J., Cotterell, R., and Sachan, M. Differentiable subset pruning of transformer heads. Transactions of the As- sociation for Computational Linguisties. 9:1442 1459, 2021. doi: 10.1162/tac 000433.. URL https: / aclanthology org/ 2201/.221. tacl-1.86/.
Han, Z. Gao, C., Liu, J., Zhang, J., and Zhang, S. Q Parameter-efficient fine-tuning for large models: A com- prehensive survey. arXiv preprint arXiv:2403. 14608, 2024.
Li, Z. and Arora, S. An exponential learning rate sched- ule for deep learning. arXiv preprint arXiv: 191.009910.07454. 2019.
He, J., Rungta, M., Koleczek, D., Sekhon, A., Wang, F. X., and Hasan, S. Does prompt formatting have any impact on Ilm performance? arXiv preprint arXiv:2411.1055 2024.
Lin, B. Y., Zhou, W., Shen, M., Zhou, P., Bhagavat- ula, C., Choi, Y., and Ren, X. CommonGen: A con- strained text generation challenge for generative com- monsense reasoning. In Cohn, T., He, Y., and Liu, Y. (eds.), Findings of the Association for Compula- tional Linguistics: EMNLP 2020, pp. 1823-1840, On- line, November 2020. Association for Computational Linguistics. doi: /tt////aaaatttaaaee antho ogy.ooy...oooooo......... org.oogg//00 165. URL. https: /ac//aclll// findings seriinggs-emmlp..633
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790- 2799. PMLR, 2019.
Lin, C.-Y. ROUGE: A package for automatic evalua- tion of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Asso- ciation for Computational Linguistics. URL https: / //aclanthology - org wrg/w04-1013//
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, 7., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 (0,,060, 2021.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint 1oXiv:1711.05/000 2017.
Louizos, C., Welling, M., and Kingma, D. P. Learning sparse neural networks through I0 regularization. In International Conference on Learning Representations, 2018
Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E.PP., Bing, L., Xu,
X., Poria, S., and Lee, R. LLM-adapters: An adapter fam- ily for parameter-efffcient fine-tuning of large language models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 5254-5276, Singapore, December 2023. Association for Computa- tional Linguistics. doi: (0.18653//1122233.mmll main. 319. URL ht https: /aaaaaaannnnlooyooooo/o wwwpgg....o..///// org/2023. emnlp n 319/.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? new dataset for open book question answering- In Riloff, E.,, Chiang, D.,, Hockenmaier, J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391. Brussels, Belgium, Octoher-November 2018 Association for Computational

---

Joint Localization and Activation Editing for Low-Resource Fine-Tuning
Linguisties. doi: 10. 10.186533/1/D188-12000 1260. URL https: //aclanthology wwt.hh/oolooooooo1//1110000
Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. Social IQa: Commonsense reasoning about social interactions. In Imui, K., Jiang, J., Ng. V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empir- ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLPP, pp. 4463-4473, Hong Kong, China, November 2019. Association for Compu- tational Linguistics. doi: 10. 10.18653/v1/D1191145 URL
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidje- land, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518(7540); 529- 503,53332222 2015.
Narayan, S., Cohen, S. B., and Lapata, M. Don't 't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Pro- ceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, Brussels, Belgium, Detober-November 2018. Association for Com- putational Linguistics. doi: 1/D18653//1/D11--- 1206. URL https: / ac antho logy. - org/D1 8 12006/
Schlichtkrull, M. S., De Cao, N., and Titov, I. Interpreting graph neural networks for nlp with differentiable edge masking. ICLR, 2021.
Smith, L. N. Cyclical learning rates for training neural net- works. In 2017 IEFE winter conference on applications computer vision (WACV), pp. 464-472. IEEE, 2017.
Nooralahzadeh, F. and Sennrich, R. Improving the cross-
Smith, L. N. A disciplined approach to neural net- work hyper-parameters: Part I-learning rate, batch size, momentum, and weight decay. arXiv preprint arXi::1803.09800 2018.
lingual generalisation in visual question answering. In Proceedings of the AAA/ Conference on Artificial Intelli- gence, volume 37, pp. 13419-13427, 2023.
Novikova, J., Dušek, O., and Rieser, V. The E2E dataset: New challenges for end-to-end generation. In Jokinen, K., Stede, M., De Vault, D., and Louis, A. (eds.), Pro- ceedings of the 18th Annual SIGdial Meeting on Dis- course and Dialogue, pp. 201-206, Saarbrücken,, Ger- many, August 2017. Association for Computational Lin- guistics. doi: 10. 10.18653/v1/W17--2222 URL https: //aclanthology......
Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2.06. /695, 2023.
Thangarasa, V., Gupta, A., Marshall, W., Li, T., Leong, K., DeCoste, D.,, Lie, S., and Saxena, S. Spdf: Sparse pre- training and dense fine-tuning for large language models. Uncertainty in Artificial Intelligence, pp. 2134-2146. PMLR, 2023.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J Bleu: method for automatic evaluation of machine translation. In Isabelle, P., Charniak, E., and Lin, D. (eds.), Proceed- ings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computa- tional Linguistics. doi: 10.3115/10030083 10731333 URL https: / /awlaaaaaallooy....a..aaaaaalala wwtphh/wwoooo...oooooo////// 1040/.
Vig, J.. Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Sakenis, S., Huang, J., Singer, Y., and Shieber, S. Causal mediation analysis for interpreting neural nlp: The case of gender bias. arXiv preprint arXiv:2004. 12265, 2020.
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., and Titov, I. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Ko- rhonen, A., Traum, D., and Marquez, L. (eds.), Pro- ceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pp. 5797-5808, Florence, Italy, July 2019. Association for Computa- tional Linguistics. doi: 10. 11118663/1/P11--5580 URL https://aclantee
Ren, J., Guo, Q., Yan, H., Liu, D., Zhang, Q., Qiu, X., and Lin, D. Identifying semantic induction heads to understand in-context learning. In Ku, L.WW., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 6916- 6932, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10..8853/11/224 findings-acL4122 URL https:/ aaclanaooonooyy.... org/2024. f nd waw.ddl.ssccc1.//// /
Wang. L., Chen, S., Jiang, L., Pan, S., Cai, R., Yang, S., and Yang, F. Parameter-efficien fine-tuning in large models: A survey of methodologies. arXiv preprint arXiv:2410. 19878, 2024a.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. Mmlu-pro:
12

---

Joint Localization and Activation Editing for Low-Resource Fine- Tunn-n
A more robust and challenging multi-task language under- standing benchmark. arXiv preprint arXiv:2406.01574, 2024b.
Wu, M., Liu, W., Wang, X., Li, T., Lv, C., Ling, Z., Jian- Hao, Z.,, Zhang, C., Zheng, X., and Huang, X. Advane- ing parameter efficiency in fine-tuning via representa- tion editing. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 13445-13464, Bangkok, Thailand, August 2024a. Association for Computational Linguis- tics. doi: 10. 10.186633///222244cllooog 726. URL https: / //aac/Lmtthooogyyorr////2222 acl long. 726/.
Wu, Z., Arora, A., Wang. Z., Geiger, A., Jurafsky, D., Man- ning, C. D., and Potts, C. Reft: Representation finetuning for language models. arXiv preprint 0345522404.03592, 2024b.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., al. Qwen2. 5 technical report. arXiv preprint arXiv:241.115115, 2024.
Yin, F.. Ye, X., and Durrett, G. Lofit: Localized one-tuning IIm representations. arXiv preprint arXiv:2406.0/56h 2024.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can machine really finish your sen- tence? In Korhonen, A., Traum, D., and Märquez, L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791- 4800, Florence, Italy, July 2019. Association for Compu- tational Linguisties. doi: 10. 10.18663/v1/111-1472 URL https: hatppp////claaahoooog....ooooo/ /tg/P19-1/172..
Zhang, F. and Nanda, N. Towards best practices of activation patching in language models: Metrics and methods. arXiv preprint arX9v:2309. 16042, 2023.
Zhang. T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint ar arXiv:1904.09677 2019.
Zhang, Z., Qi, F. Liu, Z., Liu, Q., and Sun, M. Know what you don't need: Single-shot meta-pruning for attention heads. Al Open, 2:36 42, 2021.
Zhao, W. X., Zhou, K., Li, Tang, T., Wang, X., Hou, Y.,
Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., et al. Representation engineering: top-down approach to ai transparency. arXiv preprint arXiv:2310.0/44 2023.
13

---

Joint Localization and Activation Editing for Low-Resource Fine-Tuning
A. Comparison with Full Parameter Fine- Fine-TTuning, PEFT and Traditional Activation Editing
To better demonstrate the advantages of our approach over full-parameter fine-tuning, LoRA (Hu et al., 2021), and existing activation editing methods, we compare them across five key dimensions: the percentage modified parameters (both trainable and active), intervention modules, dynamic localization of interventions, data efficiency, and robustness across diverse tasks. Using fine-tuning of LLaMA-3-8B (Dubey al., 2024) representative case, summarize the differences in Table 3.
Number of Parameters. We distinguish between trainable parameters- those updated during training- and active parameters- those used during inference. Full fine-tuning updates and utilizes all model parameters, resulting in 100% trainable and active parameters. LoRA introduces low-rank adapters into each weight matrix. For LLaMA-3-85 with rank 8, this corresponds approximately 0.2605% of the parameters being both trainable and active (Hu et al., 2021).
For activation editing, we take LoFIT and JoLA as examples. In JoLA, trainable parameters include: (1) Multiplicative scaling vectors (l.)) and additive bias vectors a (1.5) for each attention head; (2) HardConcrete gate parameters om and 6a that determine head selection. All of these are optimized during training. However, due to Lo regularization, most gates are pushed toward zero, effectively pruning the majority of heads. At inference, only the heads with non-zero expected gate values remain active, and only their associated (4.)) and (!..3) are applied. contrast, LoFIT pre-selects a fixed subset of attention heads in two-stage process: (1) Training all heads, then (2) Fine-tuning only selected subset.
The number of trainable parameters can be approximated as:
Daun x (Nmuti + Nadd + N (Vate)
P Pariinable
(6)
PLAMS
where Dam is the dimension of each attention head, Numutii Naad and N Yaue are the numbers of multiplicative, additive, and gating parameters, respectively, and PLIMS is the total number of parameters in the base LLM.
JoLA and LoFIT exhibit similar numbers of active parameters at inference. Minor variations across tasks are expected: JoLA selects heads dynamically based on the input, while LoFIT uses a fixed, manually defined subset.
Data Efficiency.
Activation editing methods modify only small fraction of the model's representational capacity, enabling strong performance in low-resource settings. LoRA performs competitively with as few as 1,000 training examples across various NLP benchmarks (Hu et al., 2021). For detailed comparison of data efficiency between LoRA and JoLA, see Figure 8.
Robustness. Performance degradation due sensitivity to hyperparameters or intervention configuration is common concern. LoRA is relatively robust, its low-rank formulation transfers well across domains, though it still requires manual tuning of the rank parameter. Traditional activation editing (e.g.. LoFIT) depends on manual head selection and tuming, and shows high variance-up to +5% accuracy-acros across datasets. JoLA addresses this by eliminating manual gate thresholding. Instead, it optimizes HardConcrete parameters end-to-end, allowing the model to dynamically select relevant heads per input or task. This results in consistently stable performance across diverse benchmarks (Table 1). We report baseline hyperparameter sensitivity results in Appendix F.3.
Intervention Granularity. The location and granularity of interventions affect both expressiveness and computational cost. We empirically compare different intervention components Section 3.1 (Q1) and observe that attention heads are particularly crucial for task adaptation relative to other components. This insight directly motivates the architectural design of JoL.A
14

---

Joint Localization and Activation Editing for Low-Resource Fine-Tuning
Table Comparison of full parameter tuning, LoRA, and activation editing methods. We compare the intervention components, the pereentage of new parameters introduced, the data efficiency, and robustness across different tasks. We use LLaMA-3 (Dubey et al., 2024)
to compute the parameters. Note that full parameter tuning" does not introduce new parameters.
Trainahle (%)
Active Params (%)
Intervention
Dynamic Localization?
Efficientt?
Full Tuning"
100%
100%
No
LoRA (Hlu 2021)
0.26055
0.2605%
No
No
BaFi (Ben Zaken 2022)
0.0800%
0,0800%%
Bius Term
No
No
No
RED (Wu et al., 2024a)
0.0040%
0.0040%
MIP Layer
No
Yes
No
ReFT (Wu al., 2024b)
0.0300%
0.03000%
Hidden Representation
Yes
No
LoFIT (Yin al., 2024)
0.0002%
Attention
JoLA
0.0065%
0.0002%
Attention
Yes
Table 4. Comprehensive comparison between LoFIT and JoLA.
Aspect
LOFIT
JoLA
Localization
Two-stage process: (1) Select heads via learning multiplicative interventions; (2) Discard scaling, freeze heads, and train additive bias vectors.
Joint optimization: dynamically selects heads while learning interventions.
Intervention Type
Multiplicative and additive, but seperately.
Hybrid: additive biases multiplicative scaling via adaptive gating.
Sparsity Control
L1 regularization on scaling factors; top-K head selection.
Hard Concrete gates with expected-LO regulariza- tion; differentiable pruning.
Flexibility
Fixed intervention type (bias) post-localization
Learns task-specific intervention type per head.
B. Comparative Analysis between JoLA and LoFIT
To further contextualize the contributions of JoLA, we provide a detailed comparison with LoFIT (Yin et al., 2024), focusing on three key dimensions: methodology, intervention formulation, and empirical performance.
Methodological Comparison. A central distinction between JoLA and LoFIT lies in their treatment of localization and intervention optimization. While LoFTT employs a two-stage pipeline, JoLA integrates both processes into a unified end-to-end framework. This design allows JoLA to dynamically adapt head selections and intervention types based on downstream task requirements, thereby avoiding the suboptimal decoupling present in LoFIT. We present the difference between these two methods in Table 4.
Formula-Level Comparison. At the operational level, JoLA generalizes LoFIT's additive-only formulation by enabling hybrid interventions per head. The following equations illustrate the key differences:
LOFIT (Additive bias):
(1.0) 2t t (1.0)
21,0 + (1.6)
This static form is limited to linear shifts of activations and may not suffice for nuanced task demands requiring amplification or suppression.
JoLAA (Hybrid intervention):
(1.8) +- (1 + g(l..) m (1.0) Oz2fi (1.0)
+ (1.6) (.0
(8)
Scaling
Bias
- This hybrid approach allows both multiplicative and additive adjustments to token-level activations, providing more expressive control over model behavior. Task-specific gating (g1h.) (1.1) (1.0) enables adaptive modulation per head.
ga
15

---

Joint Localization and Activation Editing for Low-Resouree Fine- Tuning
SIgA
WinsGranda
Physics
126, NLG
WEB. NLC
22
52
12
Atlention
Atsention
Figure 10. Performance comparison of interventions across different Transformer components and training sample sizes.
Empirical Observations. JoLA demonstrates robust performance across 26 NLP tasks, particularly under low-resource settings. addition to improved accuracy, JoLA is more parameter-efficien than LoFTT, due shared head/intervention parameters and learned gating mechanism that selects to not edit the unnecessary heads.
C. Additional Analysis on Component Selection
To further investigate the impact of intervening on different Transformer components, we evaluate how performance scales across various data sizes (in low-resource settings) and component combinations.
Degradation from Combining Interventions. As mentioned in Section 3.1 (Q1), combining interventions interventionssppartiullalyy particularly between attention heads and MI.Ps tends to degrade performance. This is not simply a consequence of limited data, As shown in Figure 10, even with larger training sets (up to 500 examples), the combined Attention+MLP interventions underperform compared using attention alone with fewer examples (e.g.. 200). This pattern suggests form of overfitting representational interference when editing multiple components simultancously. While MLP layers contribute to intermediate representation refinement, their effects do not appear to be complementary when naively combined with attention-level interventions.
Performance Scaling with Sample Size. We also examine how performance changes as the amount of training data increases. While attention-only interventions benefit slightly from more data (as expected), the gap between attention and MLP-only interventions remains consistently large across sample sizes. More notably, increasing data for the combined MLP+Attention configuration fails to close this gap, further emphasizing the importance of careful component selection rather than relying brute-force data scaling.
These findings reinforce our conclusion:
while multiple Transformer components are modifiable, not all interventions contribute equally. Moreover, indiscriminately editing more components can be detrimental. Attention heads remain the most effective target for eetivation-based editing, and intervention strategies should prioritize precision over breadth.
D. Datasets
We conduct experiments across three tasks: commonsense reasoning (Hu et al., 2023), natural language understanding (Wang al., 2024b), and natural language generation (Gehrmann et al., 2022). Table 5 provides brief overview of the sub-datasets or sub-tasks within the three benchmarks evaluated. The commonsense reasoning task is framed as a multiple-choie problem, where the correct answer is selected from to 4 possible options. The natural language understanding task also follows a multiple-choice format, but with ten options. The natural language generation task, on the other hand, is an end-to-end text generation task, where unstructured data (such as commonsense concepts or data) is converted into coherent text. In the training phase, we simulate a low-resource scenario by using 200 examples. Section 6.5 further explores experiments with varying numbers of samples. To ensure consistency across experiments, used the same random seed (seed= 42) for data sampling, ensuring identical training samples in all runs.
E. Prompt Setting
Recent studies (He et al., 2024: Lai et al., 2024) have highlighted the substantial impact of prompt design on model performance. In our experiments, we adopt the same prompt configurations as Hu et (2023) for the commonsense
16

---

Joint Localization and Activation Editing for Low-Resource Fine- Fine-Tuning
Table Overview of the sub-datasets and sub-tasks evaluated three main tasks: commonsense reasoning, natural language under- standing, and natural language generation. Each task is designed to assess different aspects of language processing, with commonsense reasoning and natural language understanding framed as multiple-choice problems, and natural language generation as an end-to-end text generation task.
Task
Dataset
Description
Label
ARC-c
Designed challenge co-nccurrence methods, similar ARC-e more complex.
answer wetoverr//anssee /tww://///wwweeeeeeeeeeeeeeee Nanswerd
ARC.
Authentic grade school level multiple choice questions.
Commonsense
BoolQ
A daluset for unswering naturally occurring no questions.
true/false
Reasoning
(Hu et al., 2023)
HellaSwag
Select the most appropriate ending or sentence completion given a context.
endingl //wee/ingg/eddinn/////mm
OBQA
open-book book QA dataset requiring
knowlodge.
PIQA
Focuses physical commonsensse reasoning.
solution//sclution
SIQA
Involves reasoning about human actions and their social consequences.
answer wawwel//anwwe2//nsswwe
WinoGrande
Fill-in-the-Be blank task with binary options within sentence.
option1/option /option2
Biolngy
Dusiness
Chemistry
Computer Science
Ecomomics
Engineering
option Diiiiimlloopoonzollnnzzzzziiii optioas/oosiinni/oolaa///////// NNoptoon/ optionnioption10
MMLU-PP Pro
Ilealth
question-answering task spanning 14 domains, primarily from the MMLU benchmark (Hendrycks al., 2020), with additional examples from STEM resources".
(Wang et al. 2024bb
History
Law
Math
Other
Philosophy
Physics
Psychology
Common. Gea
Converts concepts into cobereat sentences.
GEM
E2E_NIg
Transforms structured natural language
No label.
(Gehrmann et al., 2022)
eod-to-cnd text generation
Web. Web.Nig
Generates text from structured data inputs.
Xsum
Performs ahstractive cummarization of documents.
Table Prompt settings are employed across various benchmarks, including Commonsense Reasoning, MML.U-ro,, GEM.
Benchmark
Task
Prompt
Commonsense Reasoning (Hu et al., 2023)
all tasks
Please choose the correct answer to the question:(questionn) \nin Optionl: (pppiioonn...ppooo///ooooooo.oe
Answer format: Option1/...\Optioon
MMLU-Pro (Wang et al., 2024b)
all 14 domains
The following are multiple choice questions (with answers) about [domain]. Please return the answer in the format of [Thh answer is (X) at the end. Question: (Question on) Options: A. (opt ion).. B. (opticntl (optionn)
Common Gen
Ignoring the order the concepts: [concepts]; \nGenerate a sentence with the concepts.
GEM
(Gehrmann al., 2022)
E2E NLG
Please generate a restaurant description from the information given be- (ow:)/n),n(ddtaa)
WEB.NLG
Take the following triple set part of Data-to-Text task: {data). }. Make lexicaliza- tion of the triple set into plain text.
Xsum
First, please read the article below celow.nn/nfarticle. nnnww can you write me
extremely short abstract for it?
17

---

Joint Localization and Activation Editing for Low-Resource Fine-Tuning
Table Hyperparameter configurations for the baseline methods evaluated in our experiments. These settings used multiple tasks to assess model performance in low-resource settings, discussed in Section and Section 4.
Baseline
Hyperparameter
Values
Bias Moudule
bias of Q.K and V from attention/bias of LayerNorm from attention outputs/bias of LayerNorm from hidden outputs
BitFit (Ben Zaken et al., 2022)
Learning Rate
le-4/ Se-4
Rank
8/
RED (Wu al., 2024a)
learning Rate
5e-5/ 2e-4 / 6e-2
REPE (Zou al., 2023)
method
Representation Reading Representation Control
Prefix + suffix posotion
+7 s7 / +s11
ReFT (Wu al., 2024b)
Layers
all /40144.1,22,44188222223991111
number of attention heads
32/64/128
LoFIT (Yin et al., 2024)
Learning Rate
5c-4 Se-3
reasoning benchmark, and used the prompts from the original paper for the MMLU-Pro benchmark (Wang et al., 2024b). For the GEM benchmark (Gehrmann al., 2022), where the original paper did not provide the prompt settings, we utilized commonly used prompts curated from PromptSource'. To ensure reproducibility of our results, we present the prompts employed in experiments in Table 6.
Table 8, Performance comparison of different learning rate (LR) schedules across six tasks for both JoLA and LoFIT models.
JoLA
LoFIT
SIQA
WinoCirande
Law
Physics
B2R_NIG
WEB_NLG
SIQA
WinoCirande
Law
Physics
E2R_NLG
WHH_NLG
Linear
62.71
56.49
38.00
42.00
14.05
22.83
54.13
53.36
35.00
6,00
13.84
16.95
Cycle
64,25
57.26
39.00
43.00
14.37
23.44
54.32
54.25
34.00
6,00
14.37
17.83
Adaptive
65.47
58.60
39.00
44.00
15.02
23.86
55.18
55.57
36.00
7,00
15.24
17.64
Exponential
66.22
58.33
40.00
46.00
15.54
24.39
55.94
55.33
36.00
7,00
14.77
18.12
F. Experiment Configurations
F.I Training Setup
We conduct all experiments using the HuggingFace Transformers library and fine-tuned the models with the TRL toolkit". The AdamW optimizer (Loshchilov, 2017) was used for fine-tuning, with c = 1e 6 and one epoch of warm-up. Given the small dataset (e-g., 200 samples our setting), overfitting was a concern. To mitigate overfitting's impact on the baseline, we introduced early stopping, which was not applied the original implementation of the baseline systems. We also found that learning rate adjustment significantly affected the results. To optimize the learning rate strategy, we evaluated four strategies: (1) linear schedule (Mnih et al., 2015), (2) Cyelic Learning Rate Schedule (Smith, 2017), (3) Adaptive Heuristic Schedule (Smith, 201 8), and (4) Exponential Decay Schedule (Li & Arora, 2019). As shown in Table 8, the exponential decay strategy proved to be the most stable, so we used it for both the baseline and our method. The exponentially decaying learning rate schedule defined by the following formula:
Ir(t) Iro X decay-!
where Itg is the initial learning rate Iroset to x 10-4, is 0.1, and the decay rate is 0.01.
For the gating units, we used a temperature of 0.33 in the Gumbel Softmax (Jang al., 2017). Fine-tuning was performed in full precision for the 7B, 8B, IB, and 3B models, while for the 70B model, applied 4-bit quantization enable
7ntpps: :ttpt:::////httu com http::/www......eeoooooooooootoo "https: //gi thub. com http.///ggggggaac///e/aaaaa/aaae Phtpps:// //gi thub. com wwt.c/hggiiggaacc///rtr.
18

---

Joint Localization and Activation Editing for Low-Resource Fine-Tuning
Different Learning RED
(b)
Suffix Positions ReFT
Different Attention in LoFIT
70
5e-5
P7
+7
60
32
2e-4
64
6c:
50-
-A- 128
50
Score
40
Score
30
30
20
20
20
10
10-
10
WinoSrande
Low
Prysics
EZE RaG
Web 18.
SlGk
Winddrando
Law
Physice
E2E ta.
Web NLG
SI0A
WindGrande
Law
Physes
ESE Nus
Web KLG
Figure 11. Performance comparison across six tasks under different experimental settings. The three subplots illustrate the sensitive of various configurations on task performance: (a) Different learning rates in RED (5c-5, 2c-4, 6c-2), (b) Different prefix and suffix positions ReFT (P7 S7, P11 S11), and (c) Different numbers of attention heads in LoFIT (32, 64, 128).
single-precision fine-tuning.
F.2 Computational Resources
All experiments for the 1B, 3B, 8B, and 13B models were conducted on single NVIDIA A100 80GB GPU server. The 70B model, described in Section 6.5, was evaluated on an NVIDIA H100 94GB GPU server. As an example, with the 8B LLaMA-3 model, JoLA converged within GPU hours on most tasks in the low-resource setting, using only 200 training samples.
F.3 Hyperparameter Search for Baselines
As discussed in Section 1 and Section 4, the performance baseline methods low-resource settings is highly sensitive to hyperparameters across different tasks. We present Figure 11 the sensitivity of hyperparameters in baseline methods, including the effects of varying learning rates in RED (Wu al., 2024a), different prefix and sufñx positions in ReFT (Wu al., 2024b), and the number of attention heads in LoFIT (Yin al., 2024). However, it is impractical conduct hyperparameter searches for each task individually, given that evaluate 26 tasks in total, and performing separate search for each would be time-consuming. To mitigate this, we perform hyperparameter selection using grid search approach. For each task, we run grid search with five different hyperparameter configurations, which are chosen to explore a diverse range of parameter settings that could provide the best model performance. We performed this search over key hyperparameters, as presented Table 7, using a validation set to select the configuration that resulted in the best performance. The final model evaluated with these hyperparameters, and we averaged the results across all tasks, as reported in Table and Figure 4.
G. Full Results Across all Tasks
Due to page limitations, we present the average performance across the 26 tasks in Table and Figure 4. In this section, we provide detailed performance metrics for each individual task. Specifically, Table 9 reports the accuracy of LLLMA-33 on the commonsense reasoning task, while Table 10 presents the accuracy of Qwen-2.5 on the same task. Table 11 shows the accuracy of LLaMA-3 on the natural language understanding task, and Table 12 shows the corresponding accuracy for Qwen-2.5, Finally, Table 13 presents the BLEU, ROUGE-L. and BERTScore for LLaMA-3 on the natural language generation task, with Table 14 displaying the corresponding metries for Qwen-2.5.
H. Case Study
To provide an intuitive evaluation of the advantages of our method, we select one representative case from each of the tasks: commonsense reasoning, natural language understanding, and natural language generation. The results generated by the baseline and our approach are presented below.
19

---

Joint Localization and Activation Editing for Low-Resource Fine- Fine-Tuning
Table 9. The accuracy of LLaMA-3 across various commonsense reasoning tasks, comparing different baseline methods and our proposed method (JoLA).
ARC-c
ARC-e
BoolQ
HellaSwag
OBQA
PIQA
SIQA
WinoGrande
# AVG
zero shot
59.56
65.40
41.99
45.19
54.80
76.01
42.78
43.88
53.70
LoRA (Hu al., 2021)
70.13
77.85
56.37
66.18
73.38
71.36
63.42
53.97
66.58
BitFit (Ben Zaken al., 2022)
64.17
72.35
49.69
63,48
74.07
71.24
58.14
51.28
63.05
RED (Wu et al., 2024a)
39.67
56.20
9.69
40.81
50.75
70.09
50.46
51.84
46.19
RePE (Zou et al., 2023)
61.34
74.07
53.41
60.32
76.06
73.18
60.53
49.95
63.61
ReFT (Wu al., 2024b)
66.36
77.37
53.34
63.96
75.43
73.50
62.27
55.36
65.95
LoFIT (Yin et al., 2024)
57.10
78.59
43.69
42.01
61.73
53.96
56.37
56.10
56.19
JoLA
74.66
80.13
62.17
70.69
76.20
76.01
66.22
58,33
70.55
Table 10. The accuracy of Qwen-2.5 across various commonsense reasoning tasks, comparing different baseline methods and our proposed method (JoLA).
ARC-c
ARC-e
BoolQ
HellaSwag
OBQA
PIQA
SIQA
WinoGrande
AVG
zero shot
88.14
94.70
55.87
82.42
81.80
87.38
76.56
62.35
78.65
LoRA (Hu al., 2021)
85.30
92.04
66.20
83.30
82.09
84.53
71.25
61.53
78.28
BitFit (Ben Zaken et al., 2022)
75.73
85.78
53.05
75.37
70.08
78.63
66.08
49.23
69.25
RED (Wu al., 2024a)
84.23
86.72
55.74
75.09
73.05
79.00
67.28
$1.08
71.52
RePE (Zou et al., 2023)
78.90
83.51
55.49
74.18
68.38
81.45
63.20
53.72
69.85
ReFT (Wu al., 2024b)
79.29
87.57
58.88
77.72
71.96
82.41
69.66
54,01
72.69
LoFIT (Yin ct al., 2024)
78.32
84.25
54.14
75.00
72.53
79.27
66.60
49,30
69.93
JoLA
88.31
95.29
68.10
88,53
86.40
87.05
75.79
69.69
82.40
Table 11. The performance of LLaMA-3 across multiple domains in the MMLU-PP benchmark.
Biology
Business
Chemistry
Computer Science
Economics
Engineering
Health
Hlistory
Law
Math
Other
l'illosohyy
Playicss
Psychology
#AVG
LoRA
65
35
30
47
40
54
36
31
36
53
42
BirFit
62
31
30
19
23
20
45
53
35
RePF
67
30
30
56
26
24
35
28
55
36
ReFT
71
31
35
57
26
55
46
29
41
JoLAA
70
42
34
43
53
40
37
39
46
62
Table 12. The performance of Qwen-2.5 across multiple domains the MMLU-Pro benchmark.
Biology
Business
Chemistry
Computer Science
Rennomies
Engincering
Health
History
Law
Math
Philosophy
Physie
Psychology
BAYG
zeroLahot
71
36
17
27
48
19
64
LoRA
68
32
45
58
36
48
34
40)
33
40
50
74
46
BlrFit
26
23
28
RePE
57
32
20
22
42
23
17
14
21
3-
22
27
68
29
ReFT
JoLA
75
41
39
49
62
38
56
$7
42
45
42
45
55
76
52
20

---

Joint Localization and Activation Editing for Low-Resource Fine-Tuning
Table 13.
The performance of ..A.M various natural language generation tasks (Commen_ E2E_NIG, WEB_NIG, and
Xsum), using BLEU, ROUGE-L, L., and BERTScore as evaluation metrics.
Commen Gen
E2ENNL
WEB.NIA
Xsum
B..FU
Rouge-I.
BertScore
BLAU
Rouge-I.
BertScore
BLEU
Rouge-I.
BertScore
B.EU
Rouge-I.
BertScore
zero shot
16.19
46.59
79.69
8.26
27.47
74.10
21.65
52.11
83.79
4.14
20.65
71.35
LoRA
18.17
49.54
81.15
13.15
39.75
77.50
19.53
34.50
82.18
2.25
24.11
70.12
BitFit
13.16
31.02
77.51
9.25
31.28
74.77
12.25
40.25
76.86
2.35
12.68
70.19
RED
17.19
45.41
80.43
10.31
30.44
75.51
14.45
42.62
78.43
2.99
11.14
70.60
RePE
11.24
30.15
76.15
8.12
25.46
74.01
12.36
42.36
76.94
2.25
12.46
70.12
ReFT
20.22
48.26
82.69
12.60
32.71
77.11
13.09
43.36
77.46
4.49
23.22
71.58
LoFIT
12.17
30.53
76.81
14.77
38.88
78.66
18.12
46,38
81.11
2.47
12.57
70.26
Our
23.13
53.47
84,93
15.54
42.52
79.22
24.39
38.09
85.92
5.24
28.50
72.07
Table 14. The performance of Qwen-2.5 across various natural language generation tasks (Commen Gen, NLG,, WEB NLG, and Xsum), using BLEU, ROUGE-L, and BERTScore as evaluation metrics.
Commen Gen
E2E NLG
WEB NLG
Xsum
BLEU
Rouge-L
BertScore
BLEU
Rouge-L
BertScore
BLEU
Rouge-L
BertScore
BLEU
Rouge-L
BertScore
zero_shor shot
14.58
41.85
78.53
8.08
25.63
73.97
31.13
$
91.40
2.32
13.59
70.17
LoRA
17.16
52.39
80.40
23.39
46.65
85.14
31.00
55.50
91.30
6.29
26.84
72.77
BitFit
14.92
40.16
78.77
15.25
35.03
79.01
21.49
43.27
83.66
2.23
13.94
70.11
RED
13.91
41.75
78.04
8.25
26.55
74.09
26.58
54.64
87.67
2.50
16.06
70.28
RePE
11.46
41.01
76.31
13.11
30.46
77.47
21.94
46.25
84.01
2.26
14.58
70.13
ReFT
15.84
41.37
79.43
18.05
35,43
81.07
25.04
48.93
86.44
5.15
23.85
72.01
LoFIT
11.23
40.73
76.15
9.17
28.47
74.72
26.47
54.50
87.58
2.36
15.02
70.19
Our
21.12
$7.54
83,38
28.32
52.60
89.08
35.32
58.54
94.99
11.24
32.25
76.15
Case 1: Commonsense Reasoning Task (ARC-c) on LLaMA-3.1-8B
Instruction: Please choose the correct answer to the question: [Question] (n)n Answerl: fanswer1 An {answer2)|nn Answer3: {answer3)]nn Answer4: (answer4) \ni, Answer format: an- ### Response:)u
Answer2:
Question: A definite shape and definite volume properties of which state of matter?
Answers:
Answerl:
solid, only'n Answer2: liquid, only' Answer3: solid and liquid' Answer4: liquid and gas
Zero-Shot: solid, only/liquid, only/solid and liquid/liquid and gas. (Non Results)
LoRA: answerl (True Answer)
BitFit: solid, only/iquid, only (Non Results)
RED: answer3 (Wrong Answer)
REPE: answer3 (Wrong Answer)
ReFT: answer3 (Wrong Answer)
LoFIT: answer4 (Wrong Answer)
JoLA: answerl (True Answer)
21

---

Joint Localization and Activation Editing for Low-Resource Fine-Tuning
Case 2: Natural Language Understanding Task (Math) on Owcn-2.5-7B
Instruction:
The following are multiple choice questions (with answers) about (doma in). Please return the answer
in the format of "The answer is (X)" at the end. Question: {question) \n Options: A. {A} B. (B) c. {c) D.
{D}EE. {a} F. {F} G. {G} H. {H} I. {I} J. {J} n #### Response: |n
Question: Use Euler's Method to calculate the approximation of y(0.2) where y(x) is the solution of the initial-value
problem that is as follows. s/ + xy + y - 0 and y(0) 2 and y0) 3.
Options: A. 2.67 B. 2.33 C. 1.76 D. 1.98 E. 2010 F. 2.58 G. 2.82 H. 3. 14 I. 3.02 2.45 \
Zero-Shot: E (Wrong Answer)
LoRA: (Wrong Answer)
BitFit: (Wrong Answer)
RED: E (Wrong Answer)
REPE: C. (Wrong Answer)
ReFT: !!!!"!!!!!!!!! (Non Results)
LoFIT: F (True Answer)
JoLA: F (True Answer)
Case 3: Natural Language Generation Task (E2E_NLG) on Qwen-2.5-7B
Instruction: Please generate restaurant description from the information given below: {data) \nin ####
Response: n
Data: name(Zizzi], [pub), near[The Sorrento]
Reference: There is pub called Zizzi located near The Sorrento.
Zero-Shot: Nestled in the heart of the city, just stone's throw away from The Sorrento, Zizzi is charming and cozy pub that offers delightful dining experience. With its warm and inviting atmosphere, Zizzi is the perfect spot for casual meal, relaxing drink,
lively gathering ... (Tend to generate without end)
Zizzi is pub located near The Sorrento. It is a great place to grab a bite to eat and enjoy a drink in a lively atmosphere. Zizzi's 's menu features a variety of delicious dishes, from classic pub fare to more adventurous options. Whether you're re the mood for a hearty burger... (Tend to generate without end)
LoRA:
BitFit: There a pub called Zizzi near The Sorrento.
a good place to eat. It pub. It is near The Sorrento. called Zizzi. It is good place eat. It is pub... (Tend generate repetitions without end)
RED: Nestled in the heart of the city, just stone's 's throw away from The Sorrento, Zizzi is charming and cozy pub that offers delightful dining experience. With its warm and inviting atmosphere, Zizzi is the perfect spot for casual meal, a relaxing drink, or a... (Tend to generate without end)
REPE: Zizzi is pub. It is located near The Sorrento. Zizzi is place where you can go to eat. Zizzi is a place where you can go to drink. Zizzi is place where you can go to socialize. Zizzi... (Tend to generate repetitions without end)
ReFT: Zizzi pub The Sorrento. Zizzi is restaurant in The Sorrentos. Zizzis restaurant the sorrentos. The Zizzis is restaurant the sorrentos.. Zizzi's is restaurant in sor.. (Tend to generate repetitions without end) LoFIT: Nestled in the heart of the city, just stone's 's throw away from The Sorrento, Zizzi is charming and cozy pub that offers delightful dining experience. With its warm and inviting atmosphere, Zizzi is the perfect spot for casual meal or relaxing drink with friends. (Ends normally, but no Sorrento-related information.) JoLA: Zizzi is a pub near The Sorrento, (True Answer)
22


Reviews:
[{"review_id": "VSYSdXX6yW", "reviewer": "Reviewer_QWX1", "summary": "This paper presents a novel extension of the activation editing approach. Its primary contribution lies in integrating localization and editing into a single process using a gating mechanism, unlike previous two-stage methods that first manually locate and then edit model components. This makes the proposed method more practical and adaptive. Extensive experiments on multiple LLMs demonstrate the effectiveness of this approach, particularly in low-resource scenarios, where it outperforms existing methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and suggestions.  \n**Comment:** The parameterized form of the hard-concrete distribution is not explicitly detailed. The trainable parameters of the gating mechanism are not clearly stated.  \n**Response:** The hard concrete distribution has two associated scalar parameters: a scale parameter and a temperature parameter. Following prior work on sparsification (e.g., Voita et al., 2019; Louizos et al., 2017), we train only the scale parameter and fix the temperature to 0.33.  To clarify, these gates do not take any input – each gate is simply an instance of the hard concrete distribution with a single learnable parameter.  We will clarify this in the new version of the paper.\n+ **Q1: What exactly do the hidden states and bias term in Fig. 2 refer to?**\n- **A1:** Figure 2 illustrates the previously proposed forms of intervening in Transformer modules. The “Hidden states” approach follows REFT, applying interventions directly to the MLP hidden states. The “Bias” approach follows BitFit, modifying only the bias terms of attention, dropout, and layer norm activations. We will elaborate on these in Section 3.1 (currently lines 86–88) to clarify further.\n+ **Q2: Could the reason for multiple components performing worse in Fig. 2 be overfitting?**\n- **A2:** We think it can be interpreted as a form of overfitting, though we do mitigate this using early stopping. More broadly, the results highlight that both deciding where we intervene (i.e., the choice and location of components) and having fewer interventions are important in our low-resource setting. As shown below, even when provided with more data (500 examples), the MLP+Attention intervention fails to match the performance of Attention-only using 200 examples across most datasets.\n\n**The same sample size as in Figure 2 (200)**\n||SIQA|WinoGrande|Law|Physics|E2E_NLG|WEB_NLG|\n|-|-|-|-|-|-|-|\n|Attention|55.94|55.33|36.00|7.00|14.77|18.12|\n|MLP|50.10|51.62|34.00|20.00|10.31|14.45|\n**Sample size (300)**\n||SIQA|WinoGrande|Law|Physics|E2E_NLG|WEB_NLG|\n|Attention|56.34|55.85|36.00|18.00|15.25|18.46|\n|MLP|53.07|51.93|35.00|20.00|10.68|14.91|\n**Sample size (500)**\n||SIQA|WinoGrande|Law|Physics|E2E_NLG|WEB_NLG|\n|Attention|56.86|56.37|37.00|20.00|15.87|18.85|\n|MLP|53.92|52.45|36.00|21.00|11.22|15.26|\n**Different Sample Size (Attention+MLP)**\n|Samplesize|SIQA|WinoGrande|Law|Physics|E2E_NLG|WEB_NLG|\n|200|52.17|48.74|23.00|13.00|8.23|12.36|\n|300|52.82|49.51|25.00|14.00|8.85|12.74|\n|500|53.24|50.36|28.00|16.00|9.39|13.03|\n\n+ **Q3: How many attention heads are typically completely shut off by the end of training?**\n+ **A3:** To clarify, we are not shutting down any attention heads. Instead, we selectively choose which heads to apply interventions to. When both the offset gate $g_a$ and the multiplicative update gate $g_m$ are set to $0$, the head’s computation becomes identical to that of the original model. By the end of training, most gates are closed  – for example, on OBQA, 86\\% of attention heads have $g_a = 0$, and 94\\% have $g_m = 0$ (See Figure 8).\n+ **Q4: How does the gating mechanism's randomness manifest during inference?**\n+ **A4:** During training, we model each gating variable (e.g., $g_a^{(l,i)}$ and $g_m^{(l,i)}$) as random, sampled from a hard-concrete distribution. However, during inference, we use their expected values, $\\mathbb{E}[g_a^{(l,i)}]$ and $\\mathbb{E}[g_m^{(l,i)}]$, instead of sampling from the distributions. This removes randomness, ensuring consistency and stability in inference. We briefly mention this in lines 165-166 and will clarify in the next version.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem and application at hand:\n\n1. The use of a gating mechanism to jointly perform localization and editing is a reasonable approach. \n\n2. The experiments are well-controlled by ensuring a fair comparison of trainable parameters.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The soundness and validity of the experimental designs and analyses were checked.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "I did not find any essential references that were missing from the discussion.", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "I find the description of the gating mechanism not entirely clear. Specifically, the parameterized form of the Hard-Concrete distribution is not explicitly detailed. The trainable parameters of the gating mechanism are not clearly stated. It would be helpful if the authors could clarify these aspects to provide a more precise understanding of how the gating mechanism operates and how it is optimized during training.", "questions_for_authors": "1. What exactly do the Hidden States and bias term in Fig. 2 refer to?\n\n2. Could the reason for multiple components performing worse in Fig. 2 be overfitting? Similarly, the authors' gating mechanism demonstrates that selectively editing attention heads is more effective than editing all heads, which is somewhat counterintuitive. Could this also be due to overfitting?\n\n3. By the end of training, how many attention heads are typically completely shut off?\n\n4. Since the gating mechanism is modeled as a distribution, how does its randomness manifest during inference?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BEIXuhraxZ", "reviewer": "Reviewer_2Bjt", "summary": "The paper proposes JORA, an interpretability-inspied parameter-effecient tuning methods. JOLA intervenes on the attention activations with both scaling and offsetting. In addition, JOLA uses HardConcrete gates with expected-L0 regularization to learns the localization together with intervention in an end to end way.\n\nThe paper conducts evaluation on multiple datasets and mainly compares against other activation editing baselines (ReFT and LoFiT). The results suggest the proposed approach is effective.\n\n## update after rebuttal\n\nThank you for your response. I've update the score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and provide valuable insights.\n\nQ1: Baseline setup seem either under-tuned or too weak to compare with?\n\nA1: Thank you for your feedback on our baseline system setup.\n\n(1) As noted in the original papers for each baseline method, these methods are sensitive to hyperparameters, and we detail this in Appendix D.3. The following table shows the sensitivity of these methods, with the results highlighting the lack of robustness, which motivates our proposal for a more stable approach.  \n**Different learning rate in RED**\n||SIQA|WinoGrande|Law|Physics|e2e_nlg|Web_nlg|\n|-|-|-|-|-|-|-|\n|5e-5|48.47|48.25|28.00|21.00|8.27|12.74|\n|2e-4|51.23|50.87|32.00|18.00|11.33|14.53|\n|6e-2|50.16|47.14|29.00|23.00|10.14|13.59|\n**Different prefix and suffix positions in ReFT**\n||SIQA|WinoGrande|Law|Physics|e2e_nlg|Web_nlg|\n|p7+s7|62.51|53.96|28.00|40.00|11.44|12.57|\n|p11+s11|58.24|51.48|30.00|41.00|12.02|11.35|\n**Number of Attention Heads in LoFIT**\n||SIQA|WinoGrande|Law|Physics|e2e_nlg|Web_nlg|\n|32|54.21|52.34|33.00|10.00|13.18|17.27|\n|64|56.52|54.56|35.00|12.00|15.56|16.34|\n|128|51.26|55.42|30.00|8.00|14.60|18.07|\n\n(2) Baseline methods require specific fine-tuning for each task, and we have 26 tasks to evaluate in our method. Fine-tuning all tasks individually is impractical due to the large hyperparameter search space. Therefore, we performed hyperparameter selection using a grid search approach. For each task, we ran a grid search with five different hyperparameter configurations, which were chosen to explore a diverse range of parameter settings that could provide the best model performance. We performed this search over key hyperparameters (e.g., learning rate, selected head/layers/positions etc. highlighted in Appendix D.3), using a validation set to select the configuration that resulted in the best performance. The results presented here correspond to the best hyperparameter configuration selected for each task based on the validation set performance. The final model was evaluated with these hyperparameters, and we averaged the results across all tasks. As observed, this had little effect on their relative performance, with our method continuing to outperform the others.\n\nTable:  best hyperparameter configuration in LLaMA\n||Reasoning|Understanding|Generation||\n|-|-|-|-|-|\n||ACC|ACC|BLEU|Rouge-L|\n|zero_shot|53.70|40.00|12.56|36.70|\n|BitFit|65.37|36.14|10.23|32.59|\n|RED|50.26|37.86|12.77|34.19|\n|REPE|66.04|37.43|11.49|31.04|\n|REFT|67.12|42.29|13.05|38.25|\n|LOFIT|57.74|32.71|13.14|35.51|\n|Our|70.55|47.00|17.07|40.65|\n\n(3) Given the small dataset (e.g., 200 samples in our setting), overfitting was a concern. To reduce overfitting's impact on the baseline, we used early stopping, which was not applied in the original implementation of the baseline systems. We also found that learning rate adjustment significantly affected the results. We evaluated four strategies, including linear schedule[1], Cyclic Learning Rate Schedule[2], Adaptive Heuristic Schedule[3] and Exponential Decay Schedule[4]. As shown in the following table, the exponential decay strategy proved most stable, so we used it for both the baseline and our method, as explained in Appendix D.1. The comparison of different learning rate decay strategies for JoLA and LoFIT is as follows.\n[1] Human-level control through deep reinforcement learning (Mnih et al., 2015)\n[2] Cyclical Learning Rates for Training Neural Networks (Smith et al., 2017)\n[3] A disciplined approach to neural network hyper-parameters (Smith et al., 2018)\n[4] An exponential learning rate sched- ule for deep learning (Li et al., 2019)\n\n**Different learning ratestrategies in JoLA**\n|Strategy|SIQA|WinoGrande|law|physics|e2e_nlg|web_nlg|\n|-|-|-|-|-|-|-|\n|Linear|62.71|56.49|38.00|42.00|14.05|22.83|\n|Cycle|64.25|57.26|39.00|43.00|14.37|23.44|\n|Adaptive|65.47|58.60|39.00|44.00|15.02|23.86|\n|Exponential|66.22|58.33|40.00|46.00|15.54|24.39|\n**Different learning rate strategies in LoFIT**\n|Strategy|SIQA|WinoGrande|law|physics|e2e_nlg|web_nlg|\n|Linear|54.13|53.36|35.00|6.00|13.84|16.95|\n|Cycle|54.32|54.25|34.00|6.00|14.37|17.83|\n|Adaptive|55.18|55.57|36.00|7.00|15.24|17.64|\n|Exponential|55.94|55.33|36.00|7.00|14.77|18.12|\n\nQ2: (same questions as Review QWX1 Review m3pE) parameter counting of the gating mechanism?\n\nA2: We mentioned the parameter counting in Table 3 of Appendix A, using the same calculation method as ReFT and LoFIT, where trainable parameters are divided by the parameters of the base LLM. For example, we calculated the parameters for the SIQA task using the LLaMA-3-8B model. JoLA’s parameter count considered only the interventions. Interestingly, JoLA’s count matches LoFIT’s. \n\nQ3: Some figures are hard to read, e.g., Figure 4 and Figure 8.\n\nA3: Thank you for your feedback. Due to space constraints, we combined multiple images into one, which affected readability. In the next version, we will adjust the layout to improve clarity and readability.", "claims_and_evidence": "The claims are mostly valid. JoLA extends previous methods by jointly learning to localize and edit activations, which seems effective. The paper also provides abundant analysis.\n\nThough I have some concerns in the way that the paper sets up the comparison with respect to the baselines, and the way that the paper discusses the number or parameters.", "methods_and_evaluation_criteria": "The proposed method jointly optimize for localization and editing, which is well-motivated and makes sense.\n\nThe paper also provides experiments across different model families, datasets, and model sizes", "theoretical_claims": "N/A as there aren't theoretical claims in the main paper.", "experimental_designs_or_analyses": "First, I have some concerns about the way that the paper sets up the baselines.\n\nThe Baseline results seem either under-tuned or too weak to compare with: In Table 1, a lot of the fine-tuning baselines underperform zero-shot (e.g. for Qwen2.5 7B on the reasoning tasks, none of the 6 baselines outperform the zero-shot baseline).\n\nI wonder how the bae line is set up. The paper claims that baseline results were obtained by \"selecting five hyperparameters and **averaging** the results.\" (ln 246) A more common approach is establishing the best results out of all the five hyperparameters as the baselines.\n\nSecond, I wonder how is the number of learnable parameters counted in Table 3. Is it only counting the gating variables, or is it counting the \"activated\" attention heads? If I understand correctly, JOLA incorporates two gating, one scaling, and one additive parameters for each attention heads. All these are learned end to end and need **gradient back-propagation**. Are all these being counted in the number of parameters?", "relation_to_broader_scientific_literature": "The methods improves on previous methods by jointly localizing and editing the activations, which is learnt end-to-end with gating.", "essential_references_not_discussed": "Not that I am aware of.", "other_strengths_and_weaknesses": "The paper provides abundant ablations on the design choice of the method.", "other_comments_or_suggestions": "Some figures are hard to read, e.g., Figure 4 and Figure 8.", "questions_for_authors": "See Experimental Designs Or Analyses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xDcMwngSNQ", "reviewer": "Reviewer_upuz", "summary": "This paper introduces JOLA, a novel approach to efficiently adapt large language models in low-resource settings. The method jointly learns which attention heads to modify and determines optimal activation interventions using both additive and multiplicative adjustments. By incorporating a dynamic gating mechanism based on Hard-Concrete distributions with expected-L0 regularization, JOLA selectively edits a minimal set of components, thereby significantly reducing the number of trainable parameters. Extensive experiments on tasks spanning commonsense reasoning, natural language understanding, and language generation demonstrate that JOLA consistently outperforms existing methods such as LoRA and other activation editing techniques, achieving robust and scalable improvements even with limited training data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments and the opportunity to clarify and improve our work.\n\nQ1: Huge difference in hyperparameter selection among the baseline methods and JoLA. The reproducibility of our proposed JoLA\n\nA1: As noted in the original papers for each baseline method, these methods are sensitive to hyperparameters, and we detail this in Appendix D.3. The following table shows the sensitivity of these methods, with the results highlighting the lack of robustness, which motivates our proposal for a more stable approach.\n\n**Different learning rate in RED**\n||SIQA|WinoGrande|Law|Physics|e2e_nlg|Web_nlg|\n|-|-|-|-|-|-|-|\n|5e-5|48.47|48.25|28.00|21.00|8.27|12.74|\n|2e-4|51.23|50.87|32.00|18.00|11.33|14.53|\n|6e-2|50.16|47.14|29.00|23.00|10.14|13.59|\n**Different prefix and suffix positions in ReFT**\n||SIQA|WinoGrande|Law|Physics|e2e_nlg|Web_nlg|\n|p7+s7|62.51|53.96|28.00|40.00|11.44|12.57|\n|p11+s11|58.24|51.48|30.00|41.00|12.02|11.35|\n**Number of Attention Heads in LoFIT**\n||SIQA|WinoGrande|Law|Physics|e2e_nlg|Web_nlg|\n|32|54.21|52.34|33.00|10.00|13.18|17.27|\n|64|56.52|54.56|35.00|12.00|15.56|16.34|\n|128|51.26|55.42|30.00|8.00|14.60|18.07|\n\nQ2: Providing a more detailed explanation of the statistical significance of the results.\n\nA2: Thank you for your suggestion. We will incorporate the statistical significance; note though that the gap is large \nWe agree that conducting a statistical significance analysis is important to strengthen the credibility of our results. The significance test shows a meaningful difference between our method and the baseline methods. \n\nQ3: How does the JoLA method perform when given larger resources?\n\nA3: Thank you for your suggestion. We conducted experiments on larger datasets, and as shown in the table, our method remains effective with 5,000 and 10,000 samples. However, using 20,000 and 100,000 samples, there is a slight gap between our method and LoRA. These differences are acceptable, as we update fewer parameters compared to LoRA.\n\n||SIQA||WinoGrande||\n|-|-|-|-|-|\n||JoLA|LoRA|JoLA|LoRA|\n|1000|74.77|70.04|74.30|71.16|\n|2000|75.24|73.12|74.92|71.75|\n|3000|75.43|73.85|75.37|72.32|\n|5000|75.88|74.35|75.91|73.57|\n|8000|75.91|75.14|76.20|74.69|\n|10000|75.96|75.69|76.31|75.92|\n|20000|76.02|76.48|76.55|76.47|\n|30000|76.08|76.94|76.68|77.24|\n|50000|76.15|77.23|76.84|78.33|\n|80000|76.21|77.56|76.92|78.52|\n|100000|76.26|77.81|77.04|78.96|\n\nQ4: Could the effectiveness of JoLA on the MLP layer be verified?\n\nA4: We also applied the gating mechanism-based strategy to the MLP layer. As shown in the following table, our experimental results demonstrate that the JoLA method is also effective in the MLP layer. However, it empirically tends to be more effective when used in the attention layer.\n\n||Reasoning||Understanding||Generation||\n|-|-|-|-|-|-|-|\n||SIQA|WinoGrande|Law|Physics|e2e_nlg|Web_nlg|\n|MLP w/o gate|50.10|51.62|34.00|20.00|10.31|14.45|\n|MLP with gate|52.46|52.43|36.00|23.00|11.23|16.25|\n|Attention w/o gate|55.94|55.33|36.00|7.00|14.77|18.12|\n|Attention with gate|66.22|58.33|40.00|46.00|15.54|24.39|\n|Attention+MLP w/o gate|52.17|48.74|23.00|13.00|8.23|12.36|\n|Attention+MLP with gate|53.28|52.07|27.00|16.00|10.42|14.83|", "claims_and_evidence": "The claims proposed in the paper are substantiated by extensive experimental evidence, demonstrating that the JOLA method not only consistently outperforms baseline activation editing and parameter-efficient fine-tuning techniques in low-resource scenarios, but also significantly enhances performance across a diverse set of tasks including commonsense reasoning, natural language understanding, and generation. Moreover, the ablation studies provide clear and convincing evidence that the dynamic gating mechanism and selective attention head modifications are key factors driving these improvements, effectively validating the paper's central claims.", "methods_and_evaluation_criteria": "The Joint Localization and Activation Editing (JOLA) approach presented in this paper is soundly designed. The challenge of efficiently adapting large language models with scarce data stems from the difficulty in discerning which internal components yield the greatest impact. JOLA overcomes this by simultaneously determining the most influential attention mechanisms to modify and selecting the appropriate adjustment strategy—be it scaling, shifting, or a combination—while fine-tuning the corresponding parameters, thereby ensuring that the targeted modifications are optimally calibrated for low-resource environments.", "theoretical_claims": "The paper does not include theoretical proofs.", "experimental_designs_or_analyses": "The paper presents a thorough experimental design to evaluate the proposed Joint Localization and Activation Editing (JOLA) method for low-resource fine-tuning. In addition, the inclusion of extensive ablation studies (on gate mechanisms, the number of gates, and head selection strategies) and analyses across different data and model sizes strengthens the validity of their claims. The author mentioned that there is a huge difference in hyperparameter selection among the baseline methods. Could the extreme value variance for JORA and the other methods be displayed? Furthermore, providing a more detailed explanation of the statistical significance of the results and the reproducibility of the experiments would strengthen confidence in the research findings.", "relation_to_broader_scientific_literature": "To improve low-resource adaptation of large language models, prior works have explored both parameter-efficient fine-tuning methods, such as LoRA and BitFit, and activation editing techniques like RED, REPE, and LoFIT. The JoLA method proposed in this paper innovatively combines dynamic attention head localization with joint optimization of additive and multiplicative interventions. Drawing on ideas from network pruning—specifically, HardConcrete gating and expected-L0 regularization—JoLA overcomes the static limitations of previous methods, resulting in a more adaptive and robust fine-tuning strategy that aligns well with contemporary advances in model adaptation.", "essential_references_not_discussed": "na", "other_strengths_and_weaknesses": "Strengths:\nOriginality:\nJOLA introduces an innovative approach by jointly determining which attention heads to modify and optimizing both scaling and shifting interventions, thereby creatively combining insights from activation editing and network pruning.\nSignificance:\nThe extensive experiments across commonsense reasoning, language understanding, and generation tasks clearly demonstrate that JOLA substantially outperforms existing fine-tuning methods in low-resource settings, underscoring its potential impact on efficient model adaptation.\nClarity:\nThe paper is well-structured and offers detailed explanations of the dynamic gating mechanism and head selection strategy, with comprehensive ablation studies that effectively illustrate the contribution of each component.\n\nWeaknesses:\nThe approach’s reliance on specific gating mechanisms and parameter settings may introduce additional complexity and sensitivity to hyperparameter tuning, potentially affecting reproducibility.", "other_comments_or_suggestions": "na", "questions_for_authors": "1.\tHow does the JOLA method perform when given larger resources? Can this activation-based editing approach be applied to larger datasets?\n2.\tThe paper shows that the activation editing method performs best when applied to the Attention layer. Is this also the case for JOLA? Could the effectiveness of JOLA on the MLP layer be verified?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jBzAag2D0Y", "reviewer": "Reviewer_m3pE", "summary": "The paper proposes JOLA, a novel parameter-efficient fine-tuning (PEFT) method that dynamically selects and edits the outputs of specific Transformer attention heads. JOLA jointly learns: (1) which heads to target, (2) the intervention type (additive, multiplicative, or both), and (3) the corresponding parameters (additive offsets/multiplicative scalings). Evaluated on three NLP tasks—commonsense reasoning, language understanding, and generation—JOLA outperforms baselines under low-resource settings, demonstrating improved stability and efficiency through its unified framework of localization and modular activation editing.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and suggestions\n\n**Q1: Presentation in Figure 1, Table 2.**\n\n**A1:** Thank you for your suggestion. We will make the necessary changes in the next version.\n\n**Q2: Relationship between PEFT?**\n\n**A2:** In the abstract, we introduced PEFT and its limitations in low-resource scenarios. Activation editing is a form of PEFT but updates fewer parameters compared to traditional methods like Adapter and LoRA. Traditional methods are less suitable for low-resource scenarios, and our approach aims to tackle this issue. We will clarify this logic in the next version to avoid any confusion.\n\n**Q3: Provide a detailed comparative analysis between JOLA and LOFIT.**\n\n**A3:** Below, we provide a  comparison focusing on methodological differences, intervention strategies, and empirical performance.\n\n(1) Methodological differences: Unlike LoFiT’s rigid two-stage pipeline, JoLA unifies localization and intervention into a single end-to-end framework. This allows dynamic adaptation to task requirements, avoiding suboptimal head selection caused by decoupled optimization.\n|| LoFIT|JoLA|\n|-|-|-|\n|Localization|Two-stage process: 1. Select heads via learning with multiplicative interventions 2. Freeze selection and train additive interventions (bias vectors). | Joint optimization: Dynamically selects heads while learning intervention parameters.|\n|Intervention Type|Additive only (bias vectors applied to selected heads).|Hybrid intervention: Combines additive biases and multiplicative scaling via adaptive gating.|\n|Sparsity Control|L1 regularization on scaling factors to select top-K heads.|Hard Concrete gates with expected-L0 regularization, enabling probabilistic head pruning during training.|\n|Flexibility|Fixed intervention strategy (additive) after head selection.|Learns task-specific intervention types (e.g., scaling vs. bias) per head.|\n\n(2) Formula-level comparison\n\n**LoFIT:**\n$$z^{(l,i)}_t \\leftarrow z^{(l,i)}_t + v^{(l,i)} \\quad \\text{(Additive bias)}$$\n\n**Limitation:** Static additive edits may insufficiently adapt complex tasks requiring both amplification and suppression of features.\n\n**JoLA:**\n$$\nz^{(l,i)}_t = (1 + g_m^{(l,i)} \\cdot m^{(l,i)}) \\odot z^{(l,i)}_t + g_a^{(l,i)} \\cdot a^{(l,i)}\n$$\n\nwhere,\nScaling: $(1 + g_m^{(l,i)} \\cdot m^{(l,i)})$,\nBias: $g_a^{(l,i)} \\cdot a^{(l,i)}$\n\n**Advantage:** The hybrid operation enables fine-grained control over activations—scaling amplifies/suppresses existing features, while biases shift representations.\n\n(3) Empirical Performance\n+ JoLA show robustness across 26 tasks under low-resource settings.\n+ JoLA is more parameter efficient than LoFIT, we will update the parameter counting as discussed in Review 2Bjt [Q2] in the new version of the paper.\n\n**Q4: Parameter issues**\n\n**A4:** As noted in the original papers for each baseline method, these methods are sensitive to hyperparameters, and we detail this in Appendix D.3. The following table shows the sensitivity of these methods, with the results highlighting the lack of robustness, which motivates our proposal for a more stable approach.\n\n**Different learning rate in RED**\n||SIQA|WinoGrande|Law|Physics|e2e_nlg|Web_nlg|\n|-|-|-|-|-|-|-|\n|5e-5|48.47|48.25|28.00|21.00|8.27|12.74|\n|2e-4|51.23|50.87|32.00|18.00|11.33|14.53|\n|6e-2|50.16|47.14|29.00|23.00|10.14|13.59|\n**Different prefix and suffix positions in ReFT**\n||SIQA|WinoGrande|Law|Physics|e2e_nlg|Web_nlg|\n|p7+s7|62.51|53.96|28.00|40.00|11.44|12.57|\n|p11+s11|58.24|51.48|30.00|41.00|12.02|11.35|\n**Number of Attention Heads in LoFIT**\n||SIQA|WinoGrande|Law|Physics|e2e_nlg|Web_nlg|\n|32|54.21|52.34|33.00|10.00|13.18|17.27|\n|64|56.52|54.56|35.00|12.00|15.56|16.34|\n|128|51.26|55.42|30.00|8.00|14.60|18.07|\n\n**Q5: What happens on the physics task in Figure 2?**\n\n**A5:** Upon reviewing the generated text for the physics task in Figure 2, we found that in the low-resource setting, the generated outputs did not consistently follow the required format, which impacted the results. Specifically, the generated text lacked the necessary result options, leading to lower accuracy when evaluated using exact match criteria.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes, no issues.", "relation_to_broader_scientific_literature": "While the paper introduces JOLA as a novel PEFT method for joint localization and activation editing, there is insufficient clarity in distinguishing it from LOFIT, a closely related prior work that also intervenes on specific Transformer heads with additive/multiplicative editing. Both JOLA and LOFIT aim to (1) identify select heads for editing and (2) apply combined additive/multiplicative interventions. The paper does not clearly articulate what differentiates JOLA’s joint optimization framework from LOFIT’s approach, although the performance of JOLA is higher than LOFIT.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\n1.\tThe article has a clear and coherent structure with well-organized content.\n2.\tThe article provides rich visual and analytical insights. The figures and tables effectively clarify complex concepts. The visualization of gate status provides valuable empirical evidence of JOLA's mechanism.\n3.\tJOLA outperforms baselines across multiple NLP tasks, particularly under low-resource settings. The results are well-documented, and the performance gains are consistent, supporting the method's efficacy.\n\nWeaknesses:\n\n1.\tIn Figure 1, you can draw a red box structure diagram of LoFIT like RED and JOLA.\n2.\tIn Figure 1, LoFIT's Step 2, it's a \"+\", not a \"x\" sign.\n3.\tThe article mentions \"However, the effectiveness of standard PEFT methods is limited in low-resource scenarios with only a few hundred examples.\" The PEFT method is a series of fine-tuning methods proposed to reduce computing resources. This turn of the article is to explain the shortcomings of PEFT, but the low-resource background is not a defect of PEFT itself. The logical relationship between this and the following text is not clear.\n4.\tI hope the article can provide a detailed comparative analysis between JOLA and LOFIT. From the current description of the article, LOFIT's m is learnable, but the maximum value of the value range is a hyperparameter and needs to be set manually. So, is LoFIT's m the product of JoLA's g and m? If LoFIT's m is 0, does it mean that the head is not selected? Similarly, is LoFIT's a the sum of JoLA's g and a? In addition, the article mentions that LoFIT determines the selected head by multiplication, but JOLA also uses multiplication to determine the selected head. What is the specific difference here? Or are the two stages of LOFIT not distributed as shown in Figure 1, because at present, the equation for synthesizing one stage is also acceptable.", "other_comments_or_suggestions": "1.\tTo enhance clarity in Figure 1, I suggest utilizing distinct colors for the boxes representing different methods. This would allow for easier differentiation and improve the overall visual appeal of the figure.\n2.\tIn Table 2, it would be beneficial to bold the maximum value to draw attention to this important data point, thereby facilitating a quicker understanding of the results presented.", "questions_for_authors": "1.\tIt is worth exploring why JoLA incorporates two additional gates while maintaining the same overall number of parameters as LOFIT. A thorough explanation of this architectural choice would enhance the reader's understanding of how JoLA optimally manages its parameters.\n2.\tIn Figure 2, the performance of editing attention modules on the Physics task appears to be the lowest. An analysis of this outcome is warranted.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents JoLA (Joint Localization and Activation Editing), a novel and practical method for parameter-efficient fine-tuning (PEFT) of large language models (LLMs) in low-resource settings. JoLA introduces a unified framework that simultaneously: 1). Identifies which attention heads to intervene on, 2). Learns the type of intervention (additive, multiplicative, or both), 3). Optimizes the associated intervention parameters. A core innovation lies in its use of Hard Concrete gating with expected-L0 regularization, allowing selective and sparse updates. 

JoLA is a well-motivated, rigorously evaluated, and practically impactful method for PEFT in LLMs. The approach is both novel in design and strong in empirical performance [JoLA consistently outperforms baselines across 26 tasks and multiple model families, including LLaMA, Qwen, and Mistral, under low-resource settings (e.g., ~200 samples)], with substantial improvements over state-of-the-art baselines in low-resource settings. I recommend acceptance.
Decision: Accept (poster)