ICML 2025

Joint Localization and Activation Editing for Low-Resource Fine-Tuning
Wen Lai 1 2 Alexander Fraser 1 2 Ivan Titov 3 4
Abstract
Parameter-efficient fine-tuning (PEFT) methods,
such as LoRA, are commonly used to adapt LLMs.
However, the effectiveness of standard PEFT
methods is limited in low-resource scenarios with
only a few hundred examples. Recent advances
in interpretability research have inspired the emer-
gence of activation editing (or steering) tech-
niques, which modify the activations of specific
model components. Due to their extremely small
parameter counts, these methods show promise
for small datasets. However, their performance is
highly dependent on identifying the correct mod-
ules to edit and often lacks stability across differ-
ent datasets. In this paper, we propose Joint Local-
ization and Activation Editing (JOLA), a method
that jointly learns (1) which heads in the Trans-
former to edit (2) whether the intervention should
be additive, multiplicative, or both and (3) the
intervention parameters themselves - the vectors
applied as additive offsets or multiplicative scal-
ings to the head output. Through evaluations on
three benchmarks spanning commonsense reason-
ing, natural language understanding, and natural
language generation, we demonstrate that JOLA
consistently outperforms existing methods.1
1. Introduction
Parameter-efficient fine-tuning (PEFT; Han et al., 2024)
methods are widely used to adapt large language models
(LLMs). However, popular PEFT approaches like LoRA
(Hu et al., 2021) often struggle in low-resource settings with
*Work done during Wen Lai’s visit to The University of Edin-
burgh. 1Technical University of Munich 2Munich Center for Ma-
chine Learning 3ILLC, University of Edinburgh 4ILLC, University
of Amsterdam. Correspondence to: Wen Lai <wen.lai@tum.de>,
Ivan Titov <ititov@inf.ed.ac.uk>, Alexander Fraser <alexan-
der.fraser@tum.de>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. ...