ICML 2025

Resolving Lexical Bias in Model Editing
Hammad Rizwan 1 Domenic Rosati 1 Ga Wu 1 Hassan Sajjad 1
Abstract
Model editing aims to modify the knowledge
of a pre-trained language model. Previous ap-
proaches have often involved direct alterations to
model weights, which can result in model degrada-
tion. Recent weight-preserving techniques avoid
making modifications to the model’s weights
by employing an adapter that implements edits
through auxiliary components. These rely heavily
on scoping mechanisms based on distance func-
tions on the model’s representation space to de-
termine when to trigger edits. We demonstrate
that current adapter methods are critically vul-
nerable to strong lexical biases, leading to is-
sues such as applying edits to irrelevant prompts
with overlapping words. This paper presents a
principled approach to learning a disentangled
representation space that facilitates precise lo-
calization of edits by maintaining distance be-
tween irrelevant prompts while preserving prox-
imity among paraphrases. In our empirical study,
we show that our method, Projector Editor Net-
works for Model Editing - PENME, achieves
state-of-the-art model editing results while be-
ing computationally efficient during inference
compared to previous methods and adaptable
across different architectures. We provide the
codebase of PENME here: https://github.
com/hammadrizwan/PENME.git
1. Introduction
Large Language Models (LLMs) are successful in solving a
diverse range of natural language processing tasks (Devlin
et al., 2019; Liu et al., 2019; Touvron et al., 2023b; Radford
et al., 2019). Despite their successes, LLMs are fallible in
large part due to the noisy and imperfect nature of the data
used for training (Zhu et al., 2020). Moreover, as the world
1Department of Computer Science, Dalhousie University, Hal-
ifax, Canada.
Correspondence to: Hammad Rizwan <ham-
mad.rizwan@dal.ca>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canad...