Precise In-Parameter Concept Erasure in Large Language Models
Authors: name: Yoav Gur-Arieh, name: Clara Suslik, name: Yihuai Hong, name: Fazl Barez, name: Mor Geva

Large language models (LLMs) often acquire knowledge during pretraining that
is undesirable in downstream deployments, e.g., sensitive information or
copyrighted content. Existing approaches for removing such knowledge rely on
fine-tuning, training low-rank adapters or fact-level editing, but these are
either too coarse, too shallow, or ineffective. In this work, we propose PISCES
(Precise In-parameter Suppression for Concept EraSure), a novel framework for
precisely erasing entire concepts from model parameters by directly editing
directions that encode them in parameter space. PISCES uses a disentangler
model to decompose MLP vectors into interpretable features, identifies those
associated with a target concept using automated interpretability techniques,
and removes them from model parameters. Experiments on Gemma 2 and Llama 3.1
over various concepts show that PISCES achieves modest gains in efficacy over
leading erasure methods, reducing accuracy on the target concept to as low as
7.7%, while dramatically improving erasure specificity (by up to 31%) and
robustness (by up to 38%). Overall, these results demonstrate that
feature-based in-parameter editing enables a more precise and reliable approach
for removing conceptual knowledge in language models.