Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing
Authors: name: Huanqian Wang; name: Yang Yue; name: Rui Lu; name: Jingxin Shi; name: Andrew Zhao; name: Shenzhi Wang; name: Shiji Song; name: Gao Huang
Published: 2024-07-11 17:52:03

Abstract: Large Language Models (LLMs) have demonstrated great potential as generalist
assistants, showcasing powerful task understanding and problem-solving
capabilities. To deploy LLMs as AI assistants, it is crucial that these models
exhibit desirable behavioral traits, such as non-toxicity and resilience
against jailbreak attempts. Current methods for detoxification or preventing
jailbreaking usually involve Supervised Fine-Tuning (SFT) or Reinforcement
Learning from Human Feedback (RLHF), which requires finetuning billions of
parameters through gradient descent with substantial computation cost.
Furthermore, models modified through SFT and RLHF may deviate from the
pretrained models, potentially leading to a degradation in foundational LLM
capabilities. In this paper, we observe that surprisingly, directly editing a
small subset of parameters can effectively modulate specific behaviors of LLMs,
such as detoxification and resistance to jailbreaking. Specifically, for a
behavior that we aim to avoid, we employ a linear classifier, which we term the
behavior probe, to classify binary behavior labels within the hidden state
space of the LLM. Using this probe, we introduce an algorithm to identify a
critical subset of LLM parameters that significantly influence this targeted
behavior. Then we directly edit these selected parameters by shifting them
towards the behavior probe. Such a direct parameter editing method necessitates
only inference-level computational resources. Experiments demonstrate that in
the representative detoxification task, our approach achieves reductions of up
to 90.0\% in toxicity on the RealToxicityPrompts dataset and 49.2\% on ToxiGen,
while maintaining the LLM's general capabilities in areas such as common sense,
question answering, and mathematics. Our code is available at
https://github.com/lucywang720/model-surgery.