LLM Unlearning Should Be Form-Independent
Authors: name: Xiaotian Ye, name: Mengqi Zhang, name: Shu Wu

Large Language Model (LLM) unlearning aims to erase or suppress undesirable
knowledge within the model, offering promise for controlling harmful or private
information to prevent misuse. However, recent studies highlight its limited
efficacy in real-world scenarios, hindering practical adoption. In this study,
we identify a pervasive issue underlying many downstream failures: the
effectiveness of existing unlearning methods heavily depends on the form of
training samples and frequently fails to generalize to alternate expressions of
the same knowledge. We formally characterize this problem as Form-Dependent
Bias and systematically investigate its specific manifestation patterns across
various downstream tasks. To quantify its prevalence and support future
research, we introduce ORT, a novel benchmark designed to evaluate the
robustness of unlearning methods against variations in knowledge expression.
Results reveal that Form-Dependent Bias is both widespread and severe among
current techniques.
  We argue that LLM unlearning should be form-independent to address the
endless forms of downstream tasks encountered in real-world security-critical
scenarios. Towards this goal, we introduce Rank-one Concept Redirection (ROCR),
a novel training-free method, as a promising solution path. ROCR performs
unlearning by targeting the invariants in downstream tasks, specifically the
activated dangerous concepts. It is capable of modifying model parameters
within seconds to redirect the model's perception of a specific unlearning
target concept to another harmless concept. Extensive experiments demonstrate
that ROCR significantly improves unlearning effectiveness compared to
traditional methods while generating highly natural outputs.