Title: RuleEdit: Benchmarking Rule-Level Knowledge Editing in Large Language Models
Primary area: datasets and benchmarks
Keywords: Knowledge Editing, Rule-level Editing

Abstract: Knowledge editing seeks to update language models without full retraining. While most prior work focuses on isolated factual or instance-level edits, we explore a more structured domain: mathematical rules. We introduce \textbf{RULE-EDIT}, the first benchmark explicitly designed for editing and evaluating rule-level abstract knowledge in LLMs. Beyond measuring direct edit accuracy, our benchmark is designed to encourage deeper investigation into the interpretability and symbolic reasoning capabilities of LLMs: (1) To what extent do edits to abstract rules propagate to derived instances? and (2) How well do token-level updates align with higher-level symbolic structures across formats? To evaluate this, we propose two new metrics:\emph{Instance Portability} and \emph{Rule Understanding} that quantify whether edits correctly generalize to rule-governed examples and maintain consistency across symbolic and natural language representations. Through experiments on best-performing open-source LLMs using representative editing methods, we find that while models can often overwrite formula-level knowledge, they frequently struggle to propagate these edits to rule-derived instances and to maintain consistency across different forms of a rule.  For example, several methods achieve nearly 100\% reliability on direct rule queries, yet their rule-specific scores remain unsatisfactory (Instance Portability never exceeds 52\% and Rule Understanding stays below 26\%).  Our findings highlight the limits of current editing methods and motivate rule editing as a testbed for controllable knowledge in LLMs.