Mitigating Heterogeneous Token Overfitting in LLM Knowledge Editing
Authors: Tianci Liu
Ruirui Li
Zihan Dong
Hui Liu
Xianfeng Tang
Qingyu Yin
Linjun Zhang
Haoyu Wang
Jing Gao
Venue: ICML 2025 poster (2025)
Keywords: Knowledge Editing
Large Language Models
Primary area: deep_learning->large_language_models

Abstract: Large language models (LLMs) have achieved remarkable performance on various natural language tasks. However, they are trained on static corpora and their knowledge can become outdated quickly in the fast-changing world. 
This motivates the development of knowledge editing (KE) to update specific knowledge in LLMs without changing unrelated others or compromising their pre-trained capabilities. 
Previous efforts sought to update a small amount of parameters of a LLM and proved effective for making selective updates.  
Nonetheless, the edited LLM often exhibits degraded ability to reason about the new knowledge. 
In this work, we identify a key issue: \textit{heterogeneous token overfitting} (HTO), where the LLM overfits different tokens in the provided knowledge at varying rates.
To tackle this, we propose {OVERTONE}, a token-level smoothing method that mitigates HTO by adaptively refining the target distribution. 
Theoretically, OVERTONE offers better parameter updates with negligible computation overhead. 
It also induces an implicit DPO but does not require preference data pairs. 
Extensive experiments across four editing methods, two LLMs, and diverse scenarios demonstrate the effectiveness and versatility of our method.

Mitigating Heterogeneous Token Overfitting in LLM Knowledge Editing
Tianci Liu'
Ruirui Li
Zihan Dong²
Hui Liu?
Xianfeng Tang² Jing Gao
Qingyu Yin
2 Linjun Zhang? Haoyu Wang* +
Abstract
versatilitty, establishing them new foundations for nat- ural language processing applications (Bommasani et al., 2021; Zhou et al., 2023). Furthermore, with model sizes continually increasing, large language models (LLMs) ex- hibit emerging abilitics to follow natural language instruc- tions (Dong et al., 2022b; Ouyang et al., 2022), which em- powers their zero-shot adaptations to unseen tasks (Kojima et al., 2022), paving the way towards artificial general intel- ligence (Bubeck et al., 2023).
Large language models (LLMMs) have achieved remarkable performance on various natural lan- guage tasks. However, they are trained on static corpora and their knowledge can become outdated quickly in the fast-changing world. This moti- vates the development of knowledge editing (KE) to update specific knowledge in LLMs without changing unrelated others or compromising their pre-trained capabilities. Previous efforts sought to update a small amount of parameters of L.LM and proved effective for making selective updates. Nonetheless, the edited LLM often exhibits de- graded ability reason about the new knowledge. In this work, we identify key issue: heteroge- neous token overfitting (HTO), where the LLM overfits different tokens in the provided knowl- edge at varying rates. To tackle this, we propose OVERTONE, token-level smoothing method that mitigates HTO by adaptively refining the target distribution. Theoretically, OVERTONE offers bet- ter parameter updates with negligible computa- tion overhead. It also induces an implicit DPO but does not require preference data pairs. Ex- tensive experiments across four editing methods, two LLMs, and diverse scenarios demonstrate the effectiveness and versatility of our method.
Despite this remarkable potential, the real-world LLM de- ployment remains largely unresolved: LLMs are capable of comprehending a wide range of human instructions and queries, but they can only provide feedback based on their static knowledge from the data they were trained on. In fast-changing world, most knowledge quickly becomes outdated. For example, the updated knowledge about the president of United States would refer to Donald Trump rather than Joe Biden. Failing to maintain update-to-das knowledge could amplify critical issues such as making factual fallacy (De Cao et al., 2021) or producing harmful generations (Hartvigsen et al., 2022). However, the signifi- cant computational cost of retraining makes it impractical to frequently incorporate new knowledge.
As a remedy, knowledge editing (KE), whose goal is to update an LLM with some specific knowledge without hurt- ing irrelevant others and general ability, is proposed (Wang et al., 2023b; Zhang et al., 2024c). Full fine-tuning of LLMs proved ineffective as it severely disrupted irrelevant knowl- edge (Wang et al., 2023b), leading to an ddittng-loccllty trade-off. Here locality refers to the ability to maintain knowledge unrelated to the update, such as the prime min- ister of Canada for the previous case. To achieve a good locality, model updates need to be selective and should rely on small fraction of parameters (Wang et al., 2023b). Fol- lowing this principle, parameter-efficient fine-tuning (PEFT) methods such as LoRA (Hu et al., 2021) have achieved good performance (Wu et al., 2023). On the other hand, Huang et al. (2023); Dong et al. (2022a) restricted the updates to some pre-specified feed-forward network (FFN) layer that serves as knowledge storage (Dai et al., 2021). Meng et al. (2022a;b) refined the process by introducing locating stage to identify which layer the target knowledge is stored. These fine-grained manners have demonstrated impressive success
1. Introduction
Language models (LMs) parameterized by deep neural networks (Vaswani et al., 2017; Lewis et al., 2019; Rad- ford et al., 2019; Brown et al., 2020) demonstrate strong generalizability across various natural language genera- tion and classification tasks (See et al., 2019; Raffel al., 2020; Ji et al., 2023). These successes underscore their
'Purdue University AAmazon 'Rutgers University Universsity at Albany. Correspondence to: Haoyu Wang Bowang28@albany.ode Jing Gao <jinggao Biinggo@@purdue.com purdue.edu>>
Proceedings of the 4rnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Mitigating Hetcrogencous Token Overfitting in LLM Knowledge Editing
in maintaining high locality (Zhang et al., 2024c).
possible. The theoretical advantage of our method lies in three folds. First, our solution induces negligible computa- tion cost compared to standard training (much cheaper than LLM forward). Second, our solution provides a better pa- rameter update through the lens of importance function (Koh & Liang, 2017). Finally, OVERTONE has a close connec- tion to direct preference optimization (DPO), a widely-used framework for LLM post-training (Rafailov et al., 2024; Zhang al., 2024d), but does not require additional pref- erence data pairs. Sec 3 covers these aspects in details. The proposed OVERTONE and our theoretical analysis is another main technical contribution of this work. Remark- ably, OVERTONE can be of interest to other tasks such as machine unlearning, where selective updates of LLMs are desired. Moreover, when the training text is long, as the number of tokens to learn grows, we expect HTO to exacerbate, and OVERTONE to be helpful.
Nevertheless, existing methods still suffered from losing LLM generalizability, especially when dealing with tasks that involve the edited knowledge, due to the so-called over- fitting of KE (Zhang et al., 2024a). Specifically, KE often involves one piece of new knowledge to edit at a time, which entails updating (selected) parameters with single training instance. Consequently, edited LLMs tend to pay excessive attention to the edited subject, but fail to reason about the new knowledge (Zhong et al., 2023; Zhang et al., 2024a). Previous works highlighted this challenge, and quantified this ability with a new metric known as portability (Zhong et al., 2023; Wang et al., 2024f). However, the underlying causes of overfitting and their relationship to the KE pro- cess remain under-explored, leaving if KE overfitting can be solved in principled manner open question.
In this work, we take the first step toward a deeper under- standing of this overfitting, and pave the way for a princi- pled solution to mitigate it. We first provide strong evidence that KE overfitting leads to catastrophic degradation of an LLM's reasoning ability. In particular, we showed that as the LLM is edited with new knowledge, the probability of correct reasoning consistently decreases. To quantify this, we investigated the portability loss at each fine-tuning step (lower indicates better reasoning ability). We observed that while portability loss initially decreased, it grew up quickly thereafter. In addition, the final loss was significantly higher than the initial value. This finding confirms that overfitting a direct cause of suboptimal portability.
Our paper is organized as follows. Sec 2 and Sec 3 details the new overfitting phenomenon in KE and our proposed OVERTONE for mitigation respectively. Extensive experi- mental results in Sec 4 demonstrate the superiority of our solution. In the remaining part of this paper, we review related works in Sec 5, and conclude the paper in Sec 6.
2. Overfitting Issue in Knowledge Editing
This section presents a new token-dependent overfitting phenomenon in knowledge editing (KE) that has been over- looked in the literature. Background of KE is also provided.
To understand this overlitting, we checked how new knowl- edge is fitted during the KE process. Based on our findings, KE may only require learning a few pivotal tokens (words), as many tokens already exhibit small initial loss values. In- tuitively, an LLM's pre-trained knowledge may enable it to infer remaining parts base on pivotal tokens. However, existing methods overlook this token-level difference in KE. Even when selectively updating parameters, these meth- ods aim to maximize the likelihoods of the entire sentence describing the new knowledge, which boils down to maxi- mizing the probability of all tokens indiscriminately (Ben- gio et al., 2000; Radford et al., 2019; Brown et al., 2020). As result, this coarse- grained training paradigm leads to varying degrees of overfitting across tokens. We term this phenomenon heterogeneous token overfitting (HTO) in KE. Sec 2 details our new insight on KE overfitting and its influ- ence on portability. This is our first main contribution.
2.1. Preliminaries
Given a text x = (Z1,..., where each V; E V is a token from vocabulary V, large language model (LLM) parameterized by 0 computes probability (r(2)) based on chain rule (Bengio et al., 2000):
(T(x)= = II (o(a Z1, 1=1
To mess),
where na( (2 is the predicted distribution of token in given previous Tzgi. The LLM is usually trained with max- imum likelihood estimation (Hochreiter, 1997; Sutskever, 2014; Cho et al., 2014). To generate a sentence œ, the LLM computes To To(0; | æes)) and draws v, from it; then x; is combined with æ<s as new inputs for future steps. This pro- cess completes if a special token that marks the end of the sentence is returned, or if the maximum length is reached.
In light of how HTO roots at a token level, we propose OVERTONE, a new KE training paradigm to tackle it. OVERTONE assigns each token an adaptive training target according its (over)fitting state, An efficient solution is proposed to construct these training objectives in dynamic way that allows to maintain much pre-trained knowledge if
Knowledge Editing (KE) aims to update specific knowl- edge in pre-trained L.LM while preserving unrelated others. A knowledge can be represented by natural language (æ, y), x describes the subject and relation, and y entails corre- sponding object. For instance, suppose x is The president of United States is, y can be Donald Trump. KE asks the
2

---

Mitigating Hetcrogencous Token Overfitting in LLM Knowledge Editing
LLM respond given x with new y, while satisfying the following criteria meanwhile (Zhang al., 2024c): (1) Gen- erality: the edited model should generalize to all equivalent inquires about the US president. (2) Portability: questions reasoned from the new knowledge such as the first lady of United States should be answered correctly. (3) Locality: unrelated knowledge such as the prime minister of Canada should be unchanged. These requirements of precisely up- dating specific knowledge proves non-trivial (Wang al., 2023b; Zhang et al., 2024c).
2.3. Heterogeneous Token Overfitting
15.0
12.5
15
10.0
7.5
5.0
2.2. Overfitting in Knowledge Editing
Positions
Positions
In response precise KE requirements, existing attempts restrict the updates to only minimal amount of parameters. This design establishes remarkable progress in maintain- ing good locality (Zhang et al., 2024c: Wang et al., 2024d). However, it proves insufficient to maintain good general- izability (generalilty and portability) due to the so-called overfitting issue (Zhong et al., 2023; Zhang et al., 2024a).
(a) Initial loss.
(b) Underfitting degree (UD).
Figure 2. Token-level initial loss and UD (negative indicates over- fitted). Dashed lines mark the mean values.
Towards a deeper understanding of this overfitting phe- nomenon, we check the loss of each token, and find that different tokens tend to have distinct initial loss values. As depicted in Fig 2a, before editing LLaMA2, only certain tokens (e.g,, the beginning) have significant loss values. On the other hand, some tokens take small loss value and are initially-fitter by nature. As an intuitive explanation, con- sider the previous US president example. No matter a user wants to edit the answer to Donald Trump or Joe Biden, after seeing the first word Donald or Joe a hint, the LLM is expected to be capable of infer the remaining part based on its pretrained knowledge.
Namely, many KE tasks involve one piece of new knowledge at time, requiring to fine-tune an LLM on single training instance. In such challenging scenarios, the LLM often encounters severe overfitting even only a few parameters are updated. This greatly restricts its ability to generalize the edited knowledge. As shown Zhong et al. (2023); Zhang al. (2024a), edited LLMs usually pay excessive attention to the edited subject, but fail to address multi-hop reasoning questions involving the new knowledge. As result, this limitation results in suboptimal portability.
Nonetheless, existing KE methods overiook this token-level difference. Consequently, they tend to overfit tokens that have varied losses at different speeds. For verification, we compute the pre-edited ligeeiiiddood of tokens generated by the model with greedy decoding, and that of the editing instance during the KE process. Note that our choice of greedy decoding is on purpose, as it reflects the unedited model's most confident knowledge proper that was valid in the past. By comparing the loss of the two, we can measure if token is overfitted. Specifically, we define underfitting degree (UD) as the difference between the pre-edited and running log-likelisoods. Here negative UD indicates an overfitting. Fig 2b shows UD of different tokens when half of them are overfitted. Strong pattern of UD varies across different tokens confirms our concern. We dub this issue as heterogeneous token overfilling (HTO) of KE.
3.0
Train
Tain
1.0
3.0
0.0
8.5
4.0
3
10
15
23
20
15
20
25
Baraition
Eteratton
Figure 1. Loss (average) change of ground truth answers to gener- ality (rephrased, left) and portability (reasoning, right) questions.
As a direct evidence, Fig shows the change of generality and portability loss! at different iterations from fine-tuning LLaMA2 7B (Touvron et al., 2023) with LoRA, a represen- tative KE baseline method (Zhang al., 2024c). As the training goes on, the generality loss decreases. However, the portability loss decreases at the beginning of training, but starts to increase later. This confirms the existence of overfitting. More importantly, the ultimate portability loss significantly larger than before editing, indicating that the reasoning ability is in fact undermined by the KE process,
HTO's s direct cause lies in the training paradigm. Formally, given editing instance (xz y = [y1..... vmm)) where y con- tains m tokens, many KE methods resort to conventional LLM training objective?. In particular, they seek to maxi- mize likelihood of n (y x) by minimizing an averaged
We restrict our study the widely-used teacher-forcing mech- anism (Lamb et al., 2016).
"The perplexity loss of the ground truth answer to question.
3

---

Mitigating Hetcrogencous Token Overfitting in LLM Knowledge Editing
cross-entropy (CE) loss with gradient descent
to incorrect knowledge. Second, compared uniform prior that smooths all tokens equally, the model's own pre-trained knowledge is better prior to help mitigate forgetting prob- lem (Zhang & Sabuncu, 2020; Lee et al., 2022).
m
(CE(0) CE (v)]| x + yell
(1)
In light of the two principles, we use by. and the LLM's current knowledge from its predicted distribution x to con- struct target uar However, as will be verified later, directly use अe can be suboptimal due to the non-negligible noise carries (Hewitt et al., 2022; Tang et al., 2024). Specifically, Tang et al. (2024) argued that wy mixes a distinct subset of informative tokens, and a subset of noisy tokens associating with small logits that fall outside no dis-atttnt away from the maximal value. By filtering out noisy tokens in Te. the LLM performance can be boosted at inference time. We bring this insight to the training (editing) phase and mix the filtered (e)
log nT (y: (c)
178
VolCE(0) =
Vo log To(ss c.).
Here c; G yi denotes the context for token Vi, (v) is the Kronecker delta function', and CE[-II-] computes CE between two distributions.
CE
During training. gradient (olCE(() maximizes the proba- bility of y whiling minimizing the probabilities of all other candidates. When the model is repeatedly updated using gra- dient(s) from the single datapoint, as in KE, the probabilities of initially-fitter tokens become disproportionately large, while tokens with high initial loss values are gradually fit- ted. That is to say, HTO lies in indiscriminately optimizing CE loss of all tokens, without considering their difference. Existing attempts for mitigating overfitting such as early stopping (Yao et al., 2007) and label smoothing (Szegedy et al., 2016; Müller et al., 2019) also ignore this token-level difference, making them conceptually less suitable for HTO.
distribution 4 Fm
with 6g. by
can tar
No,, + (1 - A)m if = argmax, "ur
can
(5) Tiar
6g.
otherwise,
where À is a hyper-parameter. Namely, we adopt the candi- date mixture The if it correctly assigns the maximal prob- ability to ys. otherwise, we skip the mixing and use 6g.- This skip mechanism helps reduce potential knowledge con- flicts by discarding *M
(() (from πT) when it heavily relies on outdated knowledge, which often happens in the first few training steps, empirical benefit is shown in Sec 4.4. Algo I outlines the process of our solution.
3. Propose Method
Given the importance of token-level difference in HTO, we propose OVERTONE to offer granular control that applies to various KE methods, theoretical analysis is also provided.
3.2. Theoretical Advantages of OVERTONE
This section provides theoretical analysis on key factors that merit OVERTONE for KE. All proof's and more in-depth technical background are deferred to App A.
3.1. Counteract HTO with OVERTONE
We present OVERTONE, a token-level strategy for HTO mitigation. Our method smooths y's distribution for fit- ting in an adaptive way. Specifically, we replace each delta distribution ōg.(v) (v) with unique smoothed targer distribu- tion Tur G.). and refine the cross entropy by a clipped forward KL divergence. Our complete loss is given by
Merit 1. OVERTONE is universal and efficient.
While seemingly distinct, OVERTONE is in fact general- ization of CE loss. Moreover, our choice of गur makes it computationally efficient, with computation overhead negli- gible compared to LLM forward operation.
LOVERTONE(()) A m max((Da((Tmall (i)y c)),,e)
Proposition 3.1. OVERTONE loss generalizes CE loss and reduces to the latter when e 0, 1
(2)
Proposition 3.2. Using Alg 1, the additional computation complexity induced by OVERTONE is O(VV) when fitting token, where [V] is the vocabulary size.
where clipped ea imposes token-level early stop- ping when predicted mo is close enough to Mar-
Merit 2. OVERTONE provides better updates.
Principles of Taar design. We note that two principles should be met in order to make Tar a good distribution to target on. First, rur should convey that ground truth to- ken y: most probable, otherwise, the objective may lead
OVERTONE leads to more effective parameter updates, as demonstrated through the lens of the influence function (Koh
*For brevity mi Tn c2), Aue is defined similarly. Plain wm and Tur will be used when discussing the general idea.
(5)
ky (v) = 1 if y = y else O.

---

Mitigating Hetcrogencous Token Overfitting in LLM Knowledge Editing
Algorithm 1 OVERTONE Training Paradigm
words, OVERTONE can be seamlessly combined with exist- ing constrained optimization-base solutions for KE
Input: Editing data (z.y [y1..... Mml). LM pa- rameters lb. mixing hyper-parameter X, early-stopping threshold filtering threshold total training steps T.
Below theorem draws connection between OVERTONE and direct preference optimization (DPO), which has shown superior performance maintaining pretrained knowledge in LLM post-training (Wang et al., 2023a).
2: Initialize: 0=0g On-
3: l..t= T'.ddo do
4: # Inner loop is parallelized in practice, unroll for better readability.
Theorem 3.4. Lel e = O, optimizing OVERTONE can be seen as optimizing an unbiased estimate of a DPO objective plus some additional KI. penalty.
5: iori=
6: Set context C;
2 yei
7:
Compute logits from the LM as ( fo(c:) E
Compared with conducting explicit DPO, OVERTONE does not require collecting preference data, and is more efficient thereof. Furthermore, as highlighted in Rozner et al. (2024), another challenge of applying DPO to KE is that determin- ing win-loss data pairs can be unstraightforward in KE. In contrast, OVERTONE walks around this challenge by re- fraining from treating any token as unpreferred, and instead acts on a distribution level.
RIV. Take softmax and get =g (4)
Top no nolteilter (Tang al., 2024): Compute (4)
8:
Smax
std(s(() Define filtered logit % 00 if 8k S Smax no else 5 = (1)
max; (4)),
Take softmax on filtered and get filtered "m (2)
9:
10:
Compute target "ar based on Eq (3).
(+)
11:
Compute loss
e0 OVERRTOEE max (aI. Tear I| [ I.c.
4. Experiments
We evaluate the proposed OVERTONE paradigm four performant KE methods applying to two representative large language models (LMs) over five benchmarking datasets. Ablation studies are also conducted to help understand its effectiveness. Results show that OVERTONE helps improve editing performance by large margin on all methods. More conceptual discussions can be found in Appendix D.
12:
end for
Compute sample loss
LOVERTONE
ai) Σ
OVERTONE
14:
Update with learning rate a
θ レ A - --0-aVolovetton
4.1. Experiment Setup
Base Models. We conduct experiments on two representa- tive LMs, LLaMA 2-7b-Chat (Touvron et al., 2023) and LLaMA Inst-unst (Dubey et al., 2024), which have been widely studied in the literature (Zhang et al., 2024c; Wang et al., 2024d). From now on, we refer to the two LMs as LLaMA 2 and LLaMA 3 for brevity.
15: end for output Edited parameter 0.
Liang, 2017), outlined in the following informal theorem. Due to page limitations, the formal version and correspond- ing assumptions are deferred to Appendix A.3.
Tasks. Following Wang et al. (2023b); Zhang et al. (2024c), we edit different kinds of knowledge: WikiData WikiData,comeertett (Cohen et al., 2024), WikiBio (Hartvigsen et al., 2024), and ZsRE (Yao et al., 2023). Besides the four popular benchmarks, we also explore more complex MQUAKE (Zhong et al., 2023; Wang et al., 2024f). Due to page limitation, we refer readers to Zhang et al. (2024c) for more benchmark details. When editing an LLM,, we consider two scenarios: (1) Single Editing: one piece of knowledge is edited a time. (2) Continual Editing: mul- tiple pieces of knowledge This is more challenging due to forgetting and knowledge conflicting (Hartvigsen et al., 2024; Wang et al., 2024d).
Theorem 3.3 (Informal). Under regularity conditions, com- pared to optimizing the vanilla CE loss, OVERTONE pro- vides more favorable update direction for the parameters and has less influence on unrelated knowledge.
Merit 3. OVERTONE has close connection to DPO and other constrained optimizations.
One might question whether OveRTONe is conceptually su- perior to constrained optimization approaches, such fine- tuning only a small set of specific parameters (Dong al., 2022a; Dai et al., 2021), limiting update magnitudes (Zhu et al., 2020), or employing low-rank updates (Hu et al., 2021). We emphasize that OVERTONE introduces a new objective that can be solved with any optimization meth- ods, regardless of whether constraints are imposed. other
edited in sequential way.
Editing Methods. We apply OVERTONE to four representa- tive KE methods from different families that have achieved state-of-the-an performance (Zhang et al., 2024c; Wang
5

---

Mitigating Hetcrogencous Token Overfitting in LLM Knowledge Editing
al., 2024e). FT-M (Zhang et al., 2024c) fine-tunes a spe- cial layer identified by causal-tracing analysis wherein the knowledge is stored. LoRA (Hu et al., 2021) learns additive low-rank updates for model parameters on the new knowl- edge. MELO (Yu et al., 2024) and WISE (Wang et al., 2024d) incorporates additional parameter copies to Iearn new knowledge, along with some gating mechanism to deter- mine whether original or new knowledge should be used at inference time. Despite incorporating certain explicit or im- plicit constraints on the learnable parameters, these methods are all trained to minimize the CE loss. For better bench- marking, we also report results from two widely-studied methods ROME (Meng et al., 2022a) and MEMIT (Meng et al., 2022b). ROME applies a causal-tracing analysis to identify the layer wherein the knowledge is stored and then solves an analytic rank-one update, and MEMIT extends ROME by identifying series of layers to edit and finding the updates as least-squares solutions. To reflect the chal- lenging nature of KE under data scarcity regime, we focus on KE methods that do not require a larges-scale hard-to- access training data, or training additional models. No data augmentation were applied during the editing.
see this, note that when editing LLAMA 2 on ZsRE, LoRA reached portability that was nearly three times of the base version. Similarly, MELO also reached an almost dou- bled portability. More evidence can be found from editing LLaMA 3 as well. In addition, all methods, especially those initially fall short in maintaining good locality, achieved ex- cellent performance in this regard. As an evidence, L'R reached a nearly five times locality improvements when editing both LLaMA 2 and LLaMA 3 Wikicounerfaat We want to highlight that, all these improvements were made without compromising editing reliability. That is to say, all the four methods achieved better trade-offs between re- liability and reasoning (and locality) from the proposed OVERTONE. More importantly, this success was established in a model-agnostic manner, in the sense that OVERTONE not specialized for any particular KE method studied here. Instead, it offers a highly flexible and generie paradigm that can be combined with existing solutions in a plug-and-play manner.
More Complex Editing task. To further evaluate how OVERTONE performs on complex benchmark in the filed of KE, we test FT-M and LoRA with editing the two LLMs MQuAKE-2202 (Wang et al., 2024f)², following Zhong et al. (2023). This task requires the edited LLM to answer single- and multi-hop reasoning questions about the edited knowledge. Experiment results are reported in Table 2. As before, OVERTONE was capable of achieving better portability without hurting the editing performance.
Evaluation Criteria. We evaluate the performance from four aspects as discussed in Sec 2: reliability (Rel.), gener- ality (Gen.), portability (Por.), and locality (Loc.). Due to page limits we refer readers to Zhang et al. (2024c); Wang et al. (2024d) for their formulations. We report the average of different metrics for more complete comparisons.
Implementation Details. All of our experiments are imple- mented in EasyEdit (Wang et al., 2024e). More details and hyper-parameters can be found in App B.
These empirical results echo well with our theoretical anal- ysis, and confirm the superiority of OVERTONE.
4.3. Continual Editing Performance
4.2. Single Editing Performance
We next study the more challenging scenarios, where mas- sive edits are conducted in a continual (sequential) way. Experiments were again run on the four benchmarks.
We evaluate the effectiveness of OVERTONE in conducting Single Editing on ZsRE, WikiData,semm WikiDattaounterfeet and WikiBio with different KE methods. WISE was tested on ZsRE, the only benchmark that contains additional irrel- evant data during the editing time that is required by WISE.
Due to page limit, We defer the complete results to App C, and visualize the average of reliability, generality, porta- bility, and locality in Fig 3. Specifically, we evaluate the performance after new T pieces of knowledge length are edited sequentially. Different KE methods are represented in separate colors. Solid boxes indicate normal training performance, and transparent boxes show results from train- ing with OVERTONE. The unfilled area within the boxes quantifies the improvements form OVERTONE.
Single Editing results are reported in Tab From the ta- ble, all KE methods gained significant improvement from the proposed OVERToNE paradigm. Specifically, The four methods hardly performed comparable to baselines ROME and MEMIT from normal training, but were capable of ex- ceeding them when trained with OVERTONE. For instance, without OVERTONE, ROME achieved the highest and the second-highest average performance for editing LLaMA 2 and LLaMA respectively on Wikigeent- However, when equipped with OVERTONE, FT-M, LoRA, and MELO out- performed ROME on both tasks.
As in Single Editing scenarios, OVERTONE again improved the performance of four KE methods, enabling them to surpass ROME and MEMIT by large margin across di- verse settings. Furthermore, on three out of the four bench- marks (ZsRE, Wikineeets, and Wikiconnterfae), the improve- We next check where the improvement was made. From the table, the first gain was from improved portability. To
"This is cleaned version of MQuAKE by fixing knowledge confficts (Wang et al., 2024f).
6

---

Mitigating Hetcrogencous Token Overfitting in LLM Knowledge Editing
Table 1. Single Editing performance. Four KF methods gained improvement from OVERTONE training paradigm. WISE requires additional irrelevant data for training, which is only available in ZsRE benchmark.
ZsRE
yeke
oikleounterfct
WikiBio
LLaMA 2-7b-chat
Rel.
Gen.
Por.
Loc.
Avg.
Rel.
Por.
Loc.
Avg.
Rel.
Por.
Loc.
Avg.
Rel.
Loc.
Avg.
ROME
96.61
83.91
55.7
96.96
83.3
99.02
54.21
55.91
69.71
97.2
56.85
50.4
68.15
96.41
59.14
77.78
MEMIT
94.22
88.2
57.91
98.28
84.65
97.71
52.93
55.05
68.56
96.38
59.34
45.7
67.14
93.78
56.74
75.26
FT-M
99.75
99.33
54.32
93.01
86.60
100.0
62.93
45.92
69.62
100.0
74.7
54.86
76.52
100.0
90.04
95.02
Ours
99.75
96.8
57.08
96.54
87.54
100.0
63.91
60.4
74.77
100.0
73.62
75.34
82.99
100.0
93.46
96.73
LoRA
100.0
100.0
23.34
30.44
63.45
100.0
55.41
28.29
61.23
100.0
71.92
9.99
660.64
100.0
48.84
74.42
+ Ours
100.0
94.31
61.16
87.2
85.67
100.0
63.67
58.72
74.13
100.0
73.96
57.85
77.27
97.68
68.45
83.06
MELO
100.0
96.77
27.11
92.35
79.06
99.13
54.04
40.96
64.71
99.0
71.78
55.83
75.54
99.97
80.77
90.37
+ +Ours
100.0
93.31
50.36
97.2
85.22
100.0
60.25
66.48
75.58
99.91
71.81
78.09
83.27
99.68
82.58
91.13
WISE
92.42
70.86
54.57
100.0
79.46
+ +Ouss
97.55 76.09
54.17
100.0
81.95
LLaMA 3-8h-Instruct
Rel.
Gen.
Por.
Loc.
Avg.
Rel.
Por:
Loc.
Avg.
Rel.
Por.
Loc.
Avg.
Rel.
Loc.
Avg.
ROME
99.17
97.91
58.12
95.9
87.78
98.84
54.76
49.74
67.78
99,94
58.0
42.94
66.96
92.43
72.63
82.53
MEMIT
96.67
92.46
58.78
98.23
86.53
98.51
53.65
48.45
66.87
99.44
57.81
42.73
66.66
96.26
71.23
83.75
FT-M
100.0
99.75
40.43
79.43"
79.90
100.0
57.13
30.01
62.38
100.0
72.62
31.47
668.03
100.0
92.96
96.48
+ Ours
100.0
99.75
48.63
94.78
85.79
100.0
60.88
44.67
68.52
100.0
73.5
58.29
77.26
99.99
94.87
97.43
LoRA
100.0
100.0
26.55
38.85
66.35
100.0
52.99
26.46
59.82
100.0
71.1
9.02
60.04
100.0
59.77
79.88
+ Ours
100.0
98.5
51.57
93.13
85,80
100.0
61.46
56.1
72.52
100.0
72.8
37.54
76.78
98. 16
77.24
87.7
MELO
100.0
96.84
39.63
98.8
83.82
100.0
59.07
65.78
74.95
100.0
71.55
87.77
86.44
100.0
98.56
99.28
Ours
100.0
95.77
43.08
98.8
84.41
100.0
58.72
69.1
75.94
100.0
70.26
89.81
86.69
99.98
98.56
99.27
WISE
71.67
$1.29
49.27
100.0
68.06
+Ours
82.67
62.34
47.54
100.0
73.14
following findings. First, pure token-level smoothing ("w/o clip") increases both portability and locality, confirming that overfiting due to CE loss indeed hurts editing performance. Additionally, the way to smooth target distribution plays a critical role: using the unedited predicted distributions ("w/o dyn-wm.") ") leads to significant drop, due to the conflicts raise from the outdated internal knowledge. Extra evidence can be seen from ("w/o chk- ")) where the mixture (Eq (3)) is always applied without checking if the probability of label Yi is the largest. Finally, the noise in predicted distribution my also hinders the editing process: without filtering them out ("w/o fit-Tn.)) "), both generality and portability decreased. All empirical results aligns well with our analysis in Sec 3.
Table 2. Editing performance on MQUAKE.
LLaMA 2-7b-chat
LLaMA 3-80-Instruct
Rel.
Sng-Hop. Mit-Hlpp
Avg.
Rel.
Sug-Hop.
Mit-Hop.
Avg.
FT-M
100.0
83.0
30.0
71.0
100.0
82.0
24.0
68.67
Ours
99.86
89.0
37.0
75.29
100.0
85.0
30.0
71.67
LoRA
106.0
95.0
39.0
78.0"
100.0
98.0
35.00
77.67
Ours
99.75
93.0
48.0
80.25
100.0
95.0
40.0
78.33
ments were even more pronounced when the editing se- quence is longer (T 10, 100). Notably, according to our results on ZsRE, LoRA (and FT-M) achieved highly competitive continual editing performance when enhanced with OVERTONE, on par with specialized continual editing methods like MELO and WISE. In contrast, in previous works (Zhang et al., 2024c; Wang al., 2024d), vanilla LoRA is generally considered unsuitable for continual edit- ing unless significant adaptations are implemented.
Table 3. Ablation studies on OVERTONE, "w/o clip" sets O, "w/o dyn- TT= " uses unedited prediction. "w/o chk-mm" always adopt the mixture in Eq (3), "w/o fit- uses full Ta without filtering out tail (noisy) regions.
To conclude, these results clear demonstrated the flexibility and power of OVERTONE in diverse KE scenarios.
LLaMA 2-7b-chat
Rel.
Gen.
Por.
Loc.
Avg.
4.4. A Ablation Studies
LoRA
100.0
100.0
23.34
30.44
63.45
w/o clip
100.0
99.75
26.6
41.08
66.86
We end this section with an ablation study on OVERTONE to showcase how each component contributes to its final perfor- mance. Results from editing LLaMA 2 on ZsRE with LoRA are presented in Tab According to the table, we note the
w/o dyn-Fa
99.18
97.67
36.32
$1.57
71.18
w/o chk--a
95.35
86.51
57.92
90.08
82.47
w/o fit-T
100.0
83.93
58.2
90.36
83.12
Ours
100.0
94.31
61.16
87.2
85.67
7

---

Mitigating Hetcrogencous Token Overfitting in LLM Knowledge Editing
HELL
141
(a) LLaMA 2 (ZsRB)
(b) LLaMA 2 (Wikireeen)
(c) LLaMA 2 Wikicounterfheel counterfact
(d) LLaMA (WikiBio)
3G4
34
3+1
3-8
15
(e) LLaMA 3 (ZsRE)
(f) LLaMA 3 (Wiklrrent)
(g) LLaMA (Wikdewunterfae)
(h) L.LMM (WikiBio)
Figure 3. Continual Editing performance under different sequence length T. Solid and transparent bars show performance with and without OVERTONE. Unfilled area marks the performance gap. ROME and MEMIT didn't use OVERTONE.
5. Related Works
performance of two representative internal stage methods across diverse scenarios.
Existing KE methods mainly fall into two classes.
External Storage resorts to external memories without up- dating original parameters. This category includes meta- learning- cearning-based MEND (Mitchell et al.. 2021) and its multi- task varient InstructEdit (Zhang et al., 2024b), in-context learning-based IKE (Zheng et al., 2023), retrieval-based LTE (Jiang et al., 2024), augmentation -based StableKE (Wei et al., 2024), and proxy model-based SERAC (Mitchell et al., 2022). Notwithstanding, these methods often re- quire large-scale, hand-to-access dataset for retrieval (e.g.. IKE, LTE) as in retrieval-augmenned generation (RAG, (Gao et al., 2023; Wang al., 2024b; Xu et al., 2024; Yu et al., 2025; Liu al., 2025a; Xu et al., 2025)), or for training auxiliary models (e.g., MEND, InstructEdit, SERAC). As a result, their practicality is limited, and they struggle with Continual Editing that needs frequent up- dates (Wang et al., 2024d). Recently, specialized meth- ods for Continual Editing have been proposed. These ap- proaches introduce adapters (GRACE (Hartvigsen al., 2024)), LoRAs (MELO (Yu al., 2024)), weight copies (WISE (Wang et al., 2024d)) to memorize new knowledge, and learn gating mechanism to determine whether to use original or new knowledge. The gating mechanisms are of- ten learned through additional representation sirioondddtttnne--bueee codebooks (Yu et al., 2024) or distinct margin losses (Wang et al., 2024d), making external storage methods more com- plex. However, like internal storage methods, they optimize editing parameters using instance-level loss functions, ig- noring token-level differences. Consequently. they may also suffer from HTO and can benefit from our OVERTOR TONE
Internal Storage updates model parameters for the adapta- tion. Early studies fine-tuned LLM directly but suffered from severe forgetting problem (Wang et al., 2023b). For more precise editing, Zhu et al. (2020) imposed a relaxed l2 norm constraint on parameter updates, and Dong et al. (2022a); Huang al. (2023) limited the updates to some specific feed-forward network (FFN) layer(s), based on find- ings that knowledge is often stored therein (Dai et al., 2021). For further refinement, the locate-and-ed paradigm (Meng et al., 2022a;b) first identifies the layer storing the knowl- edge, then modifies its parameters in an analytic form or through least squared solution. On the other hand, PEFT methods such LoRA- (Hu al., 2021; Wang et al., 2024c) and ReFT-family (Wu et al., 2024; Liu et al., 2025b) also performed competitive to locating-bar based solutions (Wu et al., 2023; Zhang et al., 2024c). In general, these works pri- marily focus on identifying small set of parameters most relevant to the new knowledge. However, these approaches are typically trained with instance-level loss, overlooking the token-level differences. Therefore, they remain suscep- tible HTO in similar manner and cannot be mitigated by advanced PEFT methods (Chen et al., 2024; Miao et al., 2025). This work addresses HTO, orthogonal aspect of the KE process, and complements existing studies in a model-agnostic manner. Our OVERTONE established without assumptions about which parameters are updated, al- lowing it to seamlessly integrated with existing methods without compromising their selective nature. We validate approach by showing that OVERTONE enhances the
8

---

Mitigating Hetcrogencous Token Overfitting in LLM Knowledge Editing
framework. Experiments with two external storage methods demonstrate that solution can be straightforwardly in- corporated to more complex KE methods, highlighting the flexibility and versatility of OVERTONE.
References
Bengio, Y., Ducharme, R., and Vincent, P. A neural proba- bilistic language model. Advances in neural information processing systems, 13, 2000.
Overfitting and Mitigation Recent works have identified different forms of KE overfitting and proposed respective mitigation solutions. Namely. Zhang al. (2024a); Qi et al. (2024) use in-context prompted distribution as the target distribution to fit, which helps improve generalizabil- ity (Lampinen et al., 2025), and Ma et al. (2024) focuses neighboring knowledge perturbation due to the answer-level overfitting. In this work, we focus on understanding and developing generalizable KE. Unlike existing methods, our OVERToNE resorts the model's own prediction to maintain its pretrained knowledge through an adaptive token-level distribution mixing and early stopping, in light of the token- level HTO dynamic.
Bishop, C. M. and Nasrabadi, N. M. Pattern recognition and machine learning, volume 4. Springer, 2006.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2 2021.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
6. Conclusion
Bubeck, S., Chandrasckaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. Sparks of artificial general intel- ligence: Early experiments with gpt-4. arXiv preprint [tti:2303.12711,, 2023.
We study HTO, a token-dependent overfitting in KE, and show how it degrades an edited LLM's 's reasoning ability. Inspired by an in-depth analysis on its cause, we propose OVERTONE, which adaptively assigns each token unique smoothed distribution for better control to mitigate HTO. Our solution enjoys several theoretical advantages, and achieves superior performance on diverse tasks. Encour- aged by these promising results, we plan to work on the following directions in our future work. The first direction is to understand how HTO will act on broader KE meth- ods that involves more specialized losses when facing free-form editing data. The second topic we would like to explore is to unify HTO and other types of KE overfitting, thereby providing a more universal solution. Finally, we advocate for more rigorous experimental design within the KE sweenntt--eeeeiclallyale conducting multiple runs per editing instance-to ensure statistically reliable results.
Chen, W., Miao, Z., and Qiu. Q. Large convolu- tional model tuning via filter subspace. arXiv preprint arXiv:2403.002. 2024.
Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F. Schwenk, H., and Bengio, Y. Learn- ing phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406 1078, 2014.
Cohen, R., Biran, E., Yoran, O., Globerson, A., and Geva, M. Evaluating the ripple effects of knowledge editing in language models. Trunsuctions of the Association for Computational Linguistics. 12:283-298. 2024.
Cover, T. M. Elements of information theory. John Wiley Sons, 1999.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Dai, D., Dong, L.,, Hao, Y., Sui, Z., Chang, B., and Wei, F. Knowledge neurons in pretrained transformers. arXiv preprint arXi::2104.0866 2021.
De Cao. N., Aziz, W., and Titov, I. Editing fac- tual knowledge in language models. arXiv preprint arXiv:2104.08164 2021.
Acknowledgment
This work is supported in part by the US National Science Foundation under grant NSF 11$-2141037. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
Dong, Q., Dai, D., Song. Y., Xu, Sui, Z., and Li, L. Cali- brating factual knowledge in pretrained language models. arXiv preprint arXiv:2210.03322, 2022a.
Dong, Q., Li, L., Dai, D., Zheng, C., Wu, Z., Chang, B., Sun, X., Xu, J., and Sui, Z. A survey for in-context learning. arXiv preprint arXiv:2301.00244 2022b.
9

---

Mitigating Hetcrogencous Token Overfitting in LLM Knowledge Editing
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models, 2024. URL https: // //ariiv.. org/ abs/ 2407 21783.
Lampinen, A. K., Chaudhry, A., Chan, S. C., Wild, C., Wan, D., Ku, A., Bornschein, J., Pascanu, R., Shanahan, M., and McClelland, J. L. On the generalization of language models from in-context learning and finetuning: con- trolled study. arXiv preprint 2xXiv:550500066 2025.
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J. Bi, Y. Dai, Y., Sun, J., Wang, H., and Wang, H. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312. 10997, 2, 2023.
Lee, D., Cheung, K. C., and Zhang, N. L.. Adaptive la- bel smoothing with self-knowledge in natural language generation. arXiv preprint arXiv:2210. 13459, 2022.
Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., and Kamar, E. Toxigen: A large-scale mnchine-generated dataset for adversarial and implicit hate speech detection. arXiv preprint arXiv:2203.09509 2022.
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mo- hamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. Bart: Denoising sequence-tor seriennceto--equeneee pre-training for natural language generation, translation, and comprehen- sion. arXiv preprint arXiv:
Hartvigsen, T., Sankaranarayanan, S., Palangi, H., Kim, Y., and Ghassemi, M. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, 36, 2024.
Liu, T., Jiang, H., Wang, T., Xu, R., Yu, Y., Zhang, L., Zhao, T., and Wang, H. Roserag: Robust retrieval- augmented generation with small-scale llms via margin-aware pref- erence optimization. arXiv preprint arXiv:2502. 10993, 2025a.
Hewitt, J., Manning, C. D., and Liang, P. Truncation sam- pling as language model desmoothing. arXiv preprint arXiv:2210.15191, 2022.
Liu, T., Li, R., Wang, H., Qi, Y., Liu, H., Tang, X., Zheng, T., Yin, Q. Cheng, M. X., Huan, J., and Gao, J. Unlocking efficient, scalable, and continual knowledge editing with basis-level representation fine-tuning. In The Thirteenth International Conference on Learning Representations, 2025b.
Hochreiter, S. Long short-term memory. Neural Computa- lion MIT--ress, 1997.
Hu, E. J., Shen, Y., Wallis, P., Allen- Shun, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021.
J.-Y,, Ling, Z.-H., Zhang, N., and Gu, J.-C. Neighboring perturbations of knowledge editing on large language models. arXiv preprint arXiv:2401. 2024.
Huang, Z., Shen, Y., Zhang, X., Zhou, J., Rong, W., and Xiong, Z. transformer-patcheer One mistake worth one neuron. arXiv preprint arXiv:2301.09755 2023.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. Advances in Neu- ral Information Processing Systems, 35:17359-17372. 2022a.
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, J.,, Madotto, A., and Fung. P. Survey of halluci- nation in natural language generation. ACM Computing Surveys, 55 13(12):1-38, 2023.
Meng, K., Sharma, A. S., Andonian, A., Belinkov, Y., and Bau, D. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229, 2022b.
Jang, Y., Wang, Y., Wu, C., Zhong, W., Zeng, X., Gao, Li, L.,, Jiang, X., Shang, L.,, Tang, R., et al. Learning to edit: Aligning Ilms with knowledge editing. arXiv preprint arXiv:2402.1 41.e22222111199 1105, 2024.
Miao, Z. Chen, W., and Qiu, Q. Coeff-tuning: A graph filter subspace view for tuning attention-basel large models. arXiv preprint arXiv:2503. 18337, 2025.
Koh, P. W. and Liang, P. Understanding black-box predic- tions via influence functions. In International conference machine learning, pp.
Mitchell, E., Lin, C., Bosselut, A., Finn, C., and Man- ning, C. D. Fast model editing at scale. arXiv preprint a 1139, 2021.
PMLR, 2017.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Ad- vances in neural information processing systems, 35: 22199-22213, 2022.
Mitchell, E., Lin, C., Bosselut, A., Manning, C. D., and Finn, C. Memory-based model editing at scale. Inter- national Conference on Machine Learning, pp. 15817- 15831, 2022.
Lamb, A., Goyal, A., Zhang, Y., Zhang, S., Courville, A., and Bengio, Y. Professor forcing: A new algorithm for training recurrent networks, 2016. URL https: /
Müller, R., Korbblihh, S., and Hinton, G. E. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
10

---

Mitigating Hetcrogencous Token Overfitting in LLM Knowledge Editing
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C.,, Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Wang, C., Jiang. Y., Yang. C., Liu, H., and Chen, Y. Beyond reverse KL.: Generalizing direct preference optimization with diverse divergence constraints. In The Twelfth Inter- national Conference on Learning Representations, 2024a.
Wang. H., Li, R., Jiang, H., Tian, J., Wang, Z., Luo, C., Tang, X., Cheng, M., Zhao, T., and Gao, J. Blendfiltr: Advancing retrieval-augmennte large language models via query generation blending and knowledge filterin. arXiv preprini arXiv:2402.1 / 129,22922222 2024b.
Qi. S., Yang. B., Jiang, K., Wang, X., Li, J., Zhong, Y., Yang, Y., and Zheng, 7. In-context editing: Learn- ing knowledge from self-induced distributions. arXiv preprint ar orXiv:2406.11 11194, 2024.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Wang, H., Liu, T., Li, R., Cheng, M., Zhao, T., and Gao. J. Roselora: Row and column-wiss sparse low-rank adapta- tion of pre-trained language model for knowledge editing and fine-tuning. arXiv preprint arXiv:2406. 10777, 2024c.
Rafailov, R., Sharma, A., Mitchell, E., Manning, D. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Wang, P., Li, L., Chen, L., Song, F., Lin, B., Cao, Y., Liu, T., and Sui, Z. Making large language models better rea- soners with alignment. arXiv preprint arXiv:2309. 02 144, 2023a.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, ((CI):5485-5511) 2020.
Wang, P., Li, Z.,, Zhang, N., Xu, Z., Yao, Y., Jiang. Y., Xie, P., Huang, F., and Chen, H. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. arXiv preprint arXiv: 105../47000 /4768, 2024d.
Wang. P., Zhang. N., Tian, B., Xi, Z., Yao, Y., Xu, Z., Wang, M., Mao, S., Wang, X., Cheng, S., Liu, K., Ni, Y., Zheng, G., and Chen, H. Easyedit: An easy-to-use knowl- edge editing framework for large language models, 2024e. URL https: / aataaaxxvvv..o..../.aa abs / 220. 07269.
Rozner, A., Battash, B., Wolf, L., and Lindenbaum, O. Knowledge editing in language models via adapted direct preference optimization. arXiv preprint arXiv:2406.000 200.22406.09920, 2024.
See, A., Pappu, A., Saxena, R., Yerukola, A., and Manning, C. D. Do massively pretrained language models make better storytellers? arXiv preprint arXiv:1909. 10705, 2019.
Wang, S., Zhu, Y., Liu, H., Zheng, Z., Chen, C., al. Knowledge editing for large language models: A survey. arXiv preprint arXiv:2310. 10. 16218, 2023b.
Wang. Y., Chen, M., Peng. N., and Chang, K.W. Deepedit: Knowledge editing as decoding with constraints. arXiv preprint arXiv:2401.1047/ 2024f.
Sutskever, I. Sequence to sequence learning with neural networks. arXiv preprini arXiv: 1409.3215, 2014.
Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vi- sion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain -oinn-oftthhhhhh prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Tang, C., Liu, J., Xu, H., and Huang, I.. Top-no: Not all logits are you need. arXiv preprint ar114:2411.07611, 2024.
Wei, Z., Pang. L., Ding, H., Deng, J., Shen, H., and Cheng, X. Stable knowledge editing large language models. arXiv preprint arXiv:2402. 13048, 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv.2307.09288 2023.
Wu, S., Peng, M., Chen, Y., Su, J., and Sun, M. Eva-kellm: A new benchmark for evaluating knowledge editing of llms. arXiv preprint arXiv:2308. 20999444 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Wu, Z., Arora, A., Wang, Z. Geiger, A., Jurafsky, D., Man- ning, C. D., and Potts, C. Reft: Representation finetuning for language models. arXiv preprint arXiv:2404. 00550299 2024.

---

Mitigating Hetcrogencous Token Overfitting in LLM Knowledge Editing
Xu, R., Liu, H., Nag, S., Dai, Z.,, Xie, Y., Tang, X., Luo, C., Li, Y., Ho, J. C.,, Yang, C., et al. Simrag: Self-improving generation for adapting large lan- guage models to specialized domains. arXiv preprint arXiv:2410. 17952, 2024.
Zhong, Z.., Wu, Z., Manning, C. D., Potts, C., and Chen, D. Mquake: Assessing knowledge editing in lan- guage models via multi-hop questions. arXiv preprint arXiv:2305.1472 2023.
retrieval-augmentn
Zhou, C., Li, Q. Li, C., Yu, J., Liu, Y., Wang, G., Zhang, K., Ji, C., Yan, Q. He, I.,, et A comprehensive survey pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv:2302.094/9 2023.
Xu, R., Shi, W., Zhuang, Y. Yu, Y., Ho, J. C., Wang, H., and Yang. C. Collab-rag: Boosting reureeell-ugmmented generation for complex question answering via white- box and black-box Ilm collaboration. arXiv preprint arXiv:2504.04915. 2025.
Zhu, C., Rawat, S., Zaheer, M., Bhojanapalli, S., Li, D.,
Yu, F., and Kumar, S. Modifying memories in transformer models. arXiv preprint arXiv: arXiv:2012.00363, 2020.
Yao, Y., Rosasco, L., and Caponnetto, A. On early stopping in gradient descent learning. Constructive Approximation, 26(2):289-31, 2007.
Yao, Y., Wang, P., Tian, B., Cheng, S., Li, Z., Deng. S., Chen, H., and Zhang, N. Editing large language models: Problems, methods, and opportunities. arXiv preprini arXiv:2305. 13172, 2023.
Yu, I.,, Chen, Q. Zhou, J., and He, L.. Melo: Enhancing model editing with neuron-indexed dynamic lora. Proceedings of the AAAI Conference on Artificial Intell- 2024.
gence, volume 38, pp.
Yu, Y., Ping, W., Liu, Z., Wang, B., You, J., Zhang, C., Shoeybi, M., and Catanzaro, B. Rankrag: Unifying con- text ranking with retrieval-augie augmented generation in llms. Advances in Neural Information Processing Systems, 37: 121156-121184, 2025.
Zhang, M., Ye, X., Liu, Q. Ren, P., Wu, S., and Chen, Z. Uncovering overfitting large language model editing- arXiv preprint ar arXiv:2400.0811, 2024a.
Zhang, N., Tian, B., Cheng, S., Liang, X., Hu, Y., Xue, K., Gou, Y., Chen, X., and Chen, H. Instructedit: Instruction- based knowledge editing for large language models. arXiv preprint 4002 16123, 2024b.
Zhang, N., Yao, Y., Tian, B., Wang, P., Deng, S., Wang, M., Xi, Z., Mao, S., Zhang, J., Ni, Y., et al. A comprehensive study of knowledge editing for large language models. arXiv preprint 2rXiv:2401.01286, 2024c.
Zhang, R., Lin, L., Bai, Y., and Mei, S. Negative preference optimization: From catastrophic collapse to effective un- learning. arXiv preprint arXiv:2404.055 05866, 2024d.
Zhang, Z. and Sabuncu, M. Self-distillation instance- specific label smoothing. Advances in Neural Information Processing Systems, 33:2184-22 2195, 2020.
Zheng, C., Li, L., Dong, Q., Fan, Y., Wu, Z., Xu, J., and Chang, B. Can we edit factual knowledge by in-context learning? arXiv preprint arXiv:2305. 12740, 2023.
12

---

Mitigating Heterogencous Token Overfitting in LLM Knowledge Editing
A. Omitted Theorems and Proofs
In this section we present the full theoretical analysis. All theorems are (re)stated a formal manner for the convenience of reading.
A.1. Notations
For completeness we highlight important notations that will be used. Throughout this paper, we use CE[.II-] and D [-l--] to compute cross-entropy and -eiblle divergence between two distributions respectively. Specifically, given two discrete distributions p. q. CEppla] E; -P: log qı, and [ Σ. pi log &. In addition, 1(-) the indicator
pi function such that 1 (a) 1 if event a holds and O otherwise. For IR", define the norm ||a||2 V at. For
a, E Re, define the inner product as (a, b) b. Define the cosine similarity cos(a, (a.6)
10
A.2. OVERTONE is universal and efficient
The first merit of OverToNE, stated in the main body, lies in its universality and efficiency.
Proposition A.1. OVERTONE loss generalizes CE loss and reduces to the latter when e = 0, A =
Proposition A.2. Using Alg 1, the additional computation complexity induced by OVERTONE is O(IVI) when fitting token, where [V is the vocabulary size.
Our proofs rely the following lemma, which plays key role in connecting OVERTONE regularized loss.
Lemma A.3. Given yo for an arbitrary token and context c, and Tua (b) kr((y), we have
CE(Ter(y c)][Toly| c)] ACE (w)l c)] + (1 A)CE(TA | cc) (y [c].
(4)
Proof. The proof is based on the definition of cross entropy (Cover, 1999).
CE| CF c)llmo (v c)]
(1 c) log (a(y c)
(A5 (v) + 2) c)) log No c)
1V
I Tai (y c) log To(y c)
8 (v) log Tol c) +1 (1- A)
XCE (1) c)] + - ))CE(Ta(y) c))l c))-
(5)
This completes our proof.
We are ready to prove Prop 3.1.
Proof. The proof based on the fact that OVERTONE objective minimizes a forward KL-divergenne, which equivalent to
13

---

Mitigating Heterogencous Token Overfitting in LLM Knowledge Editing
minimizing cross-entropy (Cover, 1999; Bishop & Nasrabadi, 2006). Namely,
EOVERTONE (0)
(T [4) [T c))l.)
m Σ
D c:)l(TO( c:)] (Dxi. (DrI.(Tarl c; cs)l|=a(y ci)) > e)
j=l
m
(5) + (m )) DKL -uu 11 ( >
CE x tar
TT
CE tar
TT
DKL | ]1
[1 (1)) || > e) +C.
(6)
"g similarly for brevity, C' denotes terms that are constant to learnable parameter 0. Therefore, setting = 0 gets us rid of the indicator term. Further plug in Eq (5), we see that setting reduces to the standard CE loss. This completes the proof.
Starting from step (a), we denote .G) = my c;) and
In terms of Prop 3.2, the computation overhead can be seen by checking Algo 1.
Proof. The additional computation complexity of OVERTONE is due to line 8-10 in Algo 1. These steps involve finding the maximal logits, pruning small logits, and compute the probability with softmax function from the pruned logits. All of them have linear time complexity [V]. This completes our proof.
A.3. OVERTONE provides better updates
We present the formal analysis of how OVERTONE provides better parameters update as outlined in Thm 3.3. Our analysis established in the same spirit of influence function (Koh & Liang, 2017).
first restate Thm 3.3, which outlines the two aspects where OVERTONE is better than training standard CE loss.
Theorem A44 (Informal). Under regularity conditions, compared to optimizing the vanilla CE loss, OVERTONE provides more favorable update direction for the parameters and has less influence on unrelated knowledge.
The formal statement is as follows.
Theorem A.5 (Formal). Let G be the ideal gradient of retraining the LLM using gold the initial value, as defined in Eq (8). Considering the simplified case where 0 in Eq (6), under Assumptions A.6 and A.7, there exists some X E [0,1] such that
cos Volcr(2 "
cos (VofoVERTONE( new. gow, golld), G.).
In other words, using the OVERTONE loss provides a better approximation of the direction of G compared to the standard CE loss, meaning the gradient direction closer to G.
Now, denote the new estimator obtained through either Lce or VOVERTONE by Gnew or VERTONE, respectively. Let 7Mn (Xua yum) be random vector representing unrelated data. Under Assumptions A.II and A.13, we have
Ezz
I
(Z
Tiaus
(
Ezz
TT (Z
TI guse (2mh)
This result indicates that updates based on the OVERTONE loss induce smaller deviations in the predicted distribution for unrelated data compared to updates based on the standard CE loss, thereby better preserving locality.
Theorem A.5 consists of two parts: Theorem A.10 and Theorem A.15. Theorem A.10 states that our method provides a more effective direction for parameter updates, while Theorem A.15 asserts that our method results in smaller perturbation on unrelated knowledge. The assumptions and proofs will presented in Sections A.3.1 and A.3.2, respectively.
14

---

Mitigating Hetcrogencous Token Overfitting in LLM Knowledge Editing
1. OUR METHOD GIVES BETTER DIRECTION PARAMETER UPDATES
Without loss of generality, suppose that LLM pretrained on some large textual corpus {zn J=nin=1. each training sample 2n (z,,, vn where yn .y).... Vm,)) ). KE involves updating some knowledge carried by 2n (x, yddh to new new (x. "0 Let fold denote the pre-trained LLM parameters. Given this piece of new knowledge, the ideal LLM
should have parameters gnew from full retraining by solving
NV
1 N
1 N
ming
(CE(Zn:0) 0)
N (cE(="h +)+ ( 8),
(7)
12=1
where bCF denotes the standard CE loss. general, we define (0( as
E5(0) = Σ
(E((2;)) 0) + (1"m - (cE(2"M):0) 0))
==1
Moreover define
0, = arg min (0).
0
So find that d Gold and 0 -now Starting from Gold when we perform gradient descent by using loss f (0)
retrain the model, the gradient will be
G A VgL &
(Bol).
(8)
So we just take G as the optimal direction to represent that if we retrained the LLM,, i.e., the direction of the gradient descent at gold.
We make following assumption on gold such that is a local the minimizer of (0)
Assumption A.6. The pretrained L.M is converged, namely, Volo( = 0.
For brevity, denote
V TZCCE(2 ; gold), b Volcn(= gold).
V VoCE(Tmle (To()) ctew)]
0=fin
Assumption A.7. cos(b, e) satisfies
| 8||a + b1l2
cos(b, c) >1
(1 cos(a, a + b))2
(10)
Remark A.8 (Interpretation the Assumption A.7). The Assumption A.7 ensure direction and c will not be far away. Roughly speaking, when we take 81|a++011 as some constant. It says that cos(b, c) < (1 cos(a, + 0)) which means the directions of b and e are closer compared with and a + b. When we look it more carefully, Note that represents VolcE(2 (old) and a + b represents the ideal direction G. Since the old knowledge gradient b is present, directly fine-tuning lce (i.e., the baseline method) results in a deviation compared with the ideal direction G. This directional deviation is measured by cos(a, a + b). Let 5(t) denote the collection of unfiltered tokens in Tn(y c.ew)..
||b||2
b = -V VolCE(2 gold) Σ Vo log (o c
(11)
O=jua
CE Tny y
c ( dgown]
Tnl
V log To (y)
chew)
(12)
0=m
O=
Given the new knowledge c,th, when ((1), it implies that is likely close to wf with some probability. Compared to the scenario where the old knowledge cold is given, the gradients Vo log (v( cold) -- a.u and Vo log To coow)
15

---

Mitigating Hetcrogencous Token Overfitting in LLM Knowledge Editing
tend to point in opposite directions. This because both gradients are evaluated at you or a point close to you, but the first is conditioned on cold while the second is conditioned on cg Equivalently, this implies that b and aligned in the same direction. To ensure that we can find closer direction, we require b and с to be approximately as close and + 6. Our goal is to align with the negative gradient direction of the old knowledge. This ensures that when leveraging the information from e to weight our method, we can identify a direction that closely approximates the ideal optimization direction.
Remark A.9. To elaborate further, we take logistic regression as an example for illustration.
When considering only the k-th token, for training point 2μ (ck, yk), let p(ys (x) a (y& (y)))c)). where yk E (-1,1} and a (1) I+oxp(-1) is the sigmoid function. the gradient of the hog-probability with respect to 0 is given by:
V pog. 0) - o--9x0 cx)UKCk.
Then, we find that:
b == o( T ck )uk cold
Σ BLES())
Pyn. (-poo-yll "h Jynck new = -Poao (- old cx 1uk old che - Prew ( new 0J cx Duk new c%
6 =
This follows from the fact that yk E {-1,1}, Note that cx" and cf may be far apart, and Pold is likely to be large since Th a denoised version of vy. meaning contains less noise (Tang et al., 2024). As result, the directions and e will be close.
Theorem A.10. Let G be the ideal gradient of retraining the LLM using Gold as the initial value, as defined in Eq (8). Considering the simplified case where in Eq (6), under Assumptions A.6 and A.7, there exists some X E [0, 1] such that
cos (Volce(zen gold), c) < cos (Volo VOROVEETOOEE( "CW Goul), G)-
In other words, using the OVERTONE loss provides a better approximation of the direction of G compared to the standard CE loss, in the sense that OVERTONE gradient direction is closer to G.
Proof. First, by definition, the optimal gradient direction G when using fold as the initial value is given by
G Vol & (gold)
1 (ewe Volcn(20ld guld))
Volo (Gold)
N
N ( (ol)) Volcn( gous))
where (a) holds from the stationary condition of gold as per Assumption A.6. Note that this optimal direction is inaccessible since it infeasible to find the ground truth old wherefrom the LLM'' old knowledge is learned. In practice, only new is available, which is provided by the user.
To that OVERTONE provide better direction, we check the gradient of CE loss fe and our loss LOVERTOND Recall the definition of a, b, given by Eq (9), for CE loss, we have
271
Volcu(> now 0) = Σ
Vo log (v) new c = a,
(13)
where e.oo x ( yeer as derived in Sec in the main body.
16

---

Mitigating Heterogencous Token Overfitting in LLM Knowledge Editing
For OVERTONE loss, according Eq (5) and Eq (6), have
VelOVERTONE(: =0
V CE tar (v
)[[TT(( c.
CE 6 (V) ( + (1 - A)
Vo (T c, TT c,eem
V log TO )t²)t+++ + (1-
PoCE (v c )|To ct
Aa + Xje.
Next, we check cosine similarity cos (z G) and cos VVoLOVERTONE{ ; gell) G). A larger cosine similarity indicates update direction that aligns with the ideal G' better and is more effective.
Note that
(a, a + b)
COS (VOCCA(="0 new ;0ll), (owd
||a|l (a + b)ll2
cos (v (Velo new, (pold) G)
(Aa (1 A)c, a + +6)
| + -)eelaalla + b)ll
We will show that, there YF E [0. 1].sst. s.t.
(a b)
(Aa +
||a||
| + (1 A)el|2
We further denote dsc Toll Toll
6
which quantifies the directional difference between b and c. We then have:
|bbll2 + dbc ||e||2-
(14)
Take À
|e]2 1 2 by substituting by Eq (14) and applying the triangle inequality, obtain
(Aa + (1 - (1-A)c,un b)
10|12+110|1
||l||
|101|||el12 1b1l2+100
Tbil2
Ebc a
aa (1 2)cll2
||lll2
(a + 6)
|bل|||||||a
8bc
|bll+lll
[bbla+
|1a + b11 ||a + b112( ( 10bc 122111122
||a + bl12 + |6|| 2 18bcll9
|1a + bl|2 ||6|] dse 12 ||a + bll2-
aa bl|2
|b 0bcll2
Therefore, to show OVERTONE provides larger cosine similarity, it suffices to show that
|a + bll2 ||
cos(a, + b),
aa + bll2
[b6] Seell2
which is equivalent show
| (1 cos(a, a + b) ||a bll2
||0bel12
cas(a, a 1 b)
Note that | = 2 cos(b, c), it suffices to show
|b61
1 cos(o. (a, a + b) 1 cos(a, (a, a + b)
2
cos(b, c) >
2|[a
b1l2
Since cos(a, a + b) 1, this condition holds from Assumption A.7. This completes our proof.
17

---

Mitigating Hetcrogencous Token Overfitting in LLM Knowledge Editing
A.3.2. OUR METHOD LEADS TO SMALLER PERTURBATION ON UNRELATED KNOWLEDGE.
Now denote our new estimator obatined through either lcE or to VERTONE by Óeer or 0 VERTONE. After updating the model parameters to incorporate new knowledge, it is crucial to assess whether this update introduces significant changes to unrelated data.
Without loss of generality, let =! = (2et:; ynm) represent a query-answer pair, where x is an unrelated query and y is its corresponding predicted answer. To ensure good locality, the predicted distribution on nnh should remain unchanged against modifications introduced by the update, ensuring that the model's behavior on unaffected regions of the data distribution is preserved. That means want compare
Fouee (=
Tou (="n) with "h To (= T (=mn)
Now, treating 2um (XX"m, y random vector following a certain distribution, define
W
V Zun)
0=
Since W is a function of zm,, it is also a random vector. In particular, we introduce the following assumption. [ill are independent. Furthermore, assume that
Assumption A.11. Assume that W W
and
W
U(**))).
W
where 1( denotes the uniform distribution the unit sphere in IRd with d denoting the dimensionality of the parameter space.
Remark A. 12. Since it represents the gradient of the loss evaluated on unrelated data, we lack any prior information about W. Given that, we assume that W isotropically distributed. W
lella ||all2"
Recall the definition of a, b, given by Eq (9), we define KR =
Assumption A.13. We assume that Ką < 1.
Remark A. (Interpretation of the Assumption A.13), As shown in Eq. (12) and Eq. (13):
m
Ve log (b new cee),,
17
DCW V log To(y
0=
SES
This implies that is weighted combination of a and contributions from other values of S(1), Note that at gold, given c when F yoew the other points are closer to yold Since the loss has already reached its minimum, these other points tend to have smaller gradient norms compared to y!ew.
Theorem A.15. Let 7 (X"", y be a random vector representing unrelated data. Under Assumptions A.II and A.13, we have
Bzn Tege (7 Tigus (7
Tipe (7"!)
Tauu (2."))
This result indicates that updates based on the OVERTONE loss induce smaller deviations in the predicted distribution for unrelated data compared to updates based on the standard CE loss, thereby better preserving locality.
Proof. Again let gold denote the pretrained parameters. For any new parameters grew the change of iT (2 when moves from gold to gnew can be approximated by the first-order Taylor expansion with
T. grew (z
T Tzzu (2 = V VeTTo(z"))
(gnew
gold) + o gnew 1 gold 80o)l1)):
0=6
Note that when we perform one step gradient descent, the parameter change can further be expressed by
grew
Gold -a V Vol(znen (uhd)
18

---

Mitigating Hetcrogencous Token Overfitting in LLM Knowledge Editing
where {(=tow 0) can be either CE loss OVERTONE loss, and denotes the learning rate.
Then to show OVERTONE leads to smaller perturbation in expectation, it suffices to show that there exists A E [0, 1] such that
> E Aa W + (1 X)e W
By triangle inequality, we only need to show
13
aa
W|]
IE wl].
Finally, by Assumption A.1I,
|
W
~
and
[
and
[ la are independent, we have
E
w
E
w
[[[ [2
[W
Wilaa
RR
1.
B
a
TW
W
[Wlla
E
a
TW
w
B B[IlW [2]
This completes our proof.
A.4. Connection between OVERTONE and DPO
We end up this section by the following analysis on the connection between OVERTONE and direct preference optimization (Rafailov et al., 2024).
Theorem A.16. Let c= 0, then optimizing OVERTONE directly can be seen as optimizing an unbiased estimate of a DPO objective plus some additional KL penalty.
Proof. From Prop 3.1 and Lem A.3, at step i, we have the negative loss (objective) to maximize
LOVEERTNEE (0) =-DuL (y c (o c:)l
(A (y) c:)] + A)CE(m((y c:)l c:)))
=A (CE(O,, (T c:)] - CE| ci)llT(yy ct)) - CE|Ta(( c:)] ci)l
(15)
= A (log (8 c2) + C c))l= c,))) cE|m (T) [.]l
(16)
From the lens DPO, note that the editing knowledge (æx, y) can be seen as a preferred sample drawn from unknown n (e.g., retraining the LM from scratch). Consequently, Eq (16) is in fact an unbiased estimator of
A
Byttatt (vle:) Dog Dog c;)] (o
Ey -al log To c:]]
CE(Tm((1) (i) c,)
Preferred distriibution
To(y+ log TT
c.)
XEyt.y .
DEL (T c:)l( c.)] +0]++
G.)
@ Byyt -M
log TV
Ty
c.)
log
Tv ci)
+ Eny [log mn cz)) R.y Dlog Tnly cc:)]
ci)
Tn c;)
(T)
[;)]
a
log To(y* cc)
Ty( (2)
Ey
log
+ Eyy [v [:)] E,y
[TT c:)
Toly
c.)
log Tei (y c;)
constant wit 0
DDKL(TTIV (T) [c:]+
llg (i)
To(x
Alog To TV
ci)
Eyyt.r
Daul [T(( [ c))l c:) +C,
(17)
The
Tat
c.)
Additional Penalty
clipped exponential preference
where the first term incorporates preferred distribution, of which the user srer-irovided new knowledge y/ serves an unbiased estimate. Step (a) plugs in the log-likelihood ratio between the (y+, " pair from Mas which constant with respect to
19

---

Mitigating Heterogencous Token Overfitting in LLM Knowledge Editing
and doesn't I affect the objective thereof. In the final step, treat the first term as a token-level DPO objective using current Th the reference model, and the preference model is given by clipped exponential preference model
Pr(y* - L c.) = (enn(eepr((;;; y²) - 8
where Z 2
1 is some constant. Notably, since our base distribution, Tits is the clipped version of ne, and À [0, 1]. the difference in probability of (t² given is expected small, so that we can impose
To (y
C: c2)
0 S Alog
Alo
1,
m (y+
cs)
this allows us to set Z = e and get rid the clipping operator. Then, the first term becomes
Talyt ci) A log
Toly" c:)
y tyttx
Alog
Tnll
(;))
Tn -
c;)
ao Ci)
To
C2)
Byt.y .
log exp Alog
Allog
12) C2)
+ log Z
(i)
Fnd
y
=Byt.y
loog Pr(y"
y
ct] log Z,
where log Z is constant in parameter 0. Comparing this equation with Rafailov et al. (2024) draws a connection between OVERTONE and DPO. The second term of Eq (17), on the other hand, is an additional penalty to push my stay close mm by using a forward KL, which has been explored in preference learning (Wang al., 2024a).
In conclusion, OVERTONE can be
unbiased estimator of special DPO problem. This completes our proof.
B. Implementation Details
B.1. Hyperparameterss used in KE
We present the implementation details our algorithms. All of our experiments are run on EasyEdit (Wang et al., 2024e). In general, we tuned hyperparameters for each KE method basis using to the base version, if the default setting from EasyEdit showed noticable inferior performance. See below for more details.
FT-M used the following hyperparameters:
On ZsRE, Wikineeent, Wikicounterfacs and WikiBio: default training parameters from EasyEdit for both LLaMA 2 and LLaMA 3.
On MQUAKE: Layers tune: (20,21,22,23,24)) Learning rate: le-3, Others unchanged.
LoRA used the following hyperparameters:
On ZsRE, Wikigecent- Wikigonttrfaats and WikiBio: default training parameters from EasyEdit for both LLaMA 2 and LLaMA 3.
On MQUAKE: LoRA rank: 12. Iteration numbers: 50. Others unchanged.
MELO used the following hyperparameters:
We set initial radius for each code in the code-book to 60 for LLaMA 2, and 30 for LLaMA 3. Due to the fact that the default choice 0.1 was too small to retrieve any edited parameters for rephrased queries or reasoning.
Others unchanged.
WISE used the following hyperparameters:
On OVERTONE, we shrunk activation thresholds by 0.6 in consideration the milder overfitting from our method. We didn't 't tune this shrinkage factor so can be suboptimal. All other parameters used default values from EdsyEdit.
20

---

Mitigating Hetcrogencous Token Overfitting in LLM Knowledge Editing
We removed data augmentation for better measure HTO influence. This led to significantly faster editing speed (around 5 times speedup).
ROME and MEMIT used default choices from EasyEdit.
Finally, OVERTONE is tuned KE model base and applied to both LLMs. We didn't 't tune hyper- syper-parameters extensively, so below and can be suboptimal.
FT-M: 0.01, n 0.5 for no-filtering, 0.1 for mixing.
LoRA: 0.05, 0.5 for nolfiltering, À = 0.1 for mixing.
MELO: = 0.05, for no nilfiereing À = 0.1 for mixing.
WISE: < = 0.05, 71 for no-filtering, À = 0.1 for mixing.
B.2. MQuAKE Experiment Details
MQuAKE benchmark follows different evaluation pipeline for Single-Hop and Multi-Hop reasoning questions (Zhong et al., 2023; Wang et al., 2024f) that checks the existence of ground truth answer in LLM's generation. Our evaluation rubric followed Zhong et al. (2023). We noted that the reliability of evaluation results heavily relies on the use of a good prompt, our prompts are given below.
Single- Hop questions: used 1-shot prompting guide the model provide answers directly, the complete prompt is
You are helpful Al assistant. Answer questions directly.
Always format your response
Final answer: [concise and direct final answer]
Question: Who is the spouse of the head of state in United States of America?
Answer: Jill Biden
Question: Single-Hoe Hop question related to the new knowledge #
Answer:
Multi-Hop questions: Again used 1-shot prompting to guide the model provide answers based chain-of-
thought (Wei et al., 2022), the complete prompt
You are helpful AI assistant. For each question:
1. Break it down into simpler subquestions
2. Answer each subquestion step by step.
3. Use your answers to provide final answer after "Final answer:
Always format your response as:
Subquestion: [your subquestion] Generated answer: [your answer]
Final answer: [concise and direct final answer]
Question: Who is the spouse of the head of state in United States of America?
Subquestion: Who is the head of state in United States of America?
Answer: The head of state in United States of America is Joe Biden.
Subquestion: Who is the spouse of Joe Biden?
Answer: The spouse of Joe Biden is Jill Biden.
Final answer: Jill Biden
Question: Multi Hop question related to the new knowledge #
In generation, set temperature 0.1. The maximum length was 30 for Single-Hop questions, and 200 for Multi-Hop questions. Chat templates are applied.
C. More Experiment Results
We present the complete Continual Editing results here. Note that sequence T reduces to Single Edit results, but we present them again for completeness.
21

---

Mitigating Hetcrogencous Token Overfitting in LLM Knowledge Editing
Table Continual Fditing performance (I.aaMA 2).
WISE requires additional irrelevant data for training, which is only available in
ZsRE benchmark.
ZsRE
Wikirecent
Wikdgoenteract
WikiBio
T
Rel.
Gen.
Por.
Loc.
Avg.
Rel.
Por.
Loc.
Avg.
Rel.
Por.
Loc.
Avg.
Rel.
Loc.
Avg.
ROME
96.61
83.91
55.7
96.96
83.3
99.02
54.21
55.91
69.71
97.2
56.85
50.4
68.15
96.41
59.14
77.78
MEMIT
94.22
88.2
57.91
98.28
84.65
97.71
52.93
55.05
68.56
96.38
59.34
45.7
67.14
93.78
56.74
75.26
FT-M
99.75
99.33
54.32
93.01
86.60
100.0
62.93
45.92
69.62
100.0
74.7
54.86
76.52
100.0
90.04
95.02
+ Ours
99.75
96.8
57.08
96.54
87.54
100.0
63.91
60.4
74.77
100.0
73.62
75.34
82.99
100.0
93.46
96.73
LoRA
100.0
100.0
23.34
30.44
63.45
100.0
55.41
28.29
61.23
100.0
71.92
9.99
660.4
100.0
48.84
74.42
+ Ours
100.0
94.31
61.16
87.2
85.67
100.0
63.67
58.72
74.13
100.0
73.96
57.85
77.27
97.68
68.45
83.06
MELO
100.0
96.77
27.11
92.35
79.06
99.13
54.04
40.96
64.71
99.0
71.78
55.83
75.54
99.97
80.77
90.37
Ours
100.0
93.31
50.36
97.2
85.22
100.0
60.25
66.48
75.58
99.91
71.81
78.09
83.27
99.68
82.58
91.13
WISE
92.42
70.86
54.57
100.0
79.46
Ours
97.55
76.09
54.17
100.0
81.95
T = 10
Rel.
Gen.
Por.
Loc.
Avg.
Rel.
Por.
Loc.
Avg.
Rel.
Por.
Loc.
Avg.
Rel.
Loc.
Avg.
ROME
74.94
69.67
51.12
71.72
66.86
98.14
55.16
54.73
69.34
86. 17
47.36
38.99
57.51
40.55
25.98
33.27
MEMIT
68.39
66.26
46.66
84.22
66.38
96.51
54.2
52.56
67.76
89.64
54.71
38.2
60.85
52.2
38.54
45.37
FT-M
89.14
87.43
47.13
84.26
76.99
97.4
56.47
41.4
65.09
96.41
70.32
42.44
69.72
92.96
77.69
85.32
Ours
92.8
88.21
55.74
91.06
81.95
96.42
61.65
53.13
70.40
98.72
72.47
65.46
78.88
95.26
84,43
89.84
LoRA
29.25
30.41
19.83
24.81
26.07
35.17
23.8
24.98
27.98
22.64
13.87
T0.24
115.88"
770.45
46.82
58.64
+ +Ours
85.4
81.5
61.03
74.41
75.59
94.55
59.16
49.09
67.60
71.61
51.91
32.65
52.06
74.74
48.35
61.55
MELO
94.13
83.06
50.48
96.5
81.04
91.73
53.02
81.09
75.28
92.52
64.55
99.98
85.68
95.44
97.94
96.69
+ Ours
94.38
81.89
54.92
98.41
82.40
91.69
54.95
93.22
79.95
93.49
63.36
99.98
85.61
95.24
97.77
96.50
WISE
84.5
73.8
53.19
100.0
77.88
+ +urss
86.68
77.24
54.0
100.0
79.48
T = 100
Rel.
Gen.
Por.
Loc.
Avg.
Rel.
Por:
Loc.
Avg.
Rel.
Por.
Loc.
Avg.
Rel.
Loc.
Avg.
ROME
25.37
22.68
4.73
5.1
14.47
24.99
13.12
8.55
15.56
0.0
0.0
0.0
0.0
2.63
15.74
9.18
MEMIT
2.58
2.88
0.24
2.5
2.05
70.22
41.12
38.43
49.92
0.82
0.97
0.26
0.69
0.0
15.74
7.87
FT-M
88.36
84.51
41.76
54.11
67.19
97.51
53.73
33.88
61.71
95.69
66.23
26.69
62.87
93.56
67.51
80.53
Ours
89.38
82.13
52.69
72.39
74.15
96.32
58.28
47.04
67.21
95.93
68.16
44.28
69.46
95.35
74.91
85.13
LoRA
0.67
0.78
T.00
00:03
0.62
0.5
0.3
6.12
0..7
0.67
0.0
0.0
0.22
47.02
27.06
37.04
Ours
62.23
58.06
56.62
59.57
59.12
70.49
47.05
49.87
55.80
32.
28.99
29.19
30.12
$2.96
25.73
39.34
MELO
38.13
36.12
53.88
98.08
56.55
26.33
24.98
53.73
35.01
24.87
24.21
78.71
42.60
48.88
97.61
48.88
+ Ours
39.13
37.28
54.75
98.58
57.44
47.95
39.65
86.77
58.12
24.92
25.39
97.12
49.14
52.17
97.44
74.81
WISE
84.59
71.59
54.455
160.0"
77.66
-uurs
92.42
84.22
56.71
100.0
83.34
D. More Discussions
We discuss some more conceptual characteristics and potential problems that future works may work on here.
OVERTONE and ROME/MEMIT. ROME (Meng et al., 2022a) and MEMIT (Meng et al., 2022b) are representative solutions of KE through the boddte-and-editing paradigm that are built upon the causal tracing and explicitly constructed updated rules. This leads to special KE losses which contains two unique designs other than those being used in the four backbone methods the sense that the MSE loss will determine the final parameter update. Second, ROME relies on random prefix augmentation, which affects overfitting as well. Given these facts, we plan to work on a more principled way to extend OVERTONE, a augmentation-fre end end-to-end training paradigm, in light of its principle. That is, we seek a better way to smooth (relax) different token fitting adaptively with the model's 's own knowledge, following the principle of OVERTONE. Therefore, it would be interesting to bring the idea of OVERTONE to training ROME and MEMIT boost their generalizability.
studied. First, the impact of auto-regressive loss, which OVERTONE alters, on ROME is weaker, in
22

---

Mitigating Hetcrogencous Token Overfitting in LLM Knowledge Editing
Table Continual Fditing performance (ILaMMA 3).
WISE requires additional irrelevant data for training, which is only available in
ZsRE benchmark.
ZsRE
Wikirecent
Wikigoantrfact
WikiBio
T
Rel.
Gen.
Por.
Loc.
Avg.
Rel.
Por:
Loc.
Avg.
Rel.
Por.
Loc.
Avg.
Rel.
Loc.
Avg.
ROME
99.17
97.91
58.12
95.9
87.78
98.84
54.76
49.74
67.78
99.94
58.0
42.94
66.96
92.43
72.63
82.53
MEMIT
96.67
92.46
58.78
98.23
86.53
98.51
53.65
48.45
66.87
99.44
57.81
42.73
66.66
96.26
71.23
83.75
FT-M
100.0
99.75
40.43
79.43
79.90
100.0
57.13
30.01
62.38
100.0
72.62
31.47
68.03
100.0
92.96
96.48
+ Ours
100.0
99.75
48.63
94.78
85.79
100.0
60.88
44.67
68.52
100.0
73.5
58.29
77.26
99.99
94.87
97.43
LoRA
100.0
100.0
26.55
38.85
66.35
100.0
52.99
26.46
39.82
100.0
71.1
9.02
'60.0
100.0
39.17
79.88
Ours
100.0
51.57
93.13
85.80
100.0
61.46
56.1
72.52
100.0
72.8
57.54
76.78
16
77.24
87.7
MELO
100.0
96.84
39.63
98.8
83.82
100.0
59.07
65.78
74.95
100.0
71.55
87.77
86.44
100.0
98.56
99.28
Ours
100.0
95.77
43.08
98.8
84.41
100.0
58.72
69.1
75.94
100.0
70.26
89.81
86.69
99.98
98.56
99.27
WISE
71.67
51.29
49.27
100.0
68.06
Ours
82.67
62.34
47.54
100.0
73.14
T = 10
Por.
Loc.
Avg.
Rel.
Por.
Loc.
Avg.
Rel.
Por.
Loc.
Avg.
Rel.
Loc.
Avg.
ROME
43.91
40.14
25.11
31.7
35.22
91.17
51.25
43.67
62.03
86.52
45.37
32.9
54.93
4.01
7.58
5.79
MEMIT
59.74
58.36
37.34
71.06
56.62
98.38
54.42
47.08
66.63
98.61
58.48
36.28
64.46
5.4
1.61
3.5
FT-M
79.54
78.44
25.03
43.97
56.75
87.22
48.12
25.8
53.71
90.13
62.37
13.83
55.44
95.59
87.45
91.52
Ours
84.74
81,41
44.2
75.67
71.50
92.77
52.65
38.99
61.47
93.04
66.5
39.99
66.51
96.81
91.17
93.99
LoRA
18.54
17.55
6.63
6.56
12.32
21.7
13.66
11.97
15.78
12.59
3.92
0.69
6.40
"$..99
44.45
47.77
+ +Orrs
73.28
72.39
53.13
69.36
67.04
93.68
56.97
49.34
66.66
71.99
49.52
32.24
51.25
64.26
55.11
59.69
MELO
94.08
80.47
47.97
98.8
80.33
92.56
54.51
86.58
77.88
92.97
63.74
98.3
85.00
94.77
98.56
96.67
+ Ours
94.08
80.94
49.77
98.8
80.90
91.56
54.24
89.16
78.32
92.97
62.69
98.32
84.66
94.91
98.56
96.74
WISE
51.14
43.36
51.0
100.0
61.38
+ +urs
58.21
53.22
49.21
100.0
65.16
T 100
Rel.
Gen.
Por.
Loc.
Avg.
Rel.
Por.
Loc.
Avg.
Rel.
Por.
Loc.
Avg.
Rel.
Loc.
Avg.
ROME
7.18
6.02
1.04
2.24
4.12
8.89
1.36
0.31
3.52
3.92
0.99
0.0
1.64
0.88
7.47
4.18
MEMIT
0.0
0.57
0.92
0.63
0.86
0.0
0.56
0.01
23.44
11.73
FT-M
78.79
78.29
13.7
15.42
46.55
94.27
44.09
22.99
53.78
87.47
55.62
2.78
48.62
93.65
85.83
89.74
Ours
81.2
77.87
32.65
44.66
59.09
96.19
53.73
32.42
60.78
92.97
62.02
20.71
58.57
94.23
85.83
94.23
LoRA
1.75
T.81
T.29
2.13
1.74
1.35
T.58
0.93
T.28
11.00
0.00
0.00
0.33
15.88
17.61
T6.74
Ours
51.38
50.3
49.72
35.83
46.81
64.82
42.92
44.27
50.67
25.31
20.18
17.49
20.99
19.03
10.9
14.96
MELO
29.79
28.83
50.01
98.8
51.86
36.71
29.02
83.23
49.65
22.2
22.9
97.85
22.35
52.19
98.56
75.37
+ Ours
29.79
28.73
50.01
98.8
51.83
40.42
34.85
92.67
55.98
22.45
22.9
97.85
47.73
52.15
98.56
75.36
WISE
84.87
74.87
39.24
100.0
74.75
+ Ours 86.83
77.54
34.99
100.0
74.84
Potential Bias in OVERTONE Design. OVERTONE designed to the model's own prediction to extract pretrained knowledge that should be maintained. To avoid misleading knowledge conflict and general noise, OVERTONE incorporates two mechanisms. First, the unreliable (noisy) part is filtered out. Second, mixing with the model's prediction is conducted only if the mixed distribution correctly assigns the ground truth label (i.e., training token) the highest probability. However, provably solving the potential knowledge conflict and identifying the optimal target distribution for KE are still two open questions, and we advocate for future studies to work on these two directions towards better KE.
23


Reviews:
[{"review_id": "3fz3gfVmZL", "reviewer": "Reviewer_zvke", "summary": "This paper proposes OVERTONE, a token-level smoothing method to address heterogeneous token overfitting (HTO) in knowledge editing (KE) for large language models (LLMs), enabling specific knowledge updates without compromising pre-trained capabilities. Experiments across multiple methods, LLMs, and scenarios show OVERTONE improves performance and versatility over previous KE approaches, with minimal computational overhead and an implicit DPO mechanism.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one.\n>**Why does negative UD (NUD) represent overfitting?**\n\nNUD represents a *token* is overfitted as its training loss is *too small*. Specifically, NUD computes the difference between the token's training loss (of edited model), and the *greedy decoded* token's *pretrained* loss (of unedited model). The choice of greedy decoding is on purpose, as it reflects the unedited model's *most confident knowledge proper and valid in the past*. By comparing the two, NUD indicates that the edited model is overly confident, and is \"*overfitted*\" thereof. We will make this clearer. \n>**Is OVERTONE adaptable to other fine-tuning methods and tasks? On long-text scenarios?**\n\nOVERTONE is developed in light of HTO (i.e., editing knowledge is overfitted at different speeds). As per Sec 2.3, one cause of HTO is when knowledge editing (KE) involves few training data (such as single one) and trains the model with the fixed data for many steps, inevitably overfits. Similar concerns may raise in other tasks that seek *selective* updates of LLMs such as machine unlearning, wherefore OVERTONE could be applied. Moreover, when the training text is long, as the number of tokens to learn grows, we expect HTO to exacerbate, and OVERTONE to be helpful.\n>**Why does portability perform significantly worse on WISE (Table 5)?**\n\nTable 5 reports editing performance on LLaMA 3 with hyperparameter adopted from LLaMA 2, which can be suboptimal. As an evidence, the vanilla WISE also performs worse than on LLaMA 2 (WISE has an activation mechanism to determine if edited parameters should be used, which needs additional tuning as well). Nonetheless, OVERTONE was able to help achieve a better editing-generality-portability trade-off, leading to a higher *average* performance. \n>**Why a notable decline in generalization (gen) for FT and LoRA?**\n\nWe believe the generality (Gen) decrease is also caused by suboptimal hyperparameter: As shown Fig 1, Gen doesn't encounter degradation due to HTO: as Gen loss change is nearly identical to that of training loss. Therefore, when OVERTONE mitigates HTO, it slightly decreases Gen while achieving better portability and locality. Still, the *average* performance is improved, and we believe our method can benefit from a more extensive hyperparameter tuning.\n>**Why were MEND, MEMIT, and MELLO not evaluated in Table 2?**\n\nTable 2 showed that OVERTONE can help improve reasoning in more challenging scenarios. Therefore, we focused on FT and LoRA, two simple methods *suffered more degradation* from HTO, to demonstrate the effectiveness of OVERTONE. We didn't include MEND, MEMIT, and MELLO because of their different mechanisms: MEND relies on a *large external dataset* to train its hypernet, MEMIT uses a objective other than maximizing the editing data likelihood, and MELLO is training-free. \n>**Filtering tail regions makes locality decrease.**\n\nThis trend can be related to the definition of locality. Conceptually, a perfect locality only requires the edited model prediction match its pretrained output, regardless of its correctness and usefulness. Therefore, without filtering the tail region of the model's *own prediction*, the model's pretrained \"knowledge\" dominate the target $\\pi_{tar}$ to learn, leading to a higher locality. However, the tail region of the model's *own prediction* is usually noisy, and this noise can be harmful. As shown in Tab 3, both generality and portability decreased. \n>**Typos and grammatical error.**\n\nThank you for catching the error! We will fix this mistake in the revised draft.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem at hand.", "theoretical_claims": "I did not check the theorectical claim thoroughly.", "experimental_designs_or_analyses": "The soundness and validity of the experimental designs and analyses seem to be appropriate.", "relation_to_broader_scientific_literature": "The key contributions of the paper build on prior work in knowledge editing (KE) and large language model (LLM) fine-tuning. The identification of heterogeneous token overfitting (HTO) as a critical issue in KE extends the understanding of overfitting in LLMs, which has been explored in works such as Zhang et al. (2024), who investigated overfitting in fine-tuning LLMs.", "essential_references_not_discussed": "I think there are no essential related works missing from the paper that are critical to understanding the context of its key contributions.", "other_strengths_and_weaknesses": "**Strength:**\n\n* The paper is well written with clear motivations.\n\n* The paper conducts comprehensive experiments.\n\n**Weakness:**\n\n* Experimental results, particularly in Table 5 and Table 1, raise doubts due to lower portability compared to WISE and generalization declines in FT/LoRA. Please see questions.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "1. What is meant by \"We define underfitting degree (UD) as the difference between the pre-edited and running log-likelihood; negative UD indicates an overfitting\"? Why does a negative UD represent overfitting?\n\n2. Is the proposed solution (OVERTONE) adaptable to other fine-tuning methods, tasks, and datasets beyond knowledge editing tasks? Additionally, does it still perform well on long-text scenarios?\n\n3. Why does portability, an emphasized metric for overfitting in this paper, perform significantly worse than the WISE method in many scenarios (e.g., Table 5)? Additionally, why is there a notable decline in generalization for FT and LoRA?\n\n4. Why were methods like MEND, MEMIT, and MELLO not evaluated in Table 2?\n\n5. In Table 3, why does adding filtering tail regions lead to a decrease in locality? Does this imply that some useful general information is being filtered out?\n\n6. Minor: There is a grammatical error in lines 314-315: “*We next check where the improvement was made. from the table, the first gain was from improved portability.*”", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N.A."}}, {"review_id": "pV7yNsQWFk", "reviewer": "Reviewer_Dqfx", "summary": "This paper investigates the challenge of heterogeneous token overfitting in knowledge editing of large language models, where different tokens in the target knowledge generalize at varying rates during selective parameter updates. To address this, the authors propose OVERTONE—a token-level smoothing approach that adaptively refines the training target for each token to mitigate overfitting while preserving unrelated pre-trained knowledge. The paper presents both a detailed theoretical analysis, which connects OVERTONE to concepts such as DPO, and extensive empirical evaluations on multiple benchmarks using models like LLaMA 2 and LLaMA 3. The results demonstrate that OVERTONE significantly enhances editing performance by improving the model’s reasoning (portability), generality, and locality with negligible computational overhead, offering a flexible plug-and-play solution that can complement existing KE methods.\n\n## update after rebuttal\nThe additional experiments have indeed resolved most of my doubts and provided more comprehensive support for the arguments in this paper. I am inclined to accept this paper, but since I initially gave it a score of 3, which means leaning towards accept, I will maintain the score of 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one.\n>**More model architecture.**\n\nWe follow recent works (e.g. EasyEdit survey) to study the representative LLaMA family. Following the reviewer's suggestion, \nwe further study Qwen2.5-3B-Instruct. Due to time constraint, we only experiment it with ZsRE, using hyperparameters on LLaMA-2 (which can be suboptimal).\nAs shown at: https://anonymous.4open.science/r/hto-overtone, OVERTONE again helps achieve better editing performance. For the reviewer's convenience, we paste FT-M Single Edit results here.\n|        | Rel. | Gen.  | Por.  | Loc.  | Avg   |\n|--------|------|-------|-------|-------|-------|\n| FT-M   |  100.0 |  99.3 | 50.98 | 73.13 | 80.85 |\n| +Ours  |  100.0 | 96.18 | 56.26 | 80.66 | 83.28 |\n\n>**Comparison with DPO.**\n\nFollowing the reviewer's suggestion, we train LoRA with DPO.We use the pre-edited model's old knowledge as the negative data. Due to time constraint, we only conduct Single Editing on ZsRE. We note that DPO performs worse, which we presume due to the practical challenge highlighted in Sec 3.2. \n\n|        |       | Rel. | Gen.  | Por.  | Loc.  | Avg   |\n|--------|-------|------|-------|-------|-------|-------|\n| Llama2 | Ours  |  100 | 94.31 | 61.16 |  87.2 | 85.67 |\n|        | DPO   |  100 | 94.74 | 33.64 | 41.66 | 67.51 |\n| Llama3 | Ours  |  100 |  98.5 | 51.57 | 93.13 |  85.8 |\n|        | DPO   |  100 | 97.77 | 19.61 | 10.58 | 56.99 |\n\n>**Statistical significance tests along with mean and variance from multiple runs.**\n\nOur experiment design follows the convention in Knowledge Editing and is based on the widely-used EasyEdit. \nAll metrics are average from different samples, each uses a *different* initial value. The random seed is fixed to 42, so \"standard\" and \"ours\" use the identical initial value for the same sample.\nFollowing the reviewer's thought, \nwe tried seed 2025, and ran \"ours\" as in Tab 3, resulting in a new average 85.44 (Rel 100, Gen 94.88, Por 60.41, Loc 86.47), \nwhich is very close to the reported one, confirming the effectiveness of our method. \nFrom a statistical test perspective, out of 72 comparisons (\"standard\" vs \"ours\"), \nours achieved better performance in 69 cases, providing a \"significant\" improvement based on binomial test.\n\nFinally, we agree with the reviewer that conducting multiple runs on each sample (knowledge) will further enhance the reliability, which is valuable but has been largely overlooked by the community. \nWe will highlight this in the limitation section of the revised paper, and will follow this principle in our future work. \n\n>**Hyperparameter selection and sensitivity.**\n\nWe didn't conduct an extensive hyperparameter tuning. Our current selection can be found in App B, and are made as follows: $\\epsilon$ is set to close to 0, and we tried 0.05 and 0.01; $n$ for filtering tried $1$ (default in \"Top $n\\sigma$\" paper) and more aggressive $0.05$ for simpler LoRA and FT. Finally, mixing $\\lambda$ was set to $0.1$ to encourage fast integration of $\\pi_{flt}$ without tuning. We didn't notice too big difference when trying different $n$, which we found reasonable, as the *correctness-checking mechanism (Eq 4), will discard a smoothing if it is too misleading*. However, OVERTONE can be sensitive to $\\lambda$, as greater $\\lambda$ makes our method more similar to standard training.", "claims_and_evidence": "The paper's core claims about the effectiveness of the OVERTONE method are generally well-supported by extensive experimental evidence across multiple editing methods, models, and datasets. However, some claims lack sufficient supporting evidence:\n\n- Model-agnostic wide applicability: While the effectiveness has been demonstrated across 4 editing methods, testing was limited to only two LLaMA series models. Experiments on additional architectures such as Qwen or Mistral would strengthen the validation of the method's broad applicability.\n\n- Connection to DPO: The derivation linking OVERTONE to DPO is mathematically detailed; however, the practical implications of this connection are not supported by direct experimental comparisons. More empirical evidence demonstrating that the benefits of DPO carry over to the OVERTONE framework would help solidify this claim.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for the problem at hand; I believe that aside from not considering a broader range of model architectures, both the approach and evaluations are reasonable.", "theoretical_claims": "I reviewed the proofs for all the proposition. In general, the derivations appear logically align with standard techniques such as Taylor expansion, KL-divergence properties, and influence function analysis. However, several proofs rely on nontrivial assumptions—for example, assumptions about gradient isotropy, the convergence of pretrained models, and specific cosine similarity bounds—which may be strong in practice. These assumptions, along with simplifications in the first-order approximations and the filtering mechanism provide valuable theoretical intuition, further empirical validation.", "experimental_designs_or_analyses": "The experimental designs and analyses were carefully structured and exhibit strong soundness and validity. However, there are some points which can be improved:\n\n- Although the experimental results demonstrate significant improvements across various metrics, the study would benefit from deeper statistical validation. Specifically, including statistical significance tests along with reporting the mean and variance from multiple experimental runs (e.g., using different random seeds) would enhance the robustness and persuasiveness of the conclusions.\n\n- Additionally, the experiments predominantly rely on LLaMA 2 and LLaMA 3, which share substantial architectural similarities. This architectural limitation may affect the generalizability of the method to other model designs, such as Qwen and Mistral, or multimodal models. Expanding the evaluation to include a more diverse set of architectures would help in thoroughly validating the broader applicability of the proposed method.", "relation_to_broader_scientific_literature": "This paper tackles the issue of heterogeneous token overfitting in knowledge editing for large language models by building on prior work in areas like selective fine-tuning and parameter-efficient methods (e.g., LoRA, ROME, and MEMIT). It extends traditional strategies such as label smoothing and early stopping by introducing a token-level adaptive smoothing approach that preserves the model’s pre-trained knowledge while integrating new information. The authors support their method with a theoretical analysis based on influence functions and draw connections to constrained optimization techniques like direct preference optimization (DPO). Overall, the approach not only improves the reliability, generalizability, and locality of edited models but also offers a versatile, model-agnostic framework that advances both the practical and theoretical understanding of LLM knowledge editing.", "essential_references_not_discussed": "I think the paper adequately covers the essential references needed to understand the context for its key contributions. But I am not very familiar with this domain.", "other_strengths_and_weaknesses": "The paper introduces a novel token-level adaptive smoothing approach that effectively mitigates heterogeneous token overfitting in knowledge editing for large language models. The method is supported by thorough theoretical derivations and extensive empirical evaluations on LLaMA 2 and LLaMA 3, showing significant improvements in reasoning capacity, generality, and locality.\n\nHowever, the experimental validation is limited to a pair of closely related architectures, suggesting that further statistical validation and broader model evaluations are needed to confirm the method’s wide applicability and real-world effectiveness.", "other_comments_or_suggestions": "- Consider including statistical significance tests along with mean and variance from multiple runs (e.g., using different random seeds) to further validate the improvements and robustness of OVERTONE.\nBroader Model Evaluation:\n- Expanding the experimental evaluation to include architectures beyond LLaMA 2 and LLaMA 3 (such as Qwen or Mistral) would strengthen the claim regarding the method's model-agnostic benefits.", "questions_for_authors": "- Your experiments primarily focus on LLaMA 2 and LLaMA 3 models. Have you evaluated or can you comment on how your approach scales to other architectures (e.g., Qwen or Mistral)?\n- Could you provide details on how the hyperparameters were selected for the experiments and discuss any observed sensitivity to these choices?\n- The paper draws a connection between your approach and DPO. Could you provide more insights or quantitative analyses on how this theoretical connection affects the overall performance of the OVERTONE method in practice?\n- While the experimental results show promising improvements, could you elaborate on the variability of these results? Specifically, have you conducted multiple runs with different random seeds and performed statistical significance tests to verify the robustness of your improvements?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VXTfpStXnP", "reviewer": "Reviewer_h3rV", "summary": "This paper investigates the Heterogeneous Token Overfitting problem in knowledge editing. The authors first analyze the root cause of this issue, attributing it to the training paradigm that indiscriminately optimizes the probabilities of all tokens. To address this, they propose OVERTONE, which refines the traditional loss function. The theoretical advantages of OVERTONE are demonstrated, and experiments show that it outperforms several baselines across diverse experimental settings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one.\n\n>**OVERTONE can be applied to ROME (or MEMIT) to improve the loss function (Eq 4 in the original paper).**\n\nThank you for the insightful idea of extending OVERTONE to ROME. We noted two unique designs in ROME (and MEMIT) makes it differ from four methods we studied. First, the impact of auto-regressive loss, which OVERTONE alters, on ROME is weaker, in the sense that the MSE loss will determine the final parameter update. Second, ROME relies on random prefix augmentation, which affects overfitting as well. Given these facts, we plan to work on a more principled way to extend OVERTONE, a augmentation-free end-to-end training paradigm, in light of its principle. That is, we seek a better way to *smooth (relax) different token fitting adaptively* with the model's own knowledge, following the principle of OVERTONE. We will highlight this challenge, together with our future plan in the revision. \n\n>**Comparison with LTI, another method designed to alleviate overfitting.**\n\nBoth LTI and OVERTONE works on mitigating overfitting in knowledge editing. The conceptual similarity, from a high level, lies in adding pre-trained knowledge to the editing. \nBut LTI explores a distinct direction, with its difference to ours lying in X folds. \nFirst, LTI explores in-context learning (ICL) to incorporate pre-trained knowledge into the editing data, \nwhile ours designs an adaptive token-level distribution mixing, in light of the token-level HTO dynamic.\nSecond, LTI, which is primarily developed for ROME-based solution, acts on both latent representation and output prediction loss. Ours, on the other hand, is agnostic to the editing method and alters the output prediction loss only.\nFinally, LTI, same as ROME, relies on a data augmentation, while ours does not include such mechanism. \nFollowing the reviewer's suggestion, \nwe will highlight these differences in the revised paper, and will explore bridging the two directions in our future work.\n\n>**Bias in knowledge conflicts and the model's own predicted distribution could be unreliable.**\n\nWe agree that potential knowledge conflict and general noise can be misleading. \nTo reduce this risk, OVERTONE incorporates two mechanisms. \nFirst, the unreliable (noisy) part is filtered out. \nSecond, mixing with the model's prediction is conducted only if the mixed distribution correctly assigns the ground truth label (i.e., training token) the highest probability (Eq 3).\nFinally, *provably solving the potential knowledge conflict* for knowledge editing is still an open question, and we will highlight this in the revision. \n\n>**Impact of filtering threshold $n$.**\n\nMathematically, \"without filtering\" is equivalent to setting $n \\rightarrow \\infty$. \nAs shown in Tab 3, this leads to a worse performance. \nTo further study how sensitive $n$ is, \nwe follow the reviewer's suggestion and try a larger $n=1$ on LoRA, which is the default value from Top $n\\sigma$ paper. \nThis gives us average performance 85.49 (Rel: 100, Gen: 94.85, Por: 61.44, Loc: 87.01) on editing ZsRE, which is slightly lower than 85.67 from $n=0.5$. \nWe believe this insensitivity is reasonable, considering that correctness-checking mechanism will discard the mixing if it is misleading. \nWe will add this discussion in the revised paper. \n\n>**Knowledge editing in free text form.**\n\nWe thank the reviewer for coming up this interesting direction. \nAfter checking related papers, we agree that the free-form text can express more diverse knowledge, on which the editing can be important but also more challenging. Considering the common practice that knowledge to edit at a time is few, we expect a similar overfitting due to the small training size nature, and HTO because some pretrained knowledge can still be useful, making different parts (tokens) have varied difficulties to learn. Therefore, we believe that our method can shed light on this interesting problem. \nHowever, we also believe that the free-form will add additional challenge to understand and quantify the overfitting. \nTherefore, we will explore this direction in our future work, and add this discussion, together with related papers, in the revised paper.", "claims_and_evidence": "The claims are supported by evidence.", "methods_and_evaluation_criteria": "While OVERTONE effectively adjusts the target distribution by filtering out noise tokens, it may inadvertently introduce bias in scenarios such as knowledge conflicts, where the model's own predicted distribution could be unreliable.", "theoretical_claims": "I have checked the correctness of any proofs for theoretical claims.", "experimental_designs_or_analyses": "1. I think OVERTONE can be applied to ROME (or MEMIT) to improve the loss function (Equation (4) in the original paper https://arxiv.org/pdf/2202.05262), but the experiment did not show the corresponding results.\n\n2. This paper lacks a comparison with LTI ([1]), another method designed to alleviate overfitting.\n\n3. The impact of varying the parameter filtering threshold n on the performance of the proposed method is not explored in the experiments.\n\nReferences:\n[1] https://openreview.net/forum?id=t8qcGXaepr", "relation_to_broader_scientific_literature": "Prior work has identified the problem of overfitting in knowledge editing. The main contribution of this paper is to analyze this problem from the token-level, providing new insights into its underlying causes. The authors propose OVERTONE, a new method to mitigate overfitting. This work advances the field by offering more granular understanding of the problem.", "essential_references_not_discussed": "No necessary related work has been omitted.", "other_strengths_and_weaknesses": "Please see above comments.", "other_comments_or_suggestions": "Please see above comments.", "questions_for_authors": "Recent research has begun exploring knowledge editing in the form of free text. Have the authors considered the problem of overfitting in this context?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n2GQSf982I", "reviewer": "Reviewer_drcx", "summary": "This paper addresses the problem of heterogeneous token overfitting (HTO) in knowledge editing (KE) for large language models (LLMs). The authors identify that existing KE methods, which indiscriminately optimize cross-entropy loss across all tokens, lead to varying overfitting rates for different tokens, degrading reasoning capabilities. They propose OVERTONE, a token-level smoothing method that adaptively refines target distributions by blending ground-truth tokens with filtered model predictions. Experiments across four KE methods, two LLMs, and diverse benchmarks demonstrate OVERTONE’s effectiveness in improving portability and locality while maintaining reliability.\n\n## update after rebuttal\nI keep my score as the final score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate your effort and time spent reviewing our paper and thank you for your expertise and constructive comments. In the following, we address your comments and questions one by one.\n\n>**The relation between the portability loss and the underfitting degree (UD).**\n\nYes, portability loss does not directly imply UD. In Sec 2.2, Fig 1 shows that high portability loss is attributed to overfitting. Toward a deeper understanding of such overfitting, in Sec 2.3, (negative) UD (NUD) is defined to uncover the token-level HTO, which is a key contribution of this work as admired by the reviewer. \n\nSpecifically, NUD represents a token is overfitted as its training loss is too small. Specifically, NUD computes the difference between the token's training loss (of edited model), and the greedy decoded token's pretrained loss (of unedited model). The choice of greedy decoding is on purpose, as it reflects the unedited model's most confident knowledge proper and valid in the past. By comparing the two, NUD indicates that the edited model is overly confident, and is \"overfitted\" thereof. We will make this clearer.\n\n>**Related work on Answer-level overfitting.**\n\nWe thank the reviewer for bringing this interesting work to our attention. After reading it, we agree that the neighboring knowledge perturbation due to the answer-level overfitting is insightful, and it would be interesting to explore bridging the two types of overfitting and building more principled solutions. We will highlight this referred paper, the connection, and this future direction in the revision.  \n\n>**Empirical Comparison with DPO.**\n\nFollowing the reviewer's suggestion, \nwe train LoRA with DPO from EasyEdit.\nWe use the pre-edited model's old knowledge as the negative data. \nDue to time constraint, we only conduct Single Editing on ZsRE.\nWe note that DPO performs worse, which we presume due to the practical challenge analyzed in Sec 3.2. \n}\n\n|        |       | Rel. | Gen.  | Por.  | Loc.  | Avg   |\n|--------|-------|------|-------|-------|-------|-------|\n| Llama2 | Ours  |  100 | 94.31 | 61.16 |  87.2 | 85.67 |\n|        | DPO   |  100 | 94.74 | 33.64 | 41.66 | 67.51 |\n| Llama3 | Ours  |  100 |  98.5 | 51.57 | 93.13 |  85.8 |\n|        | DPO   |  100 | 97.77 | 19.61 | 10.58 | 56.99 |\n\n>**Some case studies could make the contribution clearer.**\n\nThank you very much for the suggestion. \nDue to time constraint, we will dive into more in-depth visualization on OVERTONE effect, and how OVERTONE helps multi-hop reasoning on MQuAKE in the revised paper.\n\n>**MQuAKE contains 2-4 hops questions; why does you just do the 2-hops?**\n\nWe apologize for the misleading statement. In experiments we followed MQuAKE official Repo, DeepEdit, and EasyEdit to load the multi-hop questions. We didn't manually filter out 3 and 4 hops questions. After rechecking the source documents carefully, we noted that \"2-Hop\" is inaccurate and should be \"Multi-Hop\". We will correct these descriptions in the revised paper.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes\n- I do not understand the relation between the portability loss and the underfitting degree? Why the underfitting degree prove the model is overfit? It is not clear hear.", "relation_to_broader_scientific_literature": "This work advances knowledge editing (KE) by addressing token-level overfitting—a gap in prior KE methods (e.g., MEND, LoRA) that optimize generically across tokens. By integrating token-aware regularization and influence-function theory, it bridges fine-grained training dynamics with LLM robustness, offering a universal enhancement for KE frameworks.", "essential_references_not_discussed": "There are some answer-level overfitting analyses that I think should be mentioned. Token level overfitting is good but answer-level is also important.\n\n- Neighboring Perturbations of Knowledge Editing on Large Language Models, ICML 2024", "other_strengths_and_weaknesses": "Strengths:\n- The analysis is good and tackles the issues well.\n- The experiments are adequate and convincing.\n- The identification of HTO as a key bottleneck in KE is a significant contribution. The analysis of token-level loss dynamics provides a fresh perspective on overfitting in LLM editing.\n\nWeakness:\n\n- While the connection to DPO is intriguing, the paper does not empirically compare OVERTONE with DPO-based editing methods, leaving its practical advantages under-explored.\n- Some case studies could make the contribution clearer.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Some questions:\n\n- I remember the MQuAKE contains 2-4 hops questions; why does you just do the 2-hops?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a token-level smoothing method method called OVERTONE to address the issue of heterogeneous token overfitting (HTO) in knowledge editing tasks for large language models, aiming to improve the models' reasoning, generality, and locality with minimal computational overhead and without compromising their pre-trained capabilities.  

The method has been validated through extensive experiments involving multiple editing methods, two LLMs, and diverse scenarios, demonstrating its effectiveness and versatility. 

There are discussions and clarifies in the rebuttal period, with some additional experimental results.
All the reviewers give weak accept or accept recommendations.
Decision: Accept (poster)