BalancEdit: Dynamically Balancing the Generality-Locality Trade-off in Multi-modal Model Editing
Authors: Dongliang Guo
Mengxuan Hu
Zihan Guan
Thomas Hartvigsen
Sheng Li
Venue: ICML 2025 poster (2025)
Keywords: Model editing
Multi-modal model
Primary area: deep_learning->foundation_models

Abstract: Large multi-modal models inevitably decay over time as facts update and previously learned information becomes outdated. Traditional approaches such as fine-tuning are often impractical for updating these models due to their size and complexity. Instead, direct knowledge editing within the models presents a more viable solution. Current model editing techniques, however, typically overlook the unique influence ranges of different facts, leading to compromised model performance in terms of both generality and locality. To address this issue, we introduce the concept of the generality-locality trade-off in multi-modal model editing. We develop a new model editing dataset named OKEDIT, specifically designed to effectively evaluate this trade-off. Building on this foundation, we propose \textbf{BalancEdit}, a novel method for balanced model editing that dynamically achieves an optimal balance between generality and locality. BalancEdit utilizes a unique mechanism that generates both positive and negative samples for each fact to accurately determine its influence scope and incorporates these insights into the model's latent space using a discrete, localized codebook of edits, without modifying the underlying model weights. To our knowledge, this is the first approach explicitly addressing the generality-locality trade-off in multi-modal model editing. Our comprehensive results confirm the effectiveness of BalancEdit, demonstrating minimal trade-offs while maintaining robust editing capabilities. Our code and dataset are available at https://github.com/donglgcn/BalancEdit/tree/MMOKVQA.

BalancEdit:
Dynamically Balancing the Generality-Locality Trade-off in Multi-modal Model Editing
Dongliang Guo'
Mengxuan Hu
Zihan Guan
Thomas Hartvigsen
! Sheng Li'
Abstract
1. Introduction
Large multi-modal models (Zhu et al., 2023: Radford et al., 2021; Li et al., 2023; Liu et al., 2023a; Rombach al., 2022) have recently brought about significant advancements in artificial intelligence, demonstrating impressive results in tasks such as Visual Question Answering (VQA) (An- tol et al., 2015). However, these models are susceptible to issues like hallucination (Rawte et al., 2023) and fact alter- ation (De Cao et al., 2021; Guo al., 2024; Zhu al., 2024). After deployment, these models may generate numerous er- rors, leading potential problems like the propagation of hate speech (Guan et al., 2025b;a; Hu et al., 2025; 2024) or the dissemination of outdated factual information. Given these challenges, it is critical to continually update and main- tain these large multi-modal models to ensure their accuracy and relevance.
Large multi-modal models inevitably decay over time as facts update and previously learned infor- mation becomes outdated. Traditional approaches such as fine-tuning are often impractical for up- dating these models due to their size and com- plexity. Instead, direct knowledge editing within the models presents a more viable solution. Cur- rent model editing techniques, however, typically overlook the unique influence ranges of differ- ent facts, leading to compromised model perfor- mance in terms of both generality and locality. To address this issue, we introduce the concept of the pneerality-localiin trade-off in multi-modal model editing. We develop a new model editing dataset named OKEDIT, specifically designed to effectively evaluate this trade-off. Building on this foundation, we propose BalancEdit, a novel method for balanced model editing that dynam- ically achieves optimal balance between gen- erality and locality. BalancEdit utilizes unique mechanism that generates both positive and nega- tive samples for each fact to accurately determine its influence scope and incorporates these insights into the model's latent space using a discrete, lo- calized codebook of edits, without modifying the underlying model weights. To our knowledge, this is the first approach explicitly addressing the generality-localic trade-off in multi-modal model editing. Our comprehensive results confirm the effectiveness of BalancEdit, demonstrating min- imal trade-off's while maintaining robust edit- ing capabilities. Our code and dataset are avail- able at https://github. con/ donglgen/ Bal wataaaditttree///nnO
While retraining fine-tuning can update model's knowl- edge, it is often infeasible to frequently edit individual facts due to the high computational costs involved. Fortunately, model editing techniques (Hartvigsen et al., 2024; Mitchell et al., 2021; Zheng et al., 2023) provide a promising ap- proach to implementing cost-effective, targeted updates to large pretrained models. These techniques typically involve injecting new layers or modifying weights alter the knowl- edge embedded in language models. A successful edit gen- erally exhibits three characteristics (Mitchell et al., 2021; Huang et al., 2023): reliability, which ensures the output changes to the target answer for the same question; locality, which leaves unrelated knowledge and outputs unchanged; and generality, which produces the correct answer for all questions within the influence scope. As illustrated in Fig. 1, each fact has its influence scope. For instance, if we wish to edit the name of specific cat, the influence scope would be confined to that particular cat. If we aim to edit the name of a cat breed, the influence scope would extend to cats within that breed. However, we intend to edit the name of a species, the influence scope would encompass all cats. Consequently, we should consider each fact individu- ally and dynamically determine the appropriate influence scope.
Universitty Virginia, Charlottesville, USA. Correspondence to: Sheng Li <shengli @ virginia.edu>>.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
However, current model editing techniques often overlook the dynamic nature of the influence scope. Some meth-

---

BalancEdit: Dynamically Balancing the Generality-occaiiy Trade-off in Multi-modal Model Editing
Q: What is the name of the breeds in the image?
Original Image
Related Image
Unrelated Image
Black Imuge
A: Persian cat =>> Kitty C
Q: What the name of the species the image?
A: Cal Kilty &
Question
What brund this computer?
Figure I. Illustration of various influence scope
Target
hp
lenovo
Basc
dell
black
IKE
lenovo
lenovo
lenovo
Icnovo
MEND
lenovo
lenovo
lenovo
lenova
GRACE
lenovo
hp
dell
black
ods treat all influence scopes as if they are large and uni- form, while others focus solely on a specific edit. For in- stance, IKE (Zheng et al., 2023) employs in in-context learn- ing to edit knowledge, using the closest piece of knowl- edge as prompt to guide the language model. This ap- proach causes the language model to rephrase the nearest fact, resulting in an oversized influence scope. Conversely, GRACE (Hartvigsen et al., 2024), a lifelong model editing method, assumes that each edit has a small and similar in- fluence range, leading to limited generality. Consider an example where we aim to edit "fact" that HP computers have been renamed Lenovo, as shown in Table 1. Ideally, model editing should update the answer from HP to Lenovo whenever it encounters an image of a HP computer, while leaving the answer unchanged for other brands. However, existing model editing techniques, such as IKE (Zheng et al., 2023) and MEND (Mitchell et al., 2021), may achieve the target edit but neglect the influence scope, inadvertently editing other brands as well. Even when presented with black image, these models may still output the new swer, leading to hallucination. On the other hand, while GRACE (Hartvigsen et al., 2024) maintains the backbone model's answer for unrelated images, it fails to edit the knowledge to the desired scope. These observations suggest that existing multi-modal model editing methods struggle to dynamically adjust the influence scope of knowledge edit, and to balance generality and locality effectively.
Ours
lenovo
lenovo
dell
black
Table I. example of Amneraliyy-lccaliii trade-off. Red color means the false prediction and Green color indicates the correct prediction
iors. Furthermore, since BalancEdit's codebooks do not alter model weights and are fully model-agnostic, they also pave the way for ing. This is particularly useful for making critical spot-fixes between larger retraining efforts.
cost-effectiive model edit-
Our contributions are as follows: 1) We first formulate the generality-localee trade-off in multi-modal model editing and build a dataset named OKEDIT to empirically demon- strate it. 2) We introduce BalancEdit, an efficient method for multi-modal model editing that dynamically and effectively balances generality and locality without requiring training data beyond individual edits. 3) Our experiments reveal that BalaneEdit outperforms baseline models and consistently achieves SOTA performance across a range of metrics.
2. Related Work
Model Editing. Model Editing, which has recently drawn a lot of attention, aims make precise, targeted adjustments to the behavior of foundation models. This is crucial given that large foundation models may decay over time due to domain shifts and updates in knowledge, potentially leading to the dissemination of outdated factual information. Many approaches in this area suggest regularized-finetunne auxiliary data, such as instances from the original train- ing set semantically-simila edits (Sinitsin et al., 2020), while obtaining this data is increasingly challenging. With training data becoming proprietary and the collection of comantically-similae inputs less fcasible, there's 's need for innovative solutions. Some recent strategies utilize meta- learning to forecast edits (Mitchell et al., 2022b;a; De Cao et al., 2021) or decompose weight updates into simpler components (Meng et al., 2022a:b). To make edits more targeted, techniques like MEND (Mitchell et al., 2022a) and ROME (Meng et al., 2022a) and GRACE (Hartvigsen et al., 2024) take cues from efficient finetuning strategies (Yu et al., 2023b; Huang et al., 2023; Yu et al., 2023a; Li et al., 2024; Tian et al., 2024). However, these methods some- times demand additional finetuning and may overfit more
To address this issue, we first create a dataset designed to evaluate the trade-off between generality and locality in model editing techniques. We then introduce an effi- cient multi-modal model editing method named EdiancEdit,, which dynamically balances this trade-off with minimal computational costs. Specifically, into chosen layer of vision language model without altering its weights. This adaptor modifies layer-to-layer transformations for select inputs. By caching embeddings for input edits and the updated knowledge transformation layer, BalancEdit functions as a codebook where edits are stored. To strike a balance between generality and locality, we generate the corresponding positive and negative sam- ples for each edit. The model's semantic similarity in its latent space can be visualized as dynamic spheres around cached edits, with the radius determined by the distance between positive and negative samples. By adjusting the radius over time, BalancEdit allows for immediate edits, retains previous edits, and preserves correct model behay-
using
incorporate adapter
2

---

BalancEdit: Dynamically Balancing the Generality-Localiiy Trade-off in Multi-modal Model Editing
than traditional methods (Zhong et al., 2022) and few of them consider the locality property. MEND (Mitchell et al., 2021) notices the locality issue and designed contrastive loss to keep the locality. Despite these advancements, there remains a substantial gap in model editing methods tailored for muuli--odal models. Only limited research (Cheng et al., 2023) has explored the potential of multi-modal models in this context. In our work, we stick to this problem, in- vestigating the trade-off between generality and locality in multi-modal model editing and offering an efficient method to address it.
ity samples. Additionally, there are two bonus properties. (4) Multiple Edits. The model could edit multiple times without forgetting previous edits, (5) Efficiency. The model editing method should take minimal costs to edit model, such as less training time and data costs.
3.2. BalancEdit
As illustrated in Fig. 2, to satisfy the aforementioned prop- erties, we propose BalancEdit, an efficient model editing method for multi-modal models that dynamically deter- mines the equilibrium between generality and locality with- out compromising the original model. BalancEdit operates by wrapping selected layer of the pre-trained model with BalancEdit module. This module consists of codebook and a mechanism that dynamically determines the radius of the influence scope.
Large Vision Language Models. Vision language mod- els (Radford et al., 2021; Zhu et al., 2023; Li et al., 2023; 2022; Wang et al., 2024a; Zhou et al., 2024; Lin et al., 2024; Dai et al., 2024) are one of the key part in multi-modal learn- ing, which aim to learn multi molli- foundation models with improved performance on vision language tasks (Antol et al., 2015; Guo et al.; Wang et al., 2024b). These models (Li et al., 2022; Liu et al., 2023a), by mapping image embed- dings to text embedding space, are capable of interpreting image information and handling a wide array of tasks. They demonstrate impressive abilities in image understanding, generation, and reasoning. These capabilities, however, rely heavily on millions of high-quality training data (Schuh- mann et al., 2022; 2021). Given that factual knowledge, especially visual information, changes over time, it is cru- cial to keep the model up-to-date. However, updating the model's behavior through retraining or fine-tuning is im- practical due to exorbitant training costs. In this context, multi-modal model editing techniques, which allow for tar- geted edits, provide feasible solution to this challenge.
BalancEdit Codebook. To store the updated knowledge of the pre-trained multi-modal model, we design a discrete codebook at layer I which contains three components.
Keys (K): Each & in the codebook is a represen- tative embedding for a specific edit. Each k stores the averaged embedding produced by the layer I - 1 for a specific question answer pair. Mathematically, it can be expressed as K {k ", 1. EE fi, ((2 t) E Dedic..
Transformations (V): The transformation refers to a specific layer in the LLM. During an edit, we fine-tune this transformation layer to encode the new knowledge. Each transformation v--) associated with a specific key k stores the new weights with the updated knowledge. Typically, the transformation is fine-tuned with the model's finetuning loss with updated knowledge.
3. Methods
3.1. Problem Formulation
The multi-modal model editing is to edit a multi-modie LLM
Influence radius (E): The radius corresponding to a key & indicates the influence scope of a (i, 1")) pair. It serves as a threshold for similarity matching. The edited transformation is activated only if the embed- ding falls within the influence radius. The radius varies for each key, and is determined by the positive and neg- ative samples of specific knowledge pair (i, t, v").
fiase that maps the image input (i) and text prompt (1) from the out-dated answer (y") to the new target prediction (y") with the updated model fnew- For the related inputs R.,,, the updated model should give the target prediction, while for the unrelated inputs Uz.,, the prediction should be retained. In addition, when given a batch of inputs (i, t, v²)= Dedit the updated model could remember all edits without for- getting previous edits. Specifically, the multi-modal model editing should follow the following properties: (1) Reliabil- ity. The updated model should output the target answers: fnew ti = y", (i, t,, y²) E Deai; (2) Generality. The updated model should answer the target output given re- lated inputs: fnew (i', ,t')= = y², , (i', t²)) E Re,e (3) Locality. The updated model should keep the output retained on the unrelated inputs. [i', (i', (') = Jbase ('), (i', E U..- Thus, to achieve both generality and locality properties, it necessary to distinguish the generality samples and local-
Codebook Constructions. To make an edit, the BalancEdit module needs to create a new codebook entry (h v()), e)) The key is the averaged embedding generated by the layer - 1, which is an anchor point for lookup. Thus, when a new question is passed into f. the codebook is activated to compare whether the embedding relates to any key in the codebook. If the embedding falls within the influence scope of a key, the edited transformation is activated to generate a new embedding for layer I I+ 1; otherwise, the original transformation is retained to process the question.
3

---

BalancEdit: Dynamically Balancing the Lencrality-Locality Trade-off in Multi-modal Model Editing
Vision Language Model
BalancEdit module
Transformer layers
Keys
Transformations
BalancEdit module
Neg
Edited transformation
Original transformation
Outside
Neg
Neg
Radius
Pos
radius
radius
Inside
Pos: Original image, rephrase question. Black image, original question. Radius: (1-a) [|Pa|]||Pos|]| + a //||ll
Neg:
Transformer layers
ViT + Linear Projector Tokenizer + Embedder
Pox
Pox
Figure Overview of our BalancEdit framework. BalancEdit makes edits by learning, saving, and retrieving transformational edits between layers. The BalancFdit module consists of discrete keys, transformations, and dynamic influence radius. Additionally, the BalancEdit module can handle multiple edits over time by adding new entries to the module.
The formulation is as follows:
SIEE CORE stor
Esia compater
DiE compter
this copputer
cala consutrr
(1(7)) if min(d(M K))
n1.
(H otherwise
centor
pos Five sample
generality
radlos
Tocal ity somple
negitivv sarple
sample
Editing Transformations. When new fact requires an update, the transformation revised incorporate this new fact and knowledge. To ensure that the transformation ac- curately learns the new fact, we finetune the transformation layer directly using backpropagation through the language learning loss. The target transformation 2² can be formu- lated as:
Figure Illustration of influence radius determination
without altering the overall architecture of the model. The parameters that tuned include all the weights within the specific layer of the network.
Influence Radius Determination. As shown in Fig. 1, each fact has its unique influence scope. However, exist- ing methods do not consider the dynamic influence scope during the editing process, which results an imbalanced generality-
0² arg min L(Ineso (i, v),
where the language model loss, specifically the next- token prediction loss used by the base LLM. This loss en- sures that the updated model fnew, when processing input (i,t), generates the desired new answer Yn- Specifically, if the key is empty or the new fact falls outside the influ- ence scope of existing keys, the transformation is directly finetuned from the original transformation layer. However, there may be instances where the new fact overlaps with existing keys. In such cases, we finetune the transfor- mation layer from the previously edited transformation to prevent catastrophic forgetting. Additionally, if the new key directly conflicts with previous edits, will discard the previous entry and add new one to update the knowledge. To ensure the universality, we primarily utilize the basic full fine-tue min-ttnnigg approach as the transformation method. This involves adjusting the weights of the neural network to bet- ter align with the newly introduced or modified knowledge
trade-off, as illustrated in Table To address this issue, BalaneEdit incorporates dynamic influ- radius determination mechanism. As depicted in Fig. 3, the knowledge of the fact is at the center of the influence scope. Ideally, the radius should encompass the majority of generality samples, while excluding locality samples. Since similar semantic sentences will result in close embed- dings (Liu et al., 2023b; Menon Vondrick, 2023), we can use it to find an efficient way to approximate this process. Specifically, construct positive and negative samples to dynamically estimate the influence scope without model training or external knowledge.
To construct a positive sample, we need to design general rephrasing method that is highly similar to the fact itself. We find that rephrasing the text will not affect the semantic in-

---

BalancEdit: Dynamically Balancing the Generality-Locaiit Trade-off in Multi-modal Model Editing
formation of the edited knowledge. Therefore, we rephrase the text prompt & while keeping the image input i unchanged. The positive sample can be formulated as (i, R((t), where R(t) denotes the rephrased text prompts. The generation of a rephrased prompt is efficient, requires no additional data or training process, and can be generated directly by the backbone model.
Train
Test
Generality
Locality
Goal
rundou sample,
MMEDIT
6036
2093
1 per question
casier eval
visual understanding
OKEDIT
9009
5046
10 per question
semantic sample.
harder ovail
visual reasening with open question
Table 2. Statistics comparison OKEDIT.
between
MMEDIT and
On the other hand, the negative sample should be close to the border of locality samples to accurately estimate the radius. Additionally, the generation process should be ef- ficient and fact-agnostie. In this case, we use a pure black image as the image input, which contains no semantic in- formation on the image side. The choice of black images as a proxy for out-of-scope knowledge is based on their characteristic as minimal or null visual signals. This makes them universally applicable negative samples across various visual recognition tasks. Furthermore, the generation of a negatíve sample is highly efficient, and can be applied to almost all knowledge editing tasks.
uations. 2) We introduce new dataset, OKEDIT, based on the OKVQA dataset (Marino et al., 2019), which includes 5046 testing samples, encompassing over 20 unique cate- gories such as vehicles, people, plants, animals, geography, history, language, brands, science and technology. Unlike MMEDIT, OKEDIT enhances the quality of the rephrased images and adjusts the difficulty of the locality samples to evaluate the trade-off between generality and locality. Detailed information about the datasets is provided in Ap- pendix A.
Backbone Models. Following previous work (Zheng et al., 2023), we adopt two vision language models as the base models. MiniGPTT- (Zhu et al., 2023) is a powerful vision language model, leveraging Vicunna (Chiang et al., 2023) as the language model and Vit-G/14 from EVA-CLIP (Sun et al., 2023) and a Q-former as the image encoder. BLIP-2 OPT (Li et al., 2023) utilizes a lightweight Q-former to bridge the gap between vision modality and text modality, where the ViT-L is adopted in the vision block, and the unsupervised -taiined OPT model (Zhang et al., 2022) is used for decoder-based L.M..
After obtaining the positive and negative samples, we can estimate the influence radius by aggregating the distances between the center and the constructed samples. Specifi- cally, the radius could be formulated as:
e= d(Pos, k) + a k),
where a is the hyperparameter to adjust the distance, d(-) denotes the distance function, and l is the key in the code- book entry which also represents the center of the influence scope.
Metries. Following previous work (Zheng et al., 2023), we adopt the Editing Success Accuracy (Acc); Text Generality (T-Gen); Image Generality (I-Gen); and Locality (Loc) as the main metrics. To quantify the trade-off between general- ity and locality, we introduce the harmonic mean (HM) of the T-Gen, I-Gen and Loc. The detailed informations are in Appendix C.
4. Experiments
To evaluate the properties discussed in Sec. 3.1, we conduct experiments from three perspectives: 1) The primary mo- tivation of BalancEdit is to balance generality and locality. Therefore, we create a dataset named OKEDIT to address the quality issues of existing datasets and conduct experi- ments on it. 2) We assess the performance of multiple edits, and 3) we compare the training time and the data costs of an editing method to evaluate its efficiency.
4.2. Baselines
We compare four model editing methods with different mechanisms. First, finetuning (FT) is a basic model editing method. To ensure a fair comparison, we only fine-tune the specific layer of the pre-trained model, maintaining the same parameter sizes. Second, In-context Knowledge Editing (IKE) is an in-context learning model editing method origi- nally designed for pure language models. We have revised the method to adapt it vision- languaee models. It utilizes an unsupervised retriever to prompt relevant facts from the training set. Additionally, MENN(Mitthhell et al., 2021), )), metalearning-based model editing method, requires exten- sive in-distribution training data to learn a meta-network that predicts the edited weights of the pre-trained model. Finally, we adapt GRACE (Hartvvige et al., 2024) to vision lan- guage models. GRACE, a memory- augmenee model edit-
4.1. Datasets and Backbone Models
Datasets. Since there are few published vision language model editing datasets, we perform extensive experiments on two such datasets in the vision question answering task (Antol et al., 2015): 1) MCeD al., 2023), the first multi-modal model editing dataset based on the VQoyv2(Goyal et al., 2017) dataset, which includes 2093 testing samples; However, this dataset has its limitations as shown in table 2. The content of images generated from image caption prompts can deviate from the original images, leading to inconsistencies and potentially less accurate eval-
5

---

BalancEdit: Dynamically Balancing the Generality-Localiy Trade-off in Multi-modal Model Editing
Backbone
Dataset
Method
Pretrain
miniGPT4
BI.IP-2 OPT
Acct
T-Gent
I-Gent
Loct
HM
Acct
T-Gent
I-Gent
Loct
HM
Base FT
15.04
14.21
13.56
NA
NA
8.50
8.52
6.89
NA
NA
X
96.53
95.88
96.20
3.20
9.00
99.96
99.41
97.05
0.27
0.80
IKE
v
100.00
95.57
100.00
15.47
20.07
99.83
94,47
99.58
11.96
28.77
MMEDIT
MEND
V
98.39
96.58
97.77
68.82
85.43
97.23
95.86
96.81
69.40
85,29
GRACE
X
79.82
74.49
70.
91.66
77.72
74.27
62.90
35.24
90.26
54.19
BalancEdit (Ours)
X
100.00
99.90
98.91
71.74
88.08
100.00
99.16
90.30
80.04
89.14
Base
X
30.42
45.40
72.21
NA
NA
14.35
13.96
15.22
NA
NA
FT
X
99.69
99.45
99.38
5.52
14,90
99.97
99.54
96.77
0,43
1.27
IKE
V
99.71
97.78
99.76
17.45
38.68
99.35
94.20
99.66
13.29
31.28
OKEDIT
MEND
V
94.44
90.80
95.39
36.20
61.07
90.82
82.82
88,25
28.89
51.70
GRACE
X 87.84
28.31
29.46
99.99
37.84
54.13
50.67
28.30
94,48
45.69
BalancEdit (Ours)
X
100.00
99.87
76.46
53.14
71.58
100.00
98.89
65.38
61.18
71.85
Comparison results of BdlancEdit with the model editing baselines on two backbone models. Base refers to the backbone model without any knowledge editing. The pretrain column indicates whether model editing method requires pre- pre-training model or the training data. The best results are shown in Bold.
Table 3.
ing method, also supports lifelong model editing. It caches the target value of the updated fact, achieving lightweight model editing.
ically, we examine both the accuracy and the trade-off be- tween generality and locality. First, in terms of editing success accuracy, BalancEdit achieves the highest perfor- mance, resulting in 100% editing success across all datasets and backbones. In contrast, baseline models do not consis- tently reach this level of performance. This demonstrates that our BalancEddi satisfies the Reliability Property.
4.3. Implementation Detail
In our comparisons of Finetuning, MEND and GRACE, we explore learning rates of 1.0, 1e-1, 1e-2, le 1e-4, and 1e-5 We observe that Finetuning, Memory, and MEND perform best with 1e The choice of layer to edit is another hyperparameter for all editors. In all our editor comparisons, each editor modifies the same layer. For miniGPT-4, this is the dense layer of the llama block (1lama_mode) model....... 1. layers [31] . up.pro j), for BLIP-2 OPT moder, is the OPT decoder layer fopt_mode]] wtt.mmmeemmmmelll..mmmmmmmm decoder. layers 3.1.ff22 Recent work supporting the importance of selecting the correct layers to fine-tune corroborates this (Cheng et al., 2023). However, it's important to note that the choice of layer is a practical hyperparameter: for comparison purposes, we ensure editors are compared when editing the same layers. For the distance function, we use the Euclidean distance if it is not explicitly mentioned. We select using a small held-out set of only 5 unrelated samples. Since different models may have different latent feature distributions, a is treated as model-dependent. Once chosen, the same a is fixed per model (e.g., o=0.2 for MiniGPT-4) for all evaluations.
For the Generality metric, BalancEdit achieves the best text generality performance compared to other methods. For instance, BalancEddi shows a 70% improvement in text gen- erality accuracy over the GRACE method. Additionally, it reaches comparable performance in image generality. Since the MMEDIT dataset is relatively simple, the performance is very similar across all model editing methods, converging around 99%. However, in the challenging OKEDIT dataset, where we focus on balancing trade-off performance, we must compromise on the more difficult aspects of image generality. The high generality performance underscores the Generality Property of our method.
For locality performance, BalancEdit consistently achieves the best results, with the exception of the GRACE method, which primarily focuses on specific local edits. Specifically, BalancEdit shows an improvement in locality 20% to 80% compared to other baseline methods. For example, on the OKEDIT dataset using the BLIP-2 OPT backbone, Bal- ancEdit outperforms the MEND method by 30,, despite the fact that MEND requires extensive training data and time. This further validates the Locality Property of our BalancEdit method.
4.4. Comparisons to Existing Methods
Table 3 presents the main results of our BalaneEdit and other baseline methods on the VQA task. We observe that our BalancEdit significantly outperforms the existing editing methods without requiring additional training data. Specif-
To compare the overall performance in balancing the trade- off between locality and generality, we calculate the har- monic mean of T-Gen, I-Gen, and Loc. Our BalancEdit method achieves the highest scores compared to other base-
6

---

BalancEdit: Dynamically Balancing the Generality-Localit Trade-off in Multi-modal Model Editing
Sequential
Acct
T-Geat
I-Gien
Loct
HM
Training time (h)
Editing time (s)
FT
K
99.25
99.21
98.64
0.74
2.18
FT
0
3,91
IKE
X
100.00
96.86
100.00
16.91
37.75
93.74
89.98
95.38
37.49
62.14
IKE MEND GRACE BalancEdit
12
0.38
GRACE
87.78
25.96
24.21
99.99
33.39
22
1.48
Balanckdiit (Ours)
X
100.00
100.00
72.31
54.40
71.07
32.67
BalancEdit (Ours)
100.00
99.70
72.29
46.25
65.95
8.04
Table Comparison results of BalancEdit with the model editing baselines about multiple sequential editing. The sequential column indicates whether the method uses sequential editing or not.
Table 5. Time efficiency evaluation results on BLIP-2 OPT.
What denomination is the church?
What religious affiliaion does this church have?
A: catholic
A: catholic
baplist catholic
GT: catholic
methodist
lines across all experimental combinations, demonstrating the minimal trade-off between generality and locality perfor- mance. Specifically, in the simpler MMEDIT dataset, Bal- ancEdit outperforms the strongest baseline, MEND, by 3% and surpasses other baselines by up to 89%. Furthermore, in the more challenging OKEDIT dataset, our results are even more impressive, outperforming the MEND baseline by between 10% and 20%. As expected, these performances highlight the effectiveness of our dynamic influence scope mechanism and validate the Reliability, Generality, and Locality Properties of our method.
editing knowledge
successful
fallure
Figure 4. An example of the interpretable output
training meta-net. For instance, MEND, a meta-learning method, requires 22 hours to pre-train on 6,346 training samples. IKE, a retrieval-augmentedd in-context learning method, needs 12 hours to index 6,346 knowledge facts in advance.
On the other hand, editing time refers to the duration re- quired to edit single new fact. We compare with GRACE as both methods are types of evemory-augmente model editing. A successful edit with our method takes approxi- mately 8.04 seconds, whereas GRACE takes 32.67 seconds, making our editing speed three times faster than GRACE. IKE requires less editing time because bypasses training and instead retrieves the most similar fact.
4.5. Sequential Editing Evaluation
To further investigate performance across multiple sequen- tial edits, we evaluated our BalancEdit system on 50 se- quential edits using the OKVQA dataset with a miniGPT-4 backbone. The results, as shown in Table 4, indicate a slight drop in performance for sequential edits compared non- sequential ones. Specifically, metrics such as edit success accuracy and generality remain comparable with those ob- served in non-sequential editing scenarios, suggesting that the system's 's reliability and generality are maintained. Al- though there is a slight decrease in locality performance, still exceeds that of other baselines. This decrease is expected, as an increase in the number of keys can lead to unwanted collisions, potentially degrading performance. Notably, despite the slight performance reduction in sequen- tial editing, our BalancEdit system continues to outperform baseline models that do not incorporate sequential edits. This performance across multiple edits substantiates the Multiple Edits Property of our system.
Data Efficiency. Similar to training time costs, the require- ment for additional training data significantly influences the feasibility of model editing methods. Our BalancEdit does not require any extra data, as can generate both positive and negative samples internally. Specifically, a rephrased question for a positive sample can be obtained by query- ing the backbone model, and black image for a negative sample can be directly generated. This efficiency supports the Efficiency Property. In contrast, methods like MEND and IKE require extensive additional in-distribution data, leading to less feasibility for real-world scenarios.
4.7. Interpretability
Interpretable Codebook. The codebook is interpretable because the editing knowledge is explicitly stored, with each entry corresponding to an update in knowledge and its specific influence scope. Additionally, the codebook is de- tachable and can be thoroughly inspected, allowing edits to be easily located and detected. Each piece of updated knowl- edge has an entry in the codebook, enabling it to be reversed without impacting the model, particularly in sequential edit- ing scenarios. This interpretable codebook minimizes harm to the model while maintaining controllability.
4.6. Efficiency Evaluation
We compare the efficiency of our editing approach with recent advanced baselines, focusing on both time and data efficiency. Time efficiency encompasses both training and editing time, while data efficiency refers to the amount of additional data required for editing.
Time Efficiency. Training time is divided into two compo- nents, as detailed in Table 5. The first component is pre- training time, which involves either pre-training the model editing method or preparing the augmented index, such as
Interpretable Inference. Existing model editing methods
7

---

BalancEdit: Dynamically Balancing the Lencraltty-Localiy Trade-off in Multi-modal Model Editing
100
Dutaset
Function
Acct
T-Geat
I-Gent
Loct
HMf
Buc
100
99.9
98.91
71.74
88.08
90
MMEDIT
Cos
100
99.9
97.96
76,28
90.01
80
OKEDIT
Fuc
100
99.87
76.46
53.14
71.58
Cos
100
99.87
84.26
42.37
65.95
70
Aec
Table 6. Results of the effect of different distance function.
50
T-Gen
1-Gen
Negative Anchor
Acct
T-Gen?
I-Gent
Loct
HMT
40
Loo
Black
100.00
99.00
69.16
59.99
72.76
30
White
100.00
99.00
65.79
63.85
73.23
0.10
0.15
0.20
0.25
0.30
Figure 5. Results of the effect of the hyperparameter a.
Table 7. Comparative results using alternative negative anchors
typically update the knowledge within the model but do not provide a means to trace how these updates influence the model's output. Specifically, while the model's outputs may change, it is unclear how these changes are influenced by the updates and whether they are relevant to the posed ques- tion. In contrast, BalancEdit offers a human-unddee understandable explanation for adjusting model behavior. As illustrated in Figure 4, we edit the counterfactual scenario where 'baptist church' is changed to 'catholic church'. In successful case, we correctly answer the question because the image displays a symbol of the baptist church, even though it is not explicitly shown. According to the closest BalancEdit key, we can infer that the output is influenced by the edited knowledge, In contrast, in a failure case, we can determine that the incorrect prediction arises because the image closely resembles the edited fact.
related to image generality tend to deviate from the key, despite sharing similar semantic content. Consequently, as the radius decreases, the edited model tends to overlook these questions. Conversely, the model's performance on locality improves with an increase in a. A smaller radius helps preserve the integrity of unrelated questions, ensuring that their answers remain unchanged. In this scenario, we observe that the harmonic mean of generality and locality imitially increases and then decreases, further validating the existence of this trade-off. However, our method continues to achieve relatively high performance.
Effect of Alternative Negative Anchors. To further vali- date the effectiveness of our approach, we conducte experi- ments using various negative anchors, including white nega- tive images, on a subset of the OKEDIT dataset. As shown in Table 7, both black and white negative samples achieve 100% editing accuracy and exhibited a high harmonic mean in the locality-generallee trade-off. The performance metrics for both white and black negative anchors, such as accuracy, generalization metries, and locality, are remarkably consis- tent. The slight variations in the locality and I-Gen metries suggest that white images can function as effective negative anchors, which also lack significant discriminative infor- mation. This consistency across different negative anchors highlights the robustness and adaptability of our pipeline in various settings and confirms our assumptions.
4.8. Ablation Study
In this ablation study, we analyze the key design choices of BalancEdit. We first examine the effect of the hyperparame- ter a, which controls the trade-off between generality and locality. We then evaluate the impact of alternative nega- tive anchors and different distance functions to validate the generalizability. Additionally, we compare different nega- tive sampling approaches and editing layers to assess the robustness of our method across different configurations. Finally, we conduct an extreme case analysis to demonstrate the behavior of BalancEdit in challenging scenarios.
Effect of the Distance Function. The distance function serves method for calculating the similarity between two embeddings. In particular, we employ the Euclidean distance (Euc) and cosine similarity (Cos) as the distance metrics. To assess the versatility of our BalancEdit in terms of the distance function, we compare these two popular distance functions, as illustrated Table 6. We find that the results between them are remarkably similar. Specifically, both achieve around 100% editing success accuracy and text generality. While there are some differences in image generality and locality, both functions yicld comparable results. This is expected as the distance function alters the similarity between embeddings, but the semantic meanings for the positive and negative samples are still preserved.
Effect of the Hyperparameter. In this study, we conducte a series of experiments on a subset of the OKVQA dataset to investigate how the parameter a affects the trade-off be- tween generality and locality in model editing. As illustrated in Figure 5, we vary a from 0.1 and 0.3. The results, de- picted in the figure, show that the editing success accuracy and text generality metrics consistently maintain a 100% accuracy rate. This stability is attributed to these metrics be- ing closely tied to the key, with changes in the radius having no significant impact on them. However, the image gener- ality metric, which is more challenging, shows a decline as a increases. This trend is anticipated because questions
8

---

BalancEdit: Dynamically Balancing the Generality-Locality Trade-off in Multi-modal Model Editing
Method
Edit Ace
T-Generality
I-Generality
Locality
HM
Model
BalancEdit
100.00
98.89
65.38
61.18
71.85
BLIP-2 OPT
Random Negative Sample
100.00
100.00
49.12
65.08
65.61
BLIP-2 OPT
BalancEdit
100.00
99.87
76.46
53.14
71.58
MiniGPT-4
Random Negative Sample
100.00
99.00
66.93
45.92
64.08
MiniGPT-4
Table 8, Ablation study on negative sampling approaches
Layer
Acc
T-GGe
I-Gen
Loc
HM
Model
cality score, as the edit is applied too broadly across the representation space. In contrast, when a = 1, the radius is determined solely by the positive sample, resulting in over-localization and poor generalization, e.g.. 24.22 image generality score, since the edit is confined to a region that is too narrow. These results highlight the importance of balancing positive and negative influences to achieve both generality and locality.
30
100.00
100.00
64.03
66.39
73.75
MiniGPT4
31
100.00
99.87
76.46
53.14
71.58
MiniGPT4
30
100.00
100.00
78.43
44,99
66.70
BLIP-2 OPT
31
100.00
98.89
65.38
61.18
71.85
BLIP-2 OPT
Table 9. Ablation study on different editing layers.
Acc
T-Gen
I-Gen
Loc
HM
100.00
100.00
95.19
16.30
36.65
100.00
45.94
24.22
100.00
41.06
5. Limitations
Table 10. Results of extreme cases of a
Multi-modal model editing is a novel and challenging field, with the balance between generality and locality remaining largely underexplored. It is evident that employing similar- ity search across a model's layers inevitably slows down inference times (Hartvigsen et al., 2024), despite reducing the need for extensive training. Thus, accelerating inference time represents a crucial area for future improvements.
This success highlights the effectiveness of our codebook strategy, as it can dynamically adapt to different distance functions while maintaining a similar influence scope.
Effect of Negative Sampling Approaches. To verify our negative samples constuction approach, we compare our method with random negative sampling approach by ran- domly choosing irrelevant text-pair samples. As shown in Table. 8, BalancEdit consistently outperforms the random baseline in harmonic mean, demonstrating a better balance between generality and locality. This supports the effec- tiveness of our black image-based negative anchor, which offers a fact-agnostic, consistent, and efficient way define lower bound in the representation space. In contrast, ran- dom negative samples rely on external, unrelated examples and are often unstable in both quality and relevance, requir- ing assumptions that may not hold across diverse domains. Our method avoids these issues, making it more robust and scalable for wollwwrldd editing scenarios.
Another limitation is the memory-augmented approach's handling of multi-hop model editing. The key-based sim- ilarity search struggles to capture multi-hop queries that depend on newly introduced knowledge, often due to the ambiguity of real-world facts. For example, if the CEO of X (formerly Twitter) were to change to Elon Musk, it would be difficult to update the response to the question, 'Which social app is headed by the leader of SpaceX?' A potential solution to this problem could involve dynamically defining the fine-graine fine-grained influence scope, which would allow for more precise adjustments to changes in their implications for multi-hop questions.
facts and
Effect of different editing layers. To evaluate the robust- ness of our method across different editing layers, we apply our method to two separate layers, as shown in Table 9. The results show that our method maintains strong per- formance even when a different layer is chosen. Specif- ically, we observe higher harmonic mean performance on the Min model, further demonstrating the robustness of our approach.
6. Conclusion
In conclusion, we identified the limitation of existing imban- lanced generality and locality in model editing. Specifically, we formulated the generality-locality trade-off, and devel- oped specialized dataset, OKEDIT, to empirically explore this phenomenon. In addition, we introduced BalancEdit, an innovative approach for multi-modal model editing that efficiently balances the generality and locality of edits. Our method reduces the need for extensive retraining or fine- tuning, relying solely on the data provided by individual edits. The experimental results demonstrate that BalancEdit significantly outperforms existing baseline models, consis- tently achieving state-of-the-ar performance.
Extreme case analysation. In order to study the extreme case of hyperparameter, we conduct ablation experiments using extreme values of a on MiniGPT4 with OKEDIT dataset in Table 10. When a = 0, the influence radius is entirely determined by the negative sample, which leads to over-generalization and reduced locality, e.g., 16.30 lo-
9

---

BalancEdit:
Dynamically Balancing the Gencrality-oocalit Trade-off in Multi-modal Model Editing
Acknowledgments
severely breaks safery. arXiv preprini arXiv:2505.06843 2025b.
The work is supported in part by the U.S. Office of Naval Re- search Award under Grant Number N00014-24-1-22 1-2668,, and the National Science Foundation under Grants IIS-2316306 and CNS-2330215.
Guo, D., Fu, Y., and Li, S. Ada-VAD: Domain Adaptable Video Anomaly Detection, pp. 634 642. doi: www...rg...a..o................ owg.
URL wtwpg////...oo doi abs/10.
Sttps://epuses swmuubsssiamm.... 0157/1.97861111
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
Guo, D., Hu, M., Guan, Z., Guo, J., Hartvigsen, T., and Li, S. Backdoor in seconds: Unlocking vulnerabilities in large pre-trained models via model editing, 2024. URL https: aaaii.iiiiiv..i. wttp://.rggabb////////a//
Hartvigsen, T., Sankaranarayanan, S., Palangi, H., Kim, Y., and Ghassemi, M. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, 36, 2024.
References
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L.. and Parikh, D. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Hu, M., Wu, H., Guan, Z., Zhu, R., Guo, D., Qi, D., and generation
Li, S. No free lunch: Retrieval-augmente undermines faimess in llms, even for vigilant users, 2024. URL https::/ / arxiv. org/ abs 2410. 07589.
Cheng, S., Tian, B., Liu, Q. Chen, X., Wang, Y., Chen, H., and Zhang, N. Can we edit multimodal large language models? arXiv preprint arXiv:2310.0S3 0.08475, 2023.
Hu, M., Guan, Z., Zeng, Y., Guo, J., Zhou, Z., Zhang, J., Jia, R., Vullikanti, A. K., and Li, S. Mind control through causal inference: Predicting clean images from poisoned data. In The Thirteenth International Conference on Learning Representations, 2025.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng. L., Zhuang, S., Zhuang. Y.. Gonzalez, J. E., et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https:/vicuna. Imsys. org (accessed 14 April 2023). 2(3):6, 2023.
Huang, Z., Shen, Y., Zhang, X., Zhou, J., Rong, W., and Xiong, 7. Transforrer--attteei One mistake worth one neuron. arXiv preprint arXiv:2301.0977 2023.
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P. N., and Hoi, S. Instructblip: Towards general-pune purpose vision-language models with instruction tuning. Advances in Neural Information Processing Sys- tems, 36, 2024.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Confer- ence on Machine Learning, pp. 12888-12900, PMLR, 2022.
De Cao, N. Aziz, W., and Titov, I. Editing fac- tual knowledge in language models. arXiv preprint 1oXiv:2/04.08162 2021.
Li, J., Li, D.,, Savarese, S., and Hoi, Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19742. PMLR, 2023.
Diederik, P. K. Adam: A method for stochastic optimization. 2014.
Goyal, Y., Khot, T., Summers-Stay. D., Batra, D., and Parikh, D. Making the in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 6913, 2017.
Li, X., Li, S., Song, S., Yang, J., Ma, J., and Yu, Pmet: Pre- cise model editing in a transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18564 18572, 2024.
Guan, Z., Hu, M.. Li, S., and Vullikanti, A. K. Ufid: A unified framework for black-box input-level backdoor detection on diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 27312-27320, 2025a.
Lin, B., Tang, Z., Ye, Y., Cui. J., Zhu, B., Jin, P., Zhang, J., Ning, M., and Yuan, L. Moe-Ilava: Mixture of ex- perts for large dision-languae models. arXiv preprint urXiv:2401. 15947, 2024.
Guan, Z., Hu, M., Zhu, R., Li, S., and Vullikanti, A. Benign samples matter! fine-tuning on outlier benign samples
Liu, H. Li. C., Wu, Q., and Lee, Y. J. Visual instruction tuning, 2023a.
10

---

BalancEdit: Dynamically Balancing the Generality-Localiiy Trade-off in Multi-modal Model Editing
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L.,, and Chen, W. What makes good in-context examples for GPT-3? In Agirre, E., Apidianaki, M., and Vulić, I. (eds.), Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 100- 114, Dublin, Ireland and Online, May 2022. Association for Computational Linguistics. doi: 1/20222 deelio- 1.10. URL htttps://aclaaaet http::////aaaaaooogg....... org/ 2022.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip- filtered 400 million image-text pairs. arXiv preprint arXhv:2111.02114 2021.
Liu, J., Zhang, Y., Chen, J.-N., Xiao, J., Lu, Y., A Land- man, B., Yuan, Y., Yuille, A., Tang, Y., and Zhou, Z. Clip-driven universal model for organ segmentation and tumor detection. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 21152-21166 2023b.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278 252944, 2022.
Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok-vqa: A visual question answering benchmark requir- ing external knowledge. In Proceedings of the IEEF/vvf conference on computer vision and pattern recognition, 2019.
Sinitsin, A., Plokhotnyuk, V., Pyrkin, D., Popov, S., and Babenko, A. Editable neural networks. In International Conference on Learning Representations, 2020.
pp.
Sun, Q. Fang, Y., Wu, L., Wang, X., and Cao, Y. Eva- clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. /5389, 2023.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems, 2022a.
Tian, B., Cheng, S., Liang, X., Zhang, N., Hu, Y., Xue, K., Gou, Y., Chen, X., and Chen, H. Instructedit: Instruction- based knowledge editing for large language models. arXiv preprint arXiv:2402. 16/23. 2024.
Meng, K., Sharma, A. S., Andonian, A., Belinkov, Y., and Bau, D. Mass-editing memory in transformer. arXiv preprint arXiv:2210.072299 2022b.
Wang, W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zeng, G., Luo, P., Lu, T., Zhou, J., Qiao, Y., et al. Visionllm: Large language model is also an open-ended decoder for vision- centric tasks. Advances in Neural Information Processing Systems, 36, 2024a.
Menon, S. and Vondrick, C. Visual classification via de- scription from large language models. In The Eleventh International Conference on Learning Representations, 2023. URL ntttps:/oopent http///wpp.eeeee//////pppp wwtpvviewwne/eeeeoo /are/ffoommt id= Ad=jlAjjee JNL825cs.
Wang, Y., Guo, D., Li, S., Camps, O., and Fu, Y. Explainable anomaly detection in images and videos: A survey, 2024b. URL https: / a arxiv. orgg abbs/ / 22022.222020 06670.
Mitchell, E., Lin, C., Bosselut, A., Finn, C., and Man- ning, C. D. Fast model editing at scale. arXiv preprint arXhv:2110.11309 2021.
Yu, L.., Chen, Q., Zhou, J., and He, L. Melo: Enhancing model editing with neuron-indexed dynamic lora. arXiv preprint 23 /795, 2023a.
Mitchell, E., Lin, C., Bosselut, A., Finn, C., and Manning, D. Fast model editing at scale. In International Con- ference on Learning Representations, 2022a.
Yu, Y., Yang, C.-H. H., Kolehmainen, J., Shivakumar, P. G., Gu, Y., Ren, S. R. R., Luo, Q., Gourav, A., Chen, I.-F., Liu, Y.-C., et al. Low-rank adaptation of large language model rescoring for srrameter-efficiee speech recogni- tion. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-8. IEEF, 2023b.
Mitchell, E., Lin, C., Bosselut, A., Manning, C. D., and Finn, C. Memory-based -basd model editing at scale. International Conference on Machine Learning. PMLR, 2022b.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mi- haylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, S., Sridhar, A., Wang, T., and Zettlemoyer, L. Opt: Open pre-trained transformer language models, 2022.
Rawte, V., Sheth, A., and Das, A. A survey of hallu- cination in large foundation models. arXiv preprint arXiv:2309.05922 2023.

---

BalancEdit: Dynamically Balancing the Generality-Localit Trade-off in Multi-modal Model Editing
Zheng, C., Li, L.., Dong, Q. Fan, Y., Wu, Z., Xu, J., and
Chang, B. Can we edit factual knowledge by in-context
learning? arXiv preprint 2305::2300.127400 2023.
Zhong, Q. Ding, L.., Liu, J., Du, B., and Tao, D. Panda: Prompt transfer meets knowledge distillation for effi- cient model adaptation. arXiv preprint arXiv:2208. 10160, 2022.
Zhou, G., Hong, Y., and Wu, Q. Navgpt: Explicit reasoning in uisoo-aadd-lanuage navigation with large language models. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume 38, pp. 7641-7649, 2024.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv: 10592, 2023.
Zhu, R., Guo, D., Qi D., Chu, Z., Yu, X., and Li, S. A survey of trustworthy representation learning across domains. ACM Trans. Knowl. Discov. Data, 18(7), June 2024. ISSN 1556-4681. doi: 10.1145/3657301. URL https:// doi 2e5.org/100.145/36556
12

---

BalancEdit:
Dynamically Balancing the Generality-Localiiy Trade-off in Multi-modal Model Editing
A. Dataset
Edit Semale
Locility Soer
Crieart that sinder houne?
Cr un buciiiiies spiciall farrs
Vihie thart st Tounci
Although numerous studies have been conducted on knowl- edge editing in Large Language Models (LLMs), re- search in the context of Large Vision-Language Models (LVLMs) remains relatively sparse. Only one benchmark, MMEDIT (Cheng et al., 2023), has delved into this do- main within LVLMs. This benchmark extend the concepts of Reliability, Generality, and Locality from LLM editing, incorporating diffusion- ceeesion-model-gge mooee--generated images in its Gen- erality evaluation.
dlle spasing wond??
mace Poom??
C Arut to hedddeoouaddo Coeee The logiinn
O What for octanton toptrally
A Ocaater temerr Pronch builing
Figure 6. Examples of OKEDIT dataset. "CF' represents the edited counterfactual knowledge. The red color indicates the out- dated answer and the green color indicates the updated correct answer.
However, this dataset has its limitations as shown in ta- ble 2. The content of images generated from image caption prompts can deviate from the original images, leading to inconsistencies and potentially less accurate evaluations. Furthermore, the scarcity of data in the only existing bench- mark presents a significant harm the progress in LVLM knowledge editing. Therefore, the availability of more data would greatly aid in the development and refinement of techniques in this field.
the image should be
After we obtain the image object, we can ask the diffusion model to generate with the image object. For each image, we also generate 10 images for evaluation.
For the locality evaluation, we try to generate an image that is similar enough the original image but it still unrelated to it. To achieve that, we have three steps generation. Fisrt, we will determine the locality answer with high similarity with the target answer, with the help of GPT-4.
In our research, we utilize the multimodal VQA dataset VKVQA (Marino et al., 2019), which provides hard image questions with difficult visual reasoning and open knowl- edge. Furthermore, the OKVQA dataset provide detailed question categories which could be used to evaluate the editing method on different question types.
"Given (question: {question}, A: Manswer)). B: cogumterfact.nnswele
what could be another op- tion? Short answer. C: []"
A.1. Dataset Construction Details
Then, we can follow the same steps as in the image rephras- ing process to generate locality images, including obtaining image objects and generating images with diffusion model.
OKEDIT dataset are constructed to provide pairs of edit input (i, and counterfact answer y". The edit labelis not necessarily the 'correct' label; the goal is to provide realistic instances of the types of data we would expect to see during test. For example, given the i as a HP brand computer, and f = What is the brand of it, and yf is the lenovo, even though it never happens currently. However, this fictitious example still a useful assessment of our model's ability to perform the general type of edit of "change name of an item'.
B. Dataset Samples
We present several examples from our OKEDIT dataset in Figure 6. Our dataset offers high-quality images and samples of counterfactual knowledge editing. Additionally, some samples incorporate common sense knowledge, which adds complexity to the editing tasks. These characteristics enhance the overall quality of our dataset in comparison to the existing MMEDIT dataset.
To evaluate the text generality, we generate some sam- ples using the rephrasing methods. Specifically, we use the GPT-4 API to generate the rephrased questions, with the following command.
C. Metrics
"Please rephrase the following question in {num_versions} different ways: {question)." where we generate 10 rephrased questions.
(1) Reliability. The updated model should output the target answers: (i, = yy", (i, i, ) E Deait (2) Generality, The updated model should answer the target output given re- lated inputs: fne, ,t') = y², (i', E') Ri (3) Locality, The updated model should keep the output retained on the unrelated inputs. fnew (i', (')= fhaseli', (i', ('), (i', (') E U..- Additionally, there are two bonus properties. (4) Multiple Edits. The model could edit multiple times without for- getting previous edits. (5) Efficiency. The model editing method should take minimal costs to edit a model, such as
For the image generality, we need to generate semantic similar images. To get the semantic meaning of a specific image in the question context, we first question the GPT-4 which objects and scene should be in the image.
"Given (question: {question). answer: {answer)), what ob- ject should be in the image? ? Short answer. The objects in
13

---

BalancEdit: Dynamically Balancing the Generality-Localit Trade-off in Multi-modal Model Editing
less training time and data costs.
where
Reliability The updated model should output the target answers correctly.
E == a- ((())))) + (1-a) ((F()) (10)
Mileiability
E (1.2.y")
1[now - 3
(4)
E. Baselines
Finetune In this method, we carry out a fine-tuning pro- cess on selected layer of the pretrained model using Adam optimization for fair comparison, while keeping all other layers fixed. For the training loss, the Cross Entropy loss is used for fine-tuning.
Text Generality The updated model should answer the correct answer given the related rephrased question.
Gen
E (e
(Snew R(()) = )²} (5)
Image Generality Similarily, the updated model should answer the correct answer given the similar images.
IKE (Zheng et al., 2023) IKE (In-Context Knowledge Editing introduces a system that utilises unsupervised retriever. This retriever uses cosine similarity to pinpoint pertinent demonstrations from the training set. This method is grounded in the principles set forth by (Liu et al., 2022) and aims to insert new factual knowledge into language models in a disruptue fashion, eliminating the need for direct parameter updates. IKE's approach ranks demonstra- tions according to their resemblance to the editing target and organizes them in sequence to form supplementary knowl- edge base that steers the model's generation process. This technique not only conserves the model's existing knowl- edge base but also presents a scalable and efficient method to refresh factual information. shows considerable promise in mitigating unintended side effects, such as over-editing and knowledge forgetting, typically linked with gradient- based editing methods. However, it is also designed for pure text models for retrievel, to make it adapt to vision language models, we used composed embedding as the augmented database, such that it can retrieve the image information as well.
MII.Gen
E
Unnew (R(), = )²}
(6)
(v,t,)/ JE Deat
Locality The updated model should not change the irrele- vant knowledge that is stored in the original model.
MiA
Innow (V',)) f)= Sbase (i', (1',)}} (7)
(M)
D. Theoratical Analyse
Here is brief theoretical proof about the effectiveness of our radius.
Lemma: Embeddings of semantically similar concepts are close in the embedding space.
Proof. 1. Definition of Embeddings: Embeddings vector representations of concepts in high-dimensional space. Formally, let f C + Idd be an embedding function that maps a concept e E C to a vector ((c E Rd.
Semantic Similarity: Semantic similarity between two concepts 01 and C2 can be quantified using a similarity mea- sure S((1, c2). Common choices include cosine similarity, Euclidean distance, dot product.
MEND (Mitchell et al., 2022a) MEND employs a hy- pernetwork to predict new weights for a selected layer of a pre-trained model by estimating the low-rank decomposi- tion of the weight matrix of the layer. The hypernetwork is trained on set of training edits, which comprises a new edit, set of inputs that are semantically equivalent to the edit, and samples from the model's 's pre-training data. However, MEND designed for the language model, to fit it to the vision language model, we keep the vision encoder fixed and only choose the language model layer for finetuning. In addition, in situation, we only have single edits that streaming in, train the hypernetwork to predict updated weights as edits stream in using continuous fine-tuning.
3. Objective of Embedding Training: During the training of embeddings, the objective typically to maximize the similarity of embeddings for semantically similar concepts and minimize it for dissimilar ones.
((())),(()) < S((())),(())) if ((c < S(c2,
(8)
Assumption: The generality sample (G) is semantically more similar to the editing knowledge (E) than the local- ity sample (L). That is. S(G,E) S(L, E). Accord- ing Lemma 1, we state that ((f()),(EE) < f(E)..
GRACE (Meng et al., 2022a) GRACE is lifelong model editing method for large language models. It handles se- quential edits with a discrete key-value codebook. GRACE replace one layer to GRACE adaptor which stores the key-value pair of the target edits, where the key is the last embedding of the key for the text prompt and value is trained
Conclusion: In this case, we can find a radius e such that
S(((),,(E)) <<< ((E)).
14

---

BalancEdit: Dynamically Balancing the Generality-Localiy Trade-off in Multi-modal Model Editing
Figure 7. T-sne figure of key distribution in sequential editing.
by backpropagation with the target results. Keep handling the key conflicts could make it successfully deal with the multiple sequential editing in language models. However, to adapt it to the vision language model, we select the language part as the edited layer and prepend the image embedding before the text prompt that it can be regarded as long text questions.
F. Training Specifications
We use the Adam optimizer (Diederik, 2014) for all methods. Given that edits in our setup are single and sequential, the batch size consistently We trained all methods using variety of GPUs, including 24GB NVIDIA RTX A5000s, 40GB NVIDIA A A000s, and 80GB NVIDIA A 1000s. Timing experiments are reported from experiments performed on an NVIDIA RTX A 100 GPU. The scale of BalancEdit is not dependent on the model's scale, but the model's 's scale is de- pendent on the available computational resources. To avoid sharding, we utilize models that can be accommodated on appli-
single GPU, although the principles of BalancEdit cable beyond this setup. For Adaptor-bar based editors, such as GRACE, we employ 100 iterations of gradient descent per input.
G. Key Distribution
To verify the key distribution in sequential editing, present the distributions of keys in the codebook. From the Figure 7, we observe that the keys are scattered, indicat- ing that the codebook is capable of handling multiple edits well.
15


Reviews:
[{"review_id": "viLwJuDGwf", "reviewer": "Reviewer_55Ec", "summary": "The paper introduces a method for efficiently updating multi-modal LLMs. Existing model editing struggles with balancing generality  and locality. BalancEdit addresses this by using a codebook mechanism that stores discrete edits, dynamically adjusting their influence scope with positive and negative samples. The authors introduce OKEDIT, a dataset designed to evaluate this trade-off. Experiments on MiniGPT-4 and BLIP-2 OPT show that BalancEdit surpasses baseline methods while maintaining efficiency and interpretability.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. Here we're glad for the chance to clarify some points.\n>Quality of the generality and locality samples?\n\nA: We appreciate the reviewer’s thoughtful concern. To ensure the semantic quality of the OKEDIT dataset and the meaningful separation between generality and locality, we adopt a carefully designed generation pipeline. The detailed generation process has been provided in Appendix A.1. Here we provide a concrete example.\n\nThe editing counterfactual knowledge is ***the \"HP\" looks computer is named as lenovo.*** Specifically, the original image is a HP brand computer, original question is ***'What is the brand of it'***, and the original result is ***'HP'***, and the new result is ***'lenovo'***.\n\n- **Image Generality sample generation:**\n1. **Generality object confirmation:** We ask GPT4 about objects and scene that should be in the image. The prompt is shown in Appendix A.1. The GPT would answer ***\"A HP laptop\"***.\n2. **Generality sample generation:** Based on the object, we will prompt the diffusion model to generate the images. Since we provide the exact object, the model can generate correct images.\n- **Image locality sample generation:**\n    1. **Locality answer generation**: We first prompt GPT4 to generate a distractor. Like *\"Given (question: What is the brand of it, A: HP, B: Lenovo), what could be another option? Short answer. C: []\"* The GPT would generate a distractor, such as ***\"Dell\"***\n    2. **Locality object confirmation:** Similarly, we ask GPT4 about objects and scenes that should be in the image. The GPT would answer ***\"A Dell laptop\"***.\n    3. **Locality sample generation:** Based on the locality object, we will ask diffusion model to generate the correct images.\n\nIn this way, having clear instructions for creating an image helps ensure its quality.\n>What is transformation(v)? What is L and a specific key k.\n\nA: We apologize for the confusion about Eq. (2). In the following. We clarify each component of Eq. (2) and we will update the final version of our paper accordingly:\n\n**Transformation $v$**：The transformation refers to a **specific layer in the LLM**. During an edit, we **fine-tune this transformation layer** to encode the new knowledge. Thus, $v$ is a **updated version of an existing LLM layer**, reused only for inputs within the influence radius of a particular edit.\n\n**Loss $L$**: $L$ is the **language loss**, specifically the **next-token prediction loss** used by the base LLM. This loss ensures that the updated model $f_{\\text{new}}$, when processing input$(i, t)$, generates the desired new answer $y_n$.\n\n**Key $k$**: As shown in Method 3.2, each **$k$** in the codebook is a **representative embedding** for a specific edit.\nFor example, suppose we edit the fact: *\"What brand is this computer?\" from \"HP\" to \"Lenovo\".* Then the key k is the averaged embedding from layer $l-1$ when the input is the ***HP laptop image + the original question***.\n>“By caching embeddings for input errors and the updated knowledge transformation layer that decodes into the desired model outputs”.. Could you explain it in detail?\n\nA: Thank the reviewer for pointing this out. We agree that this sentence may have been too condensed, and we are happy to explain it in detail. Similar to the Section 3.2, We mean:\n1. **Caching embeddings for input edits (keys $k$):**  \n   For each edit (i.e., input $(i, t)$ and new answer $y_n$), we extract the **averaged embedding** from the layer $l-1$ of the model. This embedding — the **key $k$** — represents the semantic location of the edit in the model’s latent space. It serves as a **reference point** to decide whether future inputs are close enough to activate the edit.\n2. **Caching the transformation $v$:**  \n   We fine-tune and cache a **copy of the LLM’s layer $l$** so that, when applied to this type of input, it produces the new target output $y_n$. \n   \n>How does α work?\n\nA: Thanks for mentioning the key point. α adjusts the **influence radius** by **weighting** the distances between the key and the positive/negative samples. It also controls the trade-off between generality and locality.\n>Statistics of categories\n\nA: Thank the reviewer for raising this point. Below, we provide detailed statistics of the knowledge categories in the OKEDIT dataset. As shown in the table, OKEDIT spans a **diverse and representative** set of domains, supporting its utility as a comprehensive and generalized benchmark for multi-modal model editing.\n\n|KnowledgeCategory|Percentage|\n|-|-|\n|Plants, Animals|17%|\n|Vehicles, Transportation|16%|\n|Cooking, Food|15%|\n|Sports, Recreation|12%|\n|People, EverydayLife|9%|\n|Objects,Material, Clothing|8%|\n|Brands,Companies, Products|3%|\n|GeographyHistory,Language, Culture|3%|\n|Weather, Climate|3%|\n|Science, Technology|2%|\n|Other|12%|\n\n>Reproduction and typos\n\nA: We thank the reviewer's attention. We will fix the typos based on the suggestions and provide the code in the final version.", "claims_and_evidence": "The claims in the submission are partially supported by empirical evidence, including quantitative evaluations, baseline comparisons, and ablation studies. The introduction of the OKEDIT dataset provides a structured way to assess the generality-locality trade-off, and experimental results on MiniGPT-4 and BLIP-2 OPT indicate that BalancEdit performs effectively in terms of accuracy, efficiency, and interpretability. Efficiency claims are supported by editing time comparisons.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for multi-modal model editing, with BalancEdit addressing the generality-locality trade-off and OKEDIT providing a structured benchmark. The chosen metrics—editing success, generality, and locality—effectively assess model edits, and comparisons with baselines support the evaluation.", "theoretical_claims": "I have checked the equation in the paper. There is no proof in the paper.", "experimental_designs_or_analyses": "The experimental design includes baseline comparisons, metrics (harmonic mean of generality and locality), and ablation studies to validate BalancEdit. The OKEDIT dataset serves as a relevant benchmark, while efficiency claims are supported by editing success and time analyses.", "relation_to_broader_scientific_literature": "LLM editing is a promising research direction for cost‑effective revision of large language models. It can further remove harmful or biased content that might persist despite standard safety alignment mechanisms.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strength\t\n1. This paper proposed BalancEdit which dynamically optimizes the generality-locality trade-off without modifying model weights.\n2.The proposed method outperforms baseline methods in accuracy, efficiency, and sequential editing while requiring minimal computational cost.\n3.This paper discuss the radius of the influence scope, which is useful for editing-related research.\n\nWeaknesses\n1. The organization of the paper, especially the organization of the method section, is not clear enough, make it hard to follow. (Detailed examples are in Question section).\n2.    It could be questionable how to ensure the quality of the generated images for generality and locality respectively, since sometimes there may not be a clear boundary between them. The test dataset is also built on generated data, which increases my concern.\n3.    The reproducibility of the method is another concern.", "other_comments_or_suggestions": "Please enlarge the font size in Figure 3 to improve readability.", "questions_for_authors": "It is still difficult for me to understand what transformations (v) is.\nIt is hard to understand the eq. (2). And what is L in this equation?\nWhat is a specific key k?\n“By caching embeddings for input errors and the updated knowledge transformation layer that decodes into the desired model outputs”.. Could you explain it in detail?\nWhy hyperparameter /alpha can “adjust the distance” in eq. (3).  if I understand correctly, /alpha is used to adjust the weight of different distances instead of “adjusting the distance”.\nHow did you ensure the quality of the generated images for generality and locality respectively?\nCould you provide the statistics of each categories (vehicles, people, etc.), the number of images under generality, locality?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jyXr74gNOh", "reviewer": "Reviewer_ZhWy", "summary": "Existing multi-modal model editing methods struggle to dynamically adjust the influence scope of an edit, balancing generality and locality.\nTo address the issue, this paper proposes a novel model editing method, i.e., BalancEdit, with process as follows\n\n* For each image-text pair to be edited, the averaged embedding and the associated transformation and radius are cached.\nThe transformation is learnt by standard fine-tuning and the radius is an interpolation of distances with positive (rephrased text, same image) and negative (same text, black image) samples.\n* For new image-text pair, an edited transformation is activated if it falls within the scope of an edited sample; otherwise, the unedited transformation is applied.\n\nThe proposed method is empirically compared with FT, IKE, MEND, and GRACE on MMEDIT and OKEDIT datasets in both single and sequential editing settings, where BalancEdit enjoys superior accuracy, text generality, harmonic mean, and efficiency.\nAblation study is conducted to illustrate the effects of hyper-parameters, distance function, and anchors.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful feedback, we are glad for the opportunity to clarify some points.\n>Why is our method parameter-efficient?\n\nA: Thanks for the valuable comment! Knowledge editing is a challenging task, and we acknowledge that memory grows linearly with the number of edits due to cached transformations. However, compared to baselines like full fine-tuning or meta-learning (e.g., MEND), BalancEdit is still parameter-efficient per edit, as it **only** modifies a single layer and avoids retraining the full model.\nMoreover, our transformation can be replaced by PEFT method (e.g., LoRA), which would further reduce memory usage. We chose full fine-tuning for **universality** and for **clearer and fair comparisons**.\n\n>Why use averaged embedding as the key?\n\nA: Thank the reviewer for the thoughtful question. We would like to clarify that using the **averaged embedding as a sentence-level representation** is a common and effective strategy(e.g., SimCSE [1], SBERT [2]), and we adopt a similar approach to cache edited knowledge in BalancEdit.\n\nWe chose this method over influential-token-based strategies (e.g., ROME [3]) for two main reasons:\n\n1. **Modality Gap:** Prior work like ROME is designed for *text-only* models (e.g., GPT), while we focus on **multi-modal** models, where token-level attributions are harder to isolate due to vision-language fusion.\n\n2. **Simplicity and Generalization:** Unlike token-attribution methods, our approach does not require edit-specific or model-specific token selection, making it easier to generalize across diverse edits and architectures.\n\nWe find that average embedding is robust, simple, and effective for our editing framework.\n\n[1] Gao et al., SimCSE, EMNLP 2021  \n[2] Reimers & Gurevych, Sentence-BERT, EMNLP 2019  \n[3] Meng et al., Locating and Editing Factual Associations in GPT, NeurIPS 2022\n\n>Why does your Negative sample selection better than randomly chosen negative sample?\n\nA: Thank the reviewer for the insightful question. We conducted an empirical comparison with a **random negative sampling baseline**, where a randomly chosen text-image pair (assumed irrelevant to the edit) is used as the negative anchor.\n\n| Method                | Edit Acc | T-Generality | I-Generality | Locality | HM    | Model    |\n|-----------------------|----------|--------------|--------------|----------|--------|-----------|\n| **BalancEdit**        | 100.00   | 98.89        | 65.38        | 61.18    | **71.85** | BLIP-2    |\n| Random Negative Sample| 100.00   | 100.00       | 49.12        | 65.08    | 65.61  | BLIP-2    |\n| **BalancEdit**        | 100.00   | 99.87        | 76.46        | 53.14    | **71.58** | MiniGPT-4 |\n| Random Negative Sample| 100.00   | 99.00        | 66.93        | 45.92    | 64.08  | MiniGPT-4 |\n\n**BalancEdit consistently outperforms the random baseline** in harmonic mean (e.g. 71.85 vs. 65.61), showing a better balance between generality and locality. This supports the effectiveness of our **black image-based negative anchor**, which provides a **fact-agnostic, consistent, and efficient way to define a lower bound** in representation space.\n\nIn contrast, random negative samples:\n\n- Depend on external unrelated examples,\n- Are unstable in quality and relevance,\n- And require assumptions that may not hold across diverse domains.\n\nOur method avoids these issues, making it more **robust and scalable** for real-world editing scenarios. We will include this analysis and clarify the design rationale in the final version.\n\n>Typos:\n\nA: Thanks for the detailed review. We will revise them accordingly.", "claims_and_evidence": "The author claims that BalancEdit is parameter-efficient on line 108, while I understand that it requires to cache a transformation for each edit, making it suffers linear memory consumption with respect to the number of edits.", "methods_and_evaluation_criteria": "I understand the intuition behind the algorithm while I am curious about whether the design is sufficiently justified.\n\n* The averaged embedding is chosen as the cached key.\nI wonder whether there is any reference or experiment to support the choice.\nWill it be better than the embedding of certain influential tokens discovered by prior work [1]?\n* The same text with black image is selected as the negative sample.\nIs there any empirical study to demonstrate its superiority to a straight-forward baseline, i.e., randomly choosing irrelevant text-pair samples?\n\n[1] Locating and Editing Factual Associations in GPT, NeurIPS 2022.", "theoretical_claims": "This work is mostly empirical.", "experimental_designs_or_analyses": "See Methods and Evaluation Criteria", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "* The idea of the work makes sense and is interesting to me.", "other_comments_or_suggestions": "* No period on line 87.\n* The Metric paragraph in Sec. 4.1 seems should be separated.", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rchMMR9iQk", "reviewer": "Reviewer_TTyf", "summary": "Large-scale multimodal models suffer from knowledge decay as facts change, and traditional fine-tuning is often impractical due to their size. Instead, direct knowledge editing is preferred, but current methods neglect differences in fact influence, creating a trade-off between generality and locality. To address this, the authors introduce the generality-locality trade-off, propose the OKEDIT dataset for evaluation, and present BalancEdit—a method that dynamically balances these aspects by generating positive and negative samples and editing the model’s latent space via a discrete, localized codebook without altering the underlying weights.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing valuable feedback to improve our paper. We have addressed your hesitations below.\n> Please highlight the contributions compared to existing works.\n\nA: Thank the reviewer for the valuable suggestions. We agree that clearer differentiation is necessary. Below, we restate our key contributions while explicitly contrasting them with existing multimodal knowledge editing works:\n\n1. **We formulate the generality-locality trade-off in multi-modal model editing.**  \n   In contrast, prior works such as MMEDIT [1] and GRACE [2] focus either on general-purpose editing or lifelong editing, but **do not explicitly define or evaluate** the trade-off between generality and locality in the multi-modal setting.\n\n2. **We introduce OKEDIT, a new benchmark dataset designed to evaluate both generality and locality.**  \n    In contrast, MMEDIT [1] uses random choose pairs as locality samples which may result in loose alignment and limited coverage.\n\n3. **We propose BalancEdit, an efficient editing framework that dynamically adjusts the influence scope of edits.**  \n   In contrast, IKE [3] assumes a wide influence range due to retrieval-based prompting, while GRACE [2] uses fixed-radius memory lookups. We introduce a radius-based mechanism using positive and negative samples to support **dynamic control over edit scoping**.\n\n4. **We design a parameter and data efficient mechanism with little training overhead per edit.**  \n   In contrast, MEND [4] and similar meta-learning approaches require pretraining on large edit datasets, which are hard to obtain in the multi-modal domain. BalancEdit **avoids pretraining**, supports multi-edit scenarios.\n\nWe will revise the Introduction and Related Work sections to make these distinctions clearer in the final version.\n \n[1] Cheng et al., Can We Edit Multimodal Large Language Models?, arXiv 2023  \n[2] Hartvigsen et al., Aging with GRACE, NeurIPS 2024  \n[3] Zheng et al., Can We Edit Factual Knowledge by In-Context Learning?, arXiv 2023  \n[4] Mitchell et al., MEND: Model Editing with Noisy Demonstrations, NeurIPS 2021\n\n> How is the quality and comprehensiveness of the OKEDIT dataset?\n\nA: Thank you for the important observation. While OKEDIT is our main evaluation benchmark, we have taken several steps to ensure the **robustness, diversity, and generalizability** of both the dataset and our method:\n\n1. **Cross-dataset Validation:**  \n   We evaluated BalancEdit not only on OKEDIT but also on the **MMEDIT dataset**, and observed consistently strong performance (see Table 3), which supports the **generalizability** of our method across datasets with different construction paradigms.\n\n2. **Grounding in a Verified Source Dataset:**  \n   OKEDIT is built upon the **OKVQA dataset**, which has been widely used and shown to exhibit **limited bias**. This provides a reliable and diverse base of real-world image-text pairs for constructing edits.\n\n3. **Diverse Category Coverage:**  \n   Our dataset spans **11 broad knowledge categories** (e.g., animals, transportation, brands, people, holidays, etc.), ensuring that our benchmark reflects both **common and long-tail factual knowledge**. This promotes **balanced evaluation** across a wide range of input types.\n\n4. **Harder and More Realistic Locality Samples:**  \n   Compared to MMEDIT, OKEDIT constructs **semantically similar locality distractors**, making the evaluation more challenging and reflective of **real-world edit interference scenarios**.\n\n5. **Human Verification and Quality Control:**  \n   We conducted **human validation** on a subset of the dataset to ensure correctness of labels and alignment with the intended edit scopes.\n\nTogether, these points demonstrate that OKEDIT is a **carefully designed and reliable benchmark**, and that BalancEdit is not overly dependent on a specific dataset structure. We will revise the manuscript to better highlight these points.\n\n> How is the generalization of the method on newer model?\n\nThank the reviewer for the constructive suggestion. To evaluate the generalization of our method on more recent vision-language backbones, we conducted additional experiments using **Qwen-VL** on the MMEDIT dataset. The results are shown below:\n|Method|Acc|T-Gen|I-Gen|Loc|HM|\n|-|-|-|-|-|-|\n|Base|18.18|17.13|14.14|NA|NA|\n|GRACE|99.88|30.36|32.18|87.78|39.78|\n|BalancEdit|100.00|70.96|71.51|41.96|**57.79**|\n\nOur method achieves a **higher harmonic mean (HM)** compared to GRACE, indicating a **more balanced trade-off between generality and locality**. These results also validate the **generalizability** of BalancEdit to newer backbone models beyond those originally reported in the paper. We will add more backbones in the future work.", "claims_and_evidence": "The claims are supported by the experiments.", "methods_and_evaluation_criteria": "The proposed method make sense for the problem.", "theoretical_claims": "This paper do not contain theoretical claims.", "experimental_designs_or_analyses": "The experimental design is valid.", "relation_to_broader_scientific_literature": "There is a lot of related work; it is recommended that this paper clearly highlights the differences and contributions compared to existing multimodal knowledge editing works.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\nIntroduces the generality-locality trade-off, a unique perspective in multimodal model editing that explicitly addresses the balance between global consistency and localized accuracy.\n\nThe creation of the OKEDIT dataset provides a targeted benchmark for evaluating this trade-off, which can facilitate further research in the area.\n\nBalancEdit’s mechanism of generating both positive and negative samples to determine the influence scope is a sophisticated approach that enhances the precision of edits.\n\nWeaknesses:\n\nThe success of the method hinges on the quality and comprehensiveness of the OKEDIT dataset; any limitations or biases in the dataset could  affect the evaluation and generalization of the results.", "other_comments_or_suggestions": "See weaknesses.", "questions_for_authors": "The multimodal large models used in the experiments are somewhat outdated. It is recommended to conduct experiments on the latest multimodal large models, such as llava-onevision, Qwen2-VL, and others.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8wBmWjJUzG", "reviewer": "Reviewer_D7mY", "summary": "The paper introduces BalancEdit, a method designed to address the challenge of balancing generality and locality in multi-modal model editing. Existing methods often fail to dynamically adjust the influence scope of edits, leading to over-correction or under-correction. The authors introduce the generality-locality trade-off in multi-modal editing and create OKEDIT, a dataset to evaluate this balance. Then, the proposed solution, BalancEdit, integrates an adapter into a vision-language model layer without altering the original weights. This adapter functions as a codebook that stores edits by caching input-error embeddings and updated transformation layers. Experiments show it outperforms baselines (e.g., IKE, MEND, GRACE) across metrics, achieving state-of-the-art results in targeted updates.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the valuable feedback! We would like to clarify some points below.\n>Why our locality sample is harder?\n\nA: Thanks for the question. Our locality sample is harder due to the **sematical similarity** to the editing knowledge. For example:\n\n|Aspect|MMEDIT|Our (OKEDIT)|\n|-|-|-|\n|**Edit Input**|Q: How many tennis balls are in the picture? A: 0 → 2| Q: What brand is this computer? A: HP → Lenovo|\n|**Locality Sample**|Q: What sport can you use this for? Image: bicycle   A: riding|Q: What brand is this computer? Image: Dell laptop   A: Dell|\n|**Relation to Edit**|Semantically **unrelated**|Semantically **similar**|\n|**Difficulty**|Low：easy to distinguish|High: requires finegrained reasoning|\n\nWe will add more qualitative comparisons in the paper.\n>How is the validity of the dataset?\n\nA: Thanks for raising this important point. We provide the **Category-level Statistics** of OKEDIT in our response to Reviewer 55Ec, showing its generality. We also performed **human verification** on a subset of samples to ensure the correctness and the alignment with the intended edit scope. This helps establish the reliability of OKEDIT.\n>How is α selected?\n\nA: Thanks for the valuable concern. We select α using a small **held-out set** of only 5 unrelated samples. Since different models may have different latent feature distributions, α is treated as **model-dependent**. Once chosen, the same α is fixed per model (e.g., α=0.2 for MiniGPT-4) for all evaluations.\n>Why is our negative sample better than generating counterfactual negative sample.\n\nA: Thanks for the thoughtful question. We want to clarify that using counterfactual samples presents several limitations:\n- **No Extra Knowledge Assumption:**  \nWe aim to **avoid requiring additional knowledge** beyond the editing input for our goal of being **realistic and efficient**, but counterfactual sample require external knowledge.\n- **Intractability of Counterfactuals:**  \nGenerating high-quality counterfactuals is costly, while ours use an **efficient, universal, and reusable** negative anchor.\n\nTo further justify the effectiveness of our negative anchors, we conducted an empirical comparison with a **random negative sample baseline** (in the second last response to reviewer ZhWy). We still achieve better results.\n>How is the editing layer chosen?\n\nA: Thanks for raising this concern. In our method, we select the editing layer based on prior work(MEND, MMEDIT).\n\nTo validate the robustness of our method, we conducte an ablation study on another editing layer below:\n\n|Method|Acc|T-Gen|I-Gen|Loc|HM|Model|\n|-|-|-|-|-|-|-|\n|BalancEdit|100|99.87|76.46|53.14|71.58|MiniGPT4|\n|diff layer|100|100|64.03|66.39|73.75||\n|BalancEdit|100|98.89|65.38|61.18|71.85|BLIP2|\n|diff layer|100|100|78.43|44.99|66.70||\n\nEven if a different layer is chosen, **we can still achieve good performance**, showing the robustness of our method.\n> Why is your positive sample better than retrieving associated positive images?\n\nA: Thanks for the suggestion. We would like to clarify that retrieval would introduce **additional overhead**, including retrieval pipelines and sample bank. Retrieval also assumes that **visually similar positive samples exist**, which may not hold for long-tail or rare edits.\n\nIn contrast, our method uses **textually rephrased questions**, enabling **efficient and consistent construction** of positive samples, keeping our framework lightweight and generalizable across edits.\n>The loss function in Eq. 2 is not clear.\n\nA: Thanks for pointing this out. $L$ is the **language loss**, specifically the **next-token prediction loss** used by the base LLM. This loss ensures that the updated model $f_{\\text{new}}$, when processing input$(i, t)$, generates the desired new answer $y_n$.\n>What does the time in table 5 mean?\n\nA: Thanks for the question. In Table 5, **“training” refers to the offline pretraining phase**, such as training a metanet (e.g., MEND). However, We don't need that.\n\nThe **time spent fine-tuning in our method is counted under \"editing time\"**, as it is performed **per edit**. The fast edit time shows that our method is **efficient**.\n>What is the extreme situation for α?\n\nA: Thanks for the suggestion. We conduct ablation experiments using extreme values of α on Minigpt4 with OKEDIT dataset:\n\n|Method|Acc|T-Gen|I-Gen|Loc|HM|\n|-|-|-|-|-|-|\n|BalancEdit|100|99.87|76.46|53.14|71.58|\n|α=0|100|100|95.19|16.30|36.65|\n|α=1|100|45.94|24.22|100|41.06|\n\n- **When α = 0**, the influence radius is entirely determined by the **negative sample**, leading to **over-generalization and lower locality** (e.g., 16.30 Loc), as the edit is applied too broadly.\n- **When α = 1**, the radius is determined by the **positive sample** only, leading to **over-localization and poor generalization** (e.g., 24.22 I-Gen), since the edit applies to narrow a region.\n\nThese results confirm that α controls the trade-off between generality and locality as intended, which validates our assumption.", "claims_and_evidence": "Strengths:\n* BalancEdit balances generality and locality better than baselines.\n* Efficiency of BalancEdit: Comparison of computational costs and data requirements strengthens this claim.\n* Support for sequential edits.\n\nWeaknesses:\n* **Superiority of OKEDIT dataset**: While OKEDIT is described as more comprehensive than MMEDIT (table 2), details about its construction (e.g., GPT-4-generated rephrasings, diffusion-model-generated images) raise questions about potential biases. No qualitative examples of \"harder\" locality samples are provided compared with MMEDIT. In addition, the lack of dataset analysis for human-annotated benchmarks weakens its validity as a gold label.\n* **Dynamic influence radius mechanism**: The radius calculation (Eq. 3) depends on hyperparameter $\\alpha$ and distances between positive/negative samples. While ablation studies show α’s impact, the paper does not explain how α is chosen or whether it generalizes across tasks.\n* **Negative sample selection**: Although the author compares the Negative Anchor of white and black, I don't understand why a pure colour is used as a negative sample, rather than a counterfactual example that is then labeled very differently.", "methods_and_evaluation_criteria": "* **Layer Selection**: The choice of layer (e.g., specific transformer blocks) is critical but not rigorously justified. Performance might degrade if suboptimal layers are selected.\n\n* **Simplistic Positive Samples**: In addition to the negative samples mentioned above, the positive samples are simply perturbed by simple textual. Is it possible to retrieve associated images and use them as positive samples?\n\n* The loss function in Eq. 2 is not clear.", "theoretical_claims": "There seems to be no issue with the theoretical part.", "experimental_designs_or_analyses": "* Why doesn't time spent fine-tuning count as time spent doing training in tabel 5?\n* I am more interested in knowing the extremes of hyperparameter $\\alpha$ in ablation experiments, e.g. 0 or 1.", "relation_to_broader_scientific_literature": "The paper’s key contributions are closely tied to and advance several strands of prior work in model editing, multi-modal learning, and parameter-efficient adaptation.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Please see the methods and evaluation criteria.", "other_comments_or_suggestions": "Please see above.", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: After the rebuttal and discussion phases, the paper received scores of 4, 4, 2, and 2, with an average rating of borderline accept. Most reviewers acknowledged the contributions of the paper, including its effective balance between global consistency and localized accuracy in model editing, the useful methodology and benchmark it provides for editing-related research, and its cost-effectiveness.

Two reviewers (Reviewer 55Ec and Reviewer ZhWy) did not provide sufficient responses during the rebuttal and discussion phases. After carefully reviewing the reviewers' comments, the authors' responses, and conducting my own brief examination of the paper, I believe that the major concerns raised by Reviewer 55Ec and Reviewer ZhWy have been adequately addressed in the rebuttal and discussion stages.

Since the paper introduces the generality-locality trade-off in multi-modal model editing for the first time, I believe it offers valuable insights to the model editing community. Therefore, I recommend acceptance of the paper.
Decision: Accept (poster)