DualEdit: Dual Editing for Knowledge Updating in Vision-Language Models
Authors: Zhiyi Shi
Binjie Wang
Chongjie Si
Yichen Wu
Junsik Kim
Hanspeter Pfister
Venue: COLM 2025 (2025)
Keywords: Model Editing
Multimodal Learning
VLM
TL;DR: We explore the importance of image and text modalities and propose a novel dual editing method—DualEdit.

Abstract: Model editing aims to efficiently update a pre-trained model’s knowledge without the need for time-consuming full retraining. While existing pioneering editing methods achieve promising results, they primarily focus on editing single-modal language models (LLMs). However, for vision-language models (VLMs), which involve multiple modalities, the role and impact of each modality on editing performance remain largely unexplored. To address this gap, we explore the impact of textual and visual modalities on model editing and find that: (1) textual and visual representations reach peak sensitivity at different layers, reflecting their varying importance; and (2) editing both modalities can efficiently update knowledge, but this comes at the cost of compromising the model’s original capabilities. Based on our findings, we propose DualEdit, an editor that modifies both textual and visual modalities at their respective key layers. Additionally, we introduce a gating module within the more sensitive textual modality, allowing DualEdit to efficiently update new knowledge while preserving the model’s original information. We evaluate DualEdit across multiple VLM backbones and benchmark datasets, demonstrating its superiority over state-of-the-art VLM editing baselines as well as adapted LLM editing methods on different evaluation metrics. Codes are available at https://github.com/zhiyiscs/DualEdit.

Published as a conference paper at COLM 2025
DualEdit: Dual Editing for Knowledge Updating in Vision- Language Models
Zhiyi Shi!; Binjie Wang Chongjie Si Yichen Wul2tt Junsik Kim 1.3 Hanspeter Pfister
"qqul contribution, +Project leader, *Corresponding author
Harvard University, 2City University of Hong Kong, Amazonn, Fudan University Moo Key Lab of Artificial Intelligence, Al Institute, Shanghai Jiao Tong University (gabshi59, bjwang02.hz, wuyichen.am991 mibastro @gmail.com m
Abstract
Model editing aims to efficiently update pre-trained model's knowledge without the need for time-consuming full retraining. While existing pio- neering editing methods achieve promising results, they primarily focus on editing single-modal language models (LLMs). However, for vision language models (VLMs), which involve multiple modalities, the role and impact of each modality on editing performance remain largely unexplored. To address this gap, we explore the impact of textual and visual modali- ties model editing and find that: (1) textual and visual representations reach peak sensitivity at different layers, reflecting their varying impor- tance and (2) editing both modalities can efficiently update knowledge, but this comes at the cost of compromising the model's original capabili- ties. Based on our findings, we propose DualEdit, an editor that modifies both textual and visual modalities at their respective key layers. Addi- tionally, we introduce gating module within the more sensitive textual modality, allowing DualEdit to efficiently update new knowledge while preserving the model's original information. We evaluate DualEdit across multiple VLM backbones and benchmark datasets, demonstrating its supe- riority over state-of-the-ar VLM editing baselines as well as adapted LLM editing methods on different evaluation metrics. Codes are available at https:/ /gittuu.bcoo com /twiyiiis//s//// Schiyiscs/DualEdit.. DutlEdit.
1 Introduction
Large Language Models (LLMs) and Vision Language Models (VLMs) have achieved remarkable success across wide range of natural language processing tasks (Dong et al., 2024; Sheng et al., 2024; Zhu et al., 2023b; Yin et al., 2024), demonstrating strong capabilities in reasoning, knowledge retrieval, and text generation. Despite these advancements, the knowledge encapsulated within LLMs and VLMs usually becomes static after training, which makes it difficult for them to update and correct errors, incorporate new knowledge, or refine specific behaviors in real-world applications (Liang et al., 2024) To efficiently alleviate this problem, model editing has emerged as a promising solution (Wang et al., 2023; Hartvigsen et al., 2023; Chen et al., 2024a; Jiang et al., 2024), allowing targeted modifications to a model's predictions while preserving overall performance and minimizing unintended changes to unrelated inputs.
Previous model editing methods have developed many efficient strategies to address this problem. For instance, methods like ROME (Meng et al., 2022a), MEND (Mitchell et al., 2021), and MEMIT (Meng et al., 2022b) achieve knowledge edits by applying offsets to specific model parameters, while memory-based methods like SERAC (Mitchell et al., 2022), GLAME (Zhang et al., 2024a), and WISE (Wang et al., 2024) leverage external memory for targeted edits. However, most current model editing methods are designed for single-modal models and cannot easily adapt to the increasing significance of multi-modal models (Liang et al., 2024), such VLMs. As pointed out by Cheng et al. (2023), who constructed
1

---

Published as a conference paper at COLM 2025
multimodal editing benchmark, traditional single-modal editing methods perform poorly in multimodal scenarios. This challenge arises because errors in multimodal models frequently stem from the complex interactions between different modalities, such as the intertwined influence of both visual and textual modalities in VLMs (Liang et al., 2024). Therefore, to effectively address the complexities of multi-modal model editing, it is crucial to first fully explore the specific role of each modality and their key layers.
In spite of the increasing prominence of VLMs, existing research on editing these multimodal models is still limited (Cheng et al., 2023). Focusing on editing in VLMs, VisEdit (Chen et al., 2024b) is the first work to identify key layers in the visual modality and edit them to update the model's knowledge. However, while achieving good performance, they concentrate solely on the visual modality and completely neglect the textual modality, which fails to fully recognize the roles of different modalities in VLMs.
To gain a deep understanding of the roles different modalities play in knowledge editing for VLMs and to offer valuable insights for designing multi-modal knowledge editing methods, we begin by conducting a thorough empirical analysis of each modality's s contribution to the model's overall performance. Specifically, through carefully designed experiments, we explore and analyze from the following two perspectives:
Layer-wise and modality-wise importance. In order to investigate the sensitivity of visual and textual modalities, we first analyze the attention scores of different modalities at various layers, which reflect the relative importance of each modality at each layer. The results reveal that, within the same layer, textual modalities receive higher attention scores than visual modalities. Combining this with experiments on the impact of perturbations to each modality at different layers on the final performance, we conclude that the im importance of modalities differs both across layers and within the same layer. Thus, it is crucial to address each modality separately at every layer during knowledge editing.
Trade-Off between reliability and locality. By directly editing visual and textual modal- ities, we find that they can easily adapt to injected modifications for updating new knowledge (i.e., improving the Rel. performance), but at the same time, it also faces greater risk of disrupting existing knowledge (i.e., decreasing the Loc. metric), which aligns with Gekhman et al. (2024). Consequently, while considering the importance of different modalities, we must also be mindful of the trade-off between reliability and locality when editing specific modalities.
Based on these two findings, we introduce DualEdit, a novel editing approach for VLMs that takes into account the distinct effects of textual and visual modalities during editing. Unlike conventional editing methods that treat multimodal inputs uniformly, DualEdit performs modality-aware modifications by applying edits at different layers for visual and textual features. Specifically, we design a gating module that selectively decides whether to edit model's response using a learnable adapter. By applying this mechanism separately to the textual and visual modalities, we enable modality-specifit editing in VLMs while ensuring a well-balanced trade-off between reliability and locality. To sum up, our main contributions can be summarized as follows:
We are the first to conduct comprehensive experiments that decouple the analysis of different modalities in VLM knowledge editing, examining their impact and identifying their relative and absolute importance both across layers and within individual layers.
Based on our findings, we further propose DualEdit, a modality-aware editing approach that operates on some key layers, ensuring well-balanced trade-off between reliability and locality by incorporating the designed gating module.
We conduct comprehensive quantitative and ablation experiments across multiple VLM backbones and benchmark datasets, demonstrating the superiority of DualEdit over state-of-the-art VLM editing baselines as well as adapted LLM editing methods.
2 Analysis of Different Modalities in VLMs
Most existing model editing methods primarily focus on single-modality in LLMs. However, the lack of a comprehensive analysis across different modalities has limited the development
2

---

Published as a conference paper at COLM 2025
Sample Image
0.035
Visual Attention
Textual Attantion
Visual Attention
Average Attention Score
0.030
0.025
0.020
6.015
Sample Text
hat is the name the
0.005
bird in this picture?
0.000
26
Layer
Figure 1: The average attention scores of textual representations and visual representations across different layers in LLaVA-V1.5 (Liu et al., 2023a). The dashed line illustrates the mean attention scores of the three visual representations receiving the highest attention values in each layer. The right figure also shows the attention scores of the sample at [10, 18, 25] layers.
of effective model editing techniques for VLMs. To better understand these impacts, we randomly selected 1000 samples and conducted exploratory experiments across various layers using the widely adopted LLaVA-V1.5 backbone. The results can be summarized in the following two key findings:
Finding-1: Textual and Visual representations in VLMs demonstrate varying levels of importance within the same layer, and the significance of each modality changes across different layers. shown in Figure 1, the average attention scores of textual modalities are significantly higher than those visual modalities, with only a few visual tokens (e.g., the top 3 visual tokens) exhibiting high scores, indicated by the dashed line. Moreover, the attention scores of both modalities peak at different shallow layers. These observations both highlight the distinct treatment of textual and visual modalities across the layers of VLMs, underscoring the necessity of handling different modalities separately.
Porturbation Brects Visual Rap resentations
Porturbation Bfiscts on Tarual Representations
Perturbation Effects Representatioss
0.08
Layer 20
Layer 10
7.0
Layer 10
15
0.07
20
5.5
Divergence
Divergence
Divergence
6.0
0.05
5.5
0.04
0.03
4.5
3.5
0.5
0.1
0.1
0.5
Noise Variance (a)
Noise Variance a
Noise Variance
(b)
(c)
Figure 2: KL divergence between original and perturbed output logiut when adding gaussian noise to (a) visual, (b) textual, and (c) all representations at different layers with different noise variance o.
On the other hand, explore the absolute importance of each modality different layers of VLMs, we introduce varying levels of Gaussian noise into specific layers of given modality and analyze the resulting changes in the output, as depicted in Figure 2 (b) It can be observed that perturbations at different layers affect the textual and visual modalities differently, indicating that the importance of each layer varies across modalities. addition to applying perturbations separately, we also present the results of perturbing both modali- ties simultaneously, as shown in Figure 2 (c). This reveals a distinct pattern compared to previous cases, suggesting that the influence of both modalities on the output is not simply additive. This further underscores the need to carefully consider the interactions between modalities when designing editing methods for VLMs. In summary, by combining the
3

---

Published as a conference paper at COLM 2025
results of Figure and Figure 2, we summarize Finding 1, where the important layers of both modalities lie in different layers.
Finding-2: In VLMs, editing textual and visual modalities can improve the performance of edited samples, but it significantly impacts the performance on original samples (i.e., locality performance).
As illustrated in Table 1, we perform editing on different textual layers (ie., T-Layer) and visual layers (ie., V-Layer). The results show that, compared to the visual modality, the textual modal- ity adapts more easily to edited sam- ples, achieving higher Rel. performance. However, modifying either the textual or visual modality at different layers negatively impacts the model's original performance, leading to lower Loc. performance. This underscores the importance of designing effective editing strategies that minimize unintended disruptions while ensuring successful knowledge updates.
T-Layer
V-Layer
Rel.
T-Gen.
V-Gen.
T-Lac.
M-Loc.
Avg.
16
97.79
97.23
97.48
65.28
71.26
89.81
17
97.83
97.41
97.39
86.71
71.90
90.25
18
19
97.75
96.75
97:52
89.18
80.05
92.25
16
95.82
95.35
96.76
100.00
76.68
92.72
96.05
95.31
96.72
100.00
76.14
92.64
93.89
94.59
10000
77.96
92.79
Table 1: Ablations of editing different modalities at key layers. Details of metrics please see Sec. 3.1.
3 The Proposed DualEdit
As analyzed in Sec. 2, independently editing different modalities at their key layers is essential while maintaining an optimal balance between reliability and locality. In this section, we first introduce the base setting and evaluation metrics in Sec. 3.1, followed by an introduction to the framework and training details of DualEdit in Sec. 3.2, aiming to address the issues discussed earlier.
3.1 Preliminary
Consider a vision language model fe, where 9 represent the model parameters. This VLM model fo maps the visual inputs x, and textual input x; to the original output (i.e., fa(x,, x = o .. For a given edit sample (x,, xi, 0²) where fa(x,, xf + 0r, the VLMs editor MF(-) can generate the edited model fe, = Mf (fa, x xf, o²) such that fe, (X,, xf) of. Meanwhile, a good editor Mg(-) should also satisfy the following several criteria (Chen et al., 2024b; Cheng et al., 2023).
Reliability (Rel.) evaluates the accuracy of the post-edit model fe, on edit samples:
E - H[f, (x,, xf) = of),
(1)
where D, refers to the set of edit samples, and [(.) is the indicator function.
Generality (Gen.) requires the edited model fe, to accurately predict the correct output for inputs relevant the exact edited samples. In the context of VLMs, generality can be further categorized into Textual Generality and Visual Generality. Textual Generality (T-Gen.) ensures that the model editing is robust to semantically equivalent variations in textual input. This reflects whether the edited model can correctly respond to the paraphrases or variations of a specific textual input. Similarly, Visual Generality (V-Gen.) ensures that the model editing remains effective across semantically equivalent variations in visual input. These can be individually expressed as:
(T-Gen.) I (x JDD of ( ff (X,, xS) = }
(2)
(V-Gen.) E (x Egg N(x)) [ffe, (xS, xx) = of
where N/(.) means the neighborhood of various edit inputs.
Locality (Loc.) ensures that the edited model fa, maintains consistency with the original model fo for samples unrelated to the edited samples. Similar to Generality in VLMs, Locality also consists of two items, Textual Locality (T-Loc.) and Multimodal Locality (M- Loc.). The T-Loc. measures whether the edited model produces the same output as the
4

---

Published as a conference paper at COLM 2025
Edit Samples
(Laxr))
Manti multypication
Xp
Flement wise aubittion
L Learnable A daater
L Learnable A dapter
Wi Lesttable meitix for visusl modeiclty
Wi Leucrable muttii modilitty
U Lest token rogresentation used Gatlin
oe
0000
Gating Module
Wi
Ixput Sample
Softmax
Xt
WS WW
(Layer))
(Layar))
The framework of the proposed DualEdit method. Two mppalit learnable adapters are inserted at designated layers. The gating module uses the representation of the last tokens to jointly control the activation of both adapters.
3: 3:
original model when handling text-only samples which are unrelated the edited samples, while M-Loc. reflects whether the model retains the original outputs when both visual and textual inputs are unrelated to the edited samples, as shown in the following equations:
(T-Loc.) B (x E (2) u(x) Tia, (0, fo(0, = 6'},
(3)
(M-Loc.) E (16,,,o))) (x - - fl.o (Rg, x/) (x x1) 6'},
where u() represent sample sets unrelated to the edited sample.
3.2
The Designed DualEdit Algorithm
Based on the findings discussed in Sec. 2, we propose the DualEdit algorithm, which enables editing of both modalities their respectively important layers. To achieve a balanced trade- off between Rel and Loc. performance, we introduce gating mechanism that leverages the cosine similarity of the last token representations. The framework illustrated in Figure
Gating mechanism. As shown in Figure 3, we design gating module that determines whether to apply the editing operation. If it can effectively distinguish edit samples, the gating module function operates similarly to Mixture of Experts (Cai et al., 2024): edit samples are processed through a dedicated learnable adapter (i.e., edited model fe.) to enhance reliability, while other samples are handled by the original model fe, ensuring high Loc. performance.
The key question in designing gating mechanism is how to accurately distinguish whether the input samples are edit samples. Inspired by some studies (BehnamGhader et al., 2024; Sheng et al., 2024) showing that the latent space of LLMs inherently contains rich feature information, we leverage this property in our gating mechanism by directly computing the similarity between the edit sample and the input sample in the latent space. Specifically, for a given layer, let the last token representation of the edit sample and the input sample be represented as h h Rd where is the dimension of the hidden space. Simply yet effectively, we calculate the cosine similarity of the two last token representations to determine the gating threshold by the following equation:
he hi
Sim
| |
As shown in Figure 4 (c), our gating strategy, which computes the cosine similarity the last token representations, can effectively differentiate between editing examples (indicated by

---

Published as a conference paper at COLM 2025
Visul Representation Similaity Distributon
Jestual Representation Siilantty Distrbbtion
Last Taken Representation Simlarty Distribution
17.
40
viem Mix.
viden
15.0
Miet:
Muler:
Density (96)
Density (9)
10.0
15
20
7.5
0
0.26
0.28
0.30
0.32
0.34
0.26
0.38
0.0
0.15
0.20
0.25
0.30
6.35
0.40
0.2
0.4
0.5
0.0
Similarity
Similarity
Similarity
(a)
(b)
(C)
Figure 4: Gating module analysis: cosine similarity distribution of different types of repre- sentations across different samples. (a), (b) and (c) are similarity distributions calculated based visual representations, textual representations, and the last representations.
Gen. performance) and original input samples (reflected by Loc. performance) by applying appropriate threshold T. To further verify the effectiveness of our proposed gating module based on the last token representation, we also plotted histograms showing the similarity of regular textual and visual representations. As shown in Figure 4 (a) and (b), compared to our last-token-bass approach, the regular textual and visual representations fail to effectively distinguish between the examples.
Learnable adapters across key layers in different modalities. By introducing the gating method, we effectively improve Loc. performance. To further enhance the editing performance, we investigate the impact of reliability at different layers, as shown in Figure The findings indicate that the 16th textual layer and the 19th visual layer deliver the best outcomes. Therefore, in our implementation, we focus on layer i=16 and layer j=19 as outlined in the flowchart of Figure 3.
For the learnable adapter, as shown in Figure 3, it includes two inputs: (1) the k-th layer representations h obtained by feeding the edit sample (XL, xf, 0²) into the model foi (2) the k-th layer textual representation hţ or visual rep- resentation h.. The output is the edited k-th layer tex- tual or visual representation. For simplicity, we utilize cross-attention the learnable adapter, which introduces separate learnable weights for different modalities. The concrete equation can be expressed as,
Reliability (%)
98.02
97.93
97.96
98.29
127
98.04
98.12
97.03
98.06
118
98.06
98.10
96.02
98.11
119
98.10
98.01
96.18
98.09
h Softmax (hi/ W//e (b/W)//)T) h/
Visualization of the sensitivity of learnable adapters across different layers. (1,2,3) represent the learnable
Figure 5:
where hū and füt denote the edited k-th layer textual and visual representations, respectively. The notation w textual or visual projection matrices.
Loss functions. Following Cheng et al. (2023), the training loss l of the proposed EualEdit includes three components: reliability loss Eee,, generality loss Egen and locality loss f which can be expressed as:
t= Lre + fgen + 4
(5)
where the details of individual terms of fret, Egen and fioe are listed in Appendix A.
Experiments
4.1 Experimental Settings
We evaluate E-VQA and E-IC datasets (Cheng et al., 2023; Chen et al., 2024b). For VLM backbones, we select BLIP2-OPT-2.7B (Li et al., 2023) and LLaVA-V1.5-7B (Liu et al., 2023a), which differ in architecture and training: BL.IP2 uses Q-Former and contrastive pretraining,

---

Published as a conference paper at COLM 2025
Methods
E-QQA
Avg.
EIC
Avg.
Rel.
T-Gen.
V-Gen.
o-
M-Loc.
Rel.
T-Gen.
VoGen.
T-L.cc
M-Loc.
N/ OPP 21223)
FT-V (Chenget ata, 2023)
24.01
16.00
20.22
100.00
88.65
49.78
42.11
40.74
36.43
100.00
89.73
61,80
FI-L (Cheng et 2023)
24.86
16.39
20.57
98.92
89.61
90.17
41.61
40.31
37:41
88.70
61.48
KR (De Can 2021)
67.80
63.00
66.17
97.32
45.89
68.04
69.00
62.80
61.22
96-21
48.35
66.96
IKE (Li ct 2022))
99.95
91.59
92.33
13.16
1
59.78
96.70
78.20
83.15
13.36
2.17
54.72
SERAC (Mitchell et 2022)
89.81
MEND (Mitchell
2021)
92.60
90.80
91.94
96.07
68.15
87.51
63,90
38.00
36.19
TP (1 (Ltung al., 2023)
68.31
60.58
56.35
98.49
85.27
73.86
49.71
49.03
45.46
93.88
80.88
63.79
LTE (jiang et al, 2024)
97.74
97.21
96.35
94.34
84.99
94.13
96.69
95.26
94,06
95.25
87.68
43.79
VisEdit (Chon et 2024h)
2024b)
97.34
96.62
84.90
95.16
95.00
94.24
10000
93.11
95.51
DualEdit (Ours)
97.89
98.05
100.00
99.89
96.86
96.72
96.27
100.00
100.00
97.97
ML 5 (Liu et 2023a)
FT-V (Cheng et al, 2023)
31 68
29.96
26.6
100.00
91.25
55.91
52.85
51.57
48.63
100.01
V2.55
69.12
FT-L (Cheng atal,
31.78
30:02
26.91
92.05
56.14
53.00
51.02
49.29
98.91
94.89
69.42
KE (De Cao 2021)
85.85
84.00
82.23
93.57
73.05
83.74
83.54
82.15
81.12
92.46
73,83
82.62
IKE (Li 2012)) 20023)
91.35
90.84
91.08
60.18
51.08
76.91
43.72
88.37
76.99
76.61
64.90
80.12
SERAC (Mitchell et al,, 2022)
82.51
81.60
80.05
100.00
57.48
80.33
43.08
42.57
42.85
100001
7.6
47.19
MEND (Mitchell al. 2021)
92.10
(Huang al. 2023)
38.68
36.27
31.26
91.41
58
99.07
57
55.51
89.26
65.13
LTT (liag et al, 2024)
94.16
93.54
93.06
83.76
81. 65
89.23
93.60
92.38
91.18
85.54
88.49
90.24
VisEdit (Chen et 2024bb)
95.78
94.21
94.37
100.00
91.
95.09
95. 95.06
94.87
94.35
100.00
95.23
95.90
Duallidit (Ours)
100.00
Table 2: Main results of BLIP2-OPT and LLaVA-V1.5 on E-VQA and E-IC datasets. The best
result is marked bold.
while LLaVA adopts projection layer and instruction tuning. As VisEdit is the only editing method tailored for VLMs, we compare it against adapted LLM editors including FT-V, FT-L, KE, IKE, MEND,, TP, and LTE.
4.2 Main Results
Overall analysis. According to Table 2, DualEdit demonstrates superior performance compared to other editors across both datasets. DualEdit achieves the highest average scores in both E-VQA and E-IC across different backbones, outperforming all single-modalle editors. These results validate the significance of the modifications different modalities.
Superior M-Loc. performance. DualEdit excels particularly in the M-Loc. metric, achiev- ing near-perfect scores (99.89% and 100.00% with BLIP2-OPT; 99.61% % and 99.74% with LLaVA-V1.5) across both datasets and backbones. The exceptional M-Loc. performance demonstrates the effectiveness of our novel gating mechanism based on the last token repre- sentation cosine similarity, which enables precise protection of locality-related samples. This approach allows VLMs to intelligently identify which content should remain unchanged while still implementing necessary edits, effectively solving a key challenge in model editing. In contrast, methods like MEND show extremely poor M-Loc. performance, indicating widespread unintended modifications that compromise model integrity.
4.3 Ablation Analysis
We conduct a comprehensive ablation study to evaluate the effectiveness of different com- ponents of DualEdit, with results presented in Table 3. The ablation experiments examine: (1) the impact of editing at different layers (early, middle, and late layers) in the model, (2) the effectiveness of dual-path editing compared to single-path editing, and (3) the influence of our proposed gating mechanism.
Impact of editing at different layers. Rows 1-5 in Table 3 show the performance when applying edits at different depth levels without the gating mechanism. The results demon- strate that editing performance varies significantly depending on which layers are targeted. When edits are applied to very early layers (T-Layer=1, V-Layer=2), average performance is limited (74.91% on E-VQA and 79.24% on E-IC). Performance improves as we move toward middle layers (T-Layer=5, V-Layer=5) and reaches its peak when targeting interme- diate layers (T-Layer=10, V-Layer=10), achieving 88.58% on E-VQA and 88.25% on E-IC. Interestingly, editing very deep layers (T-Layer=30, V-Layer=30) leads to a substantial drop in performance (60.97% on E-VQA and 70.93% F-IC). Row 6 shows our empirically
7

---

Published as a conference paper at COLM 2025
T-Layer V-Layer
Gating
E-VQA
Avg.
E-IC
Avg.
Rel.
T-Gen.
V-Gen.
T-Loc.
M-Loc.
Rel.
T-Gen.
V-Gen.
o-
M-Loc.
x
93.50
93.34
92.76
63.65
31.32
74.91
79.49
79.50
76.40
86.63
74.17
79.24
94.03
93.60
96.10
59.45
26.50
73.34
82.25
82.19
79.69
85.79
74.13
80.81
96.72
96.63
96.35
26-92
76.46
84,70
54,25
82.57
82.98
73.58
81,72
10
10
X
97.98
97.56
97:91
81.77
67.70
88.38
93.11
93.06
92.00
87.58
75.52
88.25
30
30
54.99
52.40
52.26
88.88
56.31
60.97
57.46
56.27
52.83
95.19
92.89
70,93
98.29
72.02
91.36
96.72
96.27
92.05
16
97.79
97.23
97.48
100.00
99.88
98. 48
82.61
82.41
80:49
99,35
96
88.74
19
95.89
94.59
95.49
100.00
99.90
97.17
94.00
93.96
93.19
100.00
100.00
96.23
16
98.29
97.89
98.05
100.00
99.89
98.92
96.86
96.72
96.27
100.00
100.00
97.97
Table 3: Ablation results of BLIP2-OPT on E-VQA and E-IC datasets, exploring different layers and the impact of our proposed gating module. The best result is marked in bold.
determined optimal layer configuration (T-Layer=16, V-Layer=19) without gating, which achieves strong performance of 91.36% on E-VQA and 96.38% on E-IC.
Dual-Editing vs. Single-Editing. Rows 7-8 examine the effectiveness of our dual-editing approach compared to single-editing alternatives, both with the gating mechanism en- abled. In row 7, we apply edits only to the textual representations, while in 8, we apply edits only to the visual representations. Our full DualEdit approach (row 9), which simultaneously edits both textual and visual representations (T-Layer==6, V-Layer=19) with gating, consistently outperforms single-editing approaches across both datasets, achieving 98.82% on E-VQA and 97.9 97.97% on E-IC. This demonstrates the complementary D benefits of modifying both modality paths, allowing the model to integrate edited knowledge more comprehensively.
Effectiveness of gating mechanism. The most significant improvement comes from our proposed gating mechanism, as evidenced by comparing rows 6 and 9, which have identical layer configurations (T-Layer=16, V-Layer=19) but differ in whether gating is enabled. Without gating (row 6), the model achieves 91. 91.36% on E- E-VQA and 81.77% on E-IC. With gating (row 9), performance jumps significantly to 98.82% E-VQA (+7.46%) and 97.97% E-IC (+16.20%). The benefits gating are particularly evident in the locality metrics (T-Loc. and M-Loc.), which measure the model's 's ability to preserve unrelated knowledge. Without gating, M-Loc. scores are 72.02% on E- E-VQA and 92.05% on E-IC. With gating enabled, these scores improve dramatically to 99. 9989% and 100.00%, respectively, approaching perfect preservation of unrelated knowledge. This substantial improvement confirms that our gating mechanism successfully prevents unwanted modifications to non-target knowledge while allowing precise edits on target knowledge.
Overall, our ablation study highlights three key insights. First, layer selection is critical: editing intermediate layers yields the best performance, while very early or very deep layers lead to suboptimal results. Second, modality-speciffi edits are complementary: editing only one modality benefits certain tasks, but dual-editing consistently achieves the best overall performance. Third, and most notably, our gating mechanism plays crucial role in balancing knowledge integration with preservation.
Related Works
5.1 Vision Languages Model
Earlier vision-language models, exemplified by CLIP (Radford et al., 2021), establish align- ment between textual and visual information within shared hidden spaces through con- trastive learning on extensive datasets. These models demonstrate remarkable generaliza- tion across diverse tasks with minimal adaptation requirements. Building upon these foun- dations, VLMs have successfully bridged the gap between visual and linguistic modalities, achieving exceptional results in various applications including in-context predictions (Liu et al., 2023b; Salewski et al., 2023), multi-image understanding, and Chain-of-thoughe rea- soning (Driess et al., 2023; Yang et al., 2023). The landscape of Large VLMs encompasses diverse architectural designs that reflect varying approaches to multimodal integration and
8

---

Published as a conference paper at COLM 2025
processing (Wadekar et al., 2024). Token Fusion represents architectural paradigm where tokenized input modalities are fed directly into the model's s input stage. This approach employs either decoder-only transformer multimodal integration mechanism, as exemplified by models like LIV (Jin et al., 2024). Deep Fusion approaches incorporate visual information into the internal layers of the LLM through cross-attention mechanisms. While models like PaLI-X (Chen et al., 2023) and Flamingo (Alayrac et al., 2022) implement standard rross-attention layers, alternatives such as LLaMA-AAapteer(haar et al., 2024b) utilize custom-designes components to process visual representations before cross arte-ttttention operations.
the an encoder-decoder style transformer
Early Projection Fusion represents another prevalent strategy where non-tokenized visual inputs undergo processing before being introduced at the model's input rather than within internal layers. Various connection modules facilitate this integration, including linear projection layers, Q-formers with linear projections, perceiver resamplers, and custom learnable components. Notable implementations of this approach include Qwen2.5-VL (Bai et al., 2025), LlavaV1.5 (Liu et al., 2024), and MiniGIT4 (Zhu et al., 2023a), which have demonstrated superior capabilities in multimodal understanding and generation. This paper primarily focuses on examining and enhancing editing capabilities within this class of vision-language models.
5.2 Model Editing
Model editing in large language models (LLMs)) lies at the intersection of continual learn- ing (Wu et al., 2024a;b; 2025) and parameter-efficient fine-tuning (Si et al., 2025), aiming to incorporate new factual knowledge or behavioral changes into models with minimal for- getting and computational cost. From the CL perspective, model editing seeks to update model knowledge while mitigating catastrophic forgetting (Lopez-Paz & Ranzato, 2017), whereas from the PEFT angle, it emphasizes modifying only a small subset of parame- ters to achieve targeted updates (Hu et al., 2022). Recent advances in model editing for LLMs can be broadly classified into three paradigms: parameter modification, module-based augmentation, and prefix-bused instruction injection. Parameter modification methods aim to directly adjust the internal weights of a model in response to specific edit instructions. Among these, Knowledge Editor (KE) (De Cao et al., 2021) and MEND (Mitchell et al., 2021) adopt a learning-basee based approach, where an auxiliary network is trained to generate weight deltas based on edit signals. On the other hand, ROME (Meng et al., 2022a) and MEMIT extension (Meng et al., 2022b) leverage tools from causal inference. A different line of work explores module-based augmentation to achieve editing without overwriting existing model knowledge. For instance, SERAC (Mitchell et al., 2022) trains edit-aware counterfactual model that only activates under relevant conditions. TP (Huang et al., 2023) introduces editable "knowledge neuron" that can be trained separately from the base model. GRACE (Hartvigsen et al., 2023) routes inputs to a target editing output in latent space, conditional on their similarity crossing a predefined threshold. MELO (Yu et al., 2024) builds upon GRACE by retrieving and injecting editing matrices related to the input query, thereby enabling efficient updates to the model's predictions. In contrast, prefix-tuning approaches avoid changing model weights by manipulating the context seen during inference. IKE (Zheng et al., 2023) applies in-context learning to guide the model's output based on few-shot edited prompt, while LTE (Jiang et al., 2024) explicitly trains the model to follow editing instructions. RECIPE (Chen et al., 2024a) further advances this line by introducing learnable prompt generator that finds the shortest continuous prefix capable of inducing behavior.
the desired
Although VLMs have gained increasing attention, research editing such models remains relatively underexplored (Cheng et al., 2023). VisEdit (Chen et al., 2024b) represents the first attempt to identify and edit key layers within the visual modality. However, different from our DualEdit, it concentrates solely on the visual modality and completely neglects the textual modality, which fails to recognize the influence of different modalities in VLMs.

---

Published as a conference paper at COLM 2025
6 Conclusion
In this work, we investigate the largely unexplored problem of editing VLMs and highlight the distinct roles that textual and visual modalities play in this process. Through comprehen- sive empirical analysis, we reveal that textual and visual representations exhibit different sensitivity patterns across layers, and we also discover that editing both modalities can effectively update the model's knowledge, but it may compromise its original capabilities. Motivated by these findings, we propose DualEdit, a modality-aware editing framework that applies targeted updates to each modality at its respective key layer. Additionally, gating mechanism based on the similarity of the last token representation helps DualEdit achieve a better balance between knowledge incorporation (i.e., reliability and generality) and preservation (i.e., locality). Extensive experiments across multiple VLM backbones and datasets demonstrate that DualEdit consistently outperforms both VLM-specific and adapted LLM editing baselines, setting new standard for multimodal model editing.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Has- son, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Mon- teiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning, 2022. URL /2044. 14198.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https: /ttps:///xxvvooooe abs/250.. 13923.
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. Xiv preprint ar Xii::204.005611 2024.
Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang, A survey on mixture of experts. arXiv preprint ar #xXv:2407.06204 2024.
Qizhou Chen, Taolin Zhang, Xiaofeng He, Dongyang Li, Chengyu Wang, Longtao Huang, and Hui Xue. Lifelong knowledge editing for iims with retrieval-augmented continuous prompt learning. Xiv preprint ar Xiv::2405.03279, 2024a.
Qizhou Chen, Taolin Zhang, Chengyu Wang, Xiaofeng He, Dakan Wang, and Tingting Liu. Attribution analysis meets model editing: Advancing knowledge correction in vision language models with visedit. ar Xiv preprint ar axXio:2408.09916, 2024b.
Xi Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Car- los Riquelme Ruiz, Sebastian Goodman, Xiao Wang. Tay, Siamak Shakeri, Mostafa De- hghani, Daniel Salz, Mario Lucic, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Man- dar Joshi, Bo Pang, Ceslee Montgomery, Paulina Pietrzyk, Marvin Ritter, AJ Piergiovanni, Matthias Minderer, Filip Pavetic, Austin Waters, Gang Li, Ibrahim Alabdulmohsin, Lucas Beyer, Julien Amelot, Kenton Lee, Andreas Peter Steiner, Yang Li, Daniel Keysers, Anurag Arnab, Yuanzhong Xu, Keran Rong, Alexander Kolesnikov, Mojtaba Seyedhosseini, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. Pali-x: On scaling up multi- lingual vision and language model, 2023. URL https: //tp://aaivvv///a www.ii.oorg///bbee htrrgg/bb/233555 18565.
Siyuan Cheng, Bozhong Tian, Qingbin Liu, Xi Chen, Yongheng Wang, Huajun Chen, and Ningyu Zhang. Can we edit multimodal large language models? arxie preprint ar arXit:2110.08475 2023.
10

---

Published as a conference paper at COLM 2025
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. arXiv preprint (rXiv:2144.08144, 2021.
Junnan Dong, Qinggang Zhang, Huachi Zhou, Daochen Zha, Pai Zheng, and Xiao Huang. Modality-aware integration with large language models for knowledge-based visual question answering. arXiv preprint ar arXio:2402.12728, 2024.
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model, 2023. URL https://arxiv.. org/abs/2303. 03378.
Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning llms on new knowledge encourage hallucinations? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 7765-7784, 2024.
Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. Advances in Neural Information Processing Systems, 36:47934-47959) 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. Transformer-patcher. One mistake worth one neuron. arXio preprint ar arXiv:2301.0978. 2023.
Yuxin Jiang, Yufei Wang, Chuhan Wu, Wanjun Zhong, Xingshan Zeng, Jiahui Gao, Liangyou Li, Xin Jiang, Lifeng Shang, Ruiming Tang, et al. Learning to edit: Aligning iims with knowledge editing. Xiv preprint Xxxv:22402.11903 2024.
Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Yadong Mu, et al. Uni- fied language-vision pretraining in Ilm with dynamic discrete visual tokenization. In International Conference on Learning Representations, 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19742. PMLR, 2023.
Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, pp. 405-409, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892- 34916, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024. URL https: http://aavvvvvv///e orgg /35
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shri- vastava, Ce Zhang, Yuandong Tian, Christopher Re, and Beidi Chen. Deja vu: Contextual sparsity for efficient 1lms at inference time, 2023b. URL https://arxvv.or//abb//22 17157.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 111177599-77722 2022a.
11

---

Published as a conference paper at COLM 2025
Kevin Meng, Arnb Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in a transformer. ar preprint ar arXi::2210.07229, 2022b.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing scale. arXiv preprint Xfo::2110.11309, 2021.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher Manning, and Chelsea Finn. Memory-based model editing at scale. In International Conference on Machine Learning, PP. 15817-15831. PMLR, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever: Learning transferable visual models from natural language supervision, 2021. URL https: wwreiiiiorr////o orggabs/2100) 00020.
Leonard Salewski, Stephan Alaniz, Isabel Rio-Torto, Eric Schulz, and Zeynep Akata. In- context inmerssoatm reveals large language models' strengths and biases, 2023. URL
https://arriv..or//bb/22235 14930.
Leheng Sheng, An Zhang, Yi Zhang, Yuxin Chen, Xiang Wang, and Tat-Seng Chua. Language representations can be what recommenders need: Findings and potentials. ar Xiv preprint arXiv:2407005441, 2024.
Chongjie Si, Zhiyi Shi, Shifan Zhang, Xiaokang Yang, Hanspeter Pfister, and Wei Shen. Unleashing the power of task-specific directions in parameter efficient fine-tuning. In The Thirteenth International Conference on Learning Representations, 2025.
Shakti N. Wadekar, Abhishek Chaurasia, Aman Chadha, and Eugenio Culurciello. The evolution of multimodal model architectures, 2024. URL https://arriv.org//b/////e 17927.
Peng Wang, Ningyu Zhang, Bozhong Tian, Zekun Xi, Yunzhi Yao, Ziwen Xu, Mengru Wang, Shengyu Mao, Xiaohan Wang, Siyuan Cheng, et al. Easyedit: An easy-to-use knowledge editing framework for large language models. arXiv preprint (xXi:2330..07269 2023.
Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Wise: Rethinking the knowledge memory for lifelong model editing of large language models. Advances in Neural Information Processing Syslems, 37: 33764-53797 2024.
Yichen Wu, Long-Ka Kai Huang, Renzhen Wang, Deyu Meng, and Ying Wei. Meta contin- ual learning revisited: Implicitly enhancing online hessian approximation via variance reduction. In The Twelfth international conference on learning representations, 2024a.
Yichen Wu, Hong Wang, Peilin Zhao, Yefeng Zheng, Ying Wei, and Long-Kai Huang. Mitigating catastrophic forgetting in online continual learning by modeling previous task interrelations via pareto optimization. In Forty-first International Conference on Machine Learning, 2024b.
Yichen Wu, Hongming Piao, Long-Kai Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, and Ying Wei. Sd-lora: Scalable decoupled low-rank adaptation for class incremental learning. The Thirteenth International conference on learning representations, 2025.
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasa,, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action, 2023. URL
22000./2333... 1381.
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Zhiyuan Zeng, Xiaonan Li, Junqi Dai, Qinyuan Cheng, Xuan-Jing Huang, and Xipeng Qiu. Reasoning in flux: Enhancing large language models reasoning through uncertainty-awant adaptive guidance. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2401-2416, 2024.
12

---

Published as a conference paper at COLM 2025
Lang Yu, Qin Chen, Jie Zhou, and Liang He. Melo: Enhancing model editing with neuron- indexed dynamic lora. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19449-19457, 2024.
Mengqi Zhang, Xiaotian Ye, Qiang Liu, Pengjie Ren, Shu Wu, and Zhumin Chen. Knowledge graph enhanced large language model editing, arv preprint ar arXic:2402.13593, 2024a.
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention, 2024b. URL https: //atpps www.://gg///b//a//eee 16199.
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? arXiv preprint arXto:2305.12740, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023a. URL http:::/aan.xv..rrg///////// 10592.
Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, and Ji-Rong Wen. Large language models for information retrieval: survey. arXiv preprint arXiv:2308.0710/ 2023b.
13

---

Published as a conference paper at COLM 2025
A
Loss Function
The individual loss functions felr fgen and Loo are expressed as,
rel
=
Σ
log (o0 xg),
J.DD
Egen
Σ
log fo. (o (0, xf) - log fo x9),
E KL (fo (0'12),,)) llfo, (o' (o112/r (611%)xx))) KL (fo (o'10, llfa (d'10xx))
B Dataset Information
E-VQA focuses on editing visual question answering capabilities, where the dataset contains examples requiring models to correct their answers to questions about images. The training set includes 6,346 examples, with 2,093 in the test set. E-IC targets image captioning abilities, containing examples where models need to improve or correct their image descriptions. has 2,849 examples for training and 1,000 for testing.
Both datasets are evaluated on reliability (successful editing of target examples), locality (preserving unrelated knowledge), and generality (maintaining consistency with similar in- puts). They'n re used to assess how effectively different editing methods can correct mistakes in multimodal LLMs without disrupting their overall performance.
C Experimental Settings
Following (Chen et al., 2024b), we utilize E-VQA (Editing Visual Question Answer- ing) and E-IC (Editing Image Caption) proposed by (Cheng et al., 2023) as our evaluation datasets.
Datasets:
VLM Backbones: BLIP2-OPT-2.7BBL. (Li al., 2023) employs a Q-Former architecture to align image-to-visual representations, while LLaVA-V1.5-77 (Liu et al., 2023a) uses projection layer for alignment. BLIP2-OPT--2.7B features fewer parameters and was pre-trained with image-text contrastive learning, whereas LLaVA-Vi.5-7B contains more parameters and underwent instruction tuning with conversation data. Considering these differences in architecture, parameter size, and training methodology, we selected these two models to serve as fair representatives for VLM backbones in our editor evaluation.
Baselines: To our knowledge, VisEdit currently stands the only editing approach specifi- cally designed for vision-language models. Consequently, most prior research has adapted general language model editing methods to the vision-language domain. These adapted approaches include FT-V (Fine-tunes visual encoder), FT-L (Fine-tunes last layer of the language model), KE (De Cao et al., 2021), IKE (Zheng et al., 2023), SERAC (Mitchell et al., 2022), MEND (Mitchell et al., 2021), TP (Huang et al., 2023) and LTE (Jiang et al., 2024). We utilize both vasiun-language model approaches and adapted large language model techniques as our comparative baselines.
D Implementation Details
Choice of gating threshold: We set the threshold to 0.6 when using LLaVA-V1.5 the backbone model and 0.8 when using BLIP2-OPT as the backbone.
Training typer-parameters: The optimal DualEdit insertion lavers and training hyperpa- rameters remain consistent across both BLIP-OPT and LLaVA-V1.5 models. The DualEdit module is optimally inserted at layers 16 and 19. Layer 16 is responsible for modifying the
14

---

Published as a conference paper at COLM 2025
textual representations, while layer 19 handles the modification of visual representations. We set the learning rate to 0.0001, the training batch size to 4, and the maximum number of iterations to 50000. A checkpoint is saved every 1000 iterations. We use the checkpoint with the smallest loss for evaluations. The training process requires approximately day on NVIDIA H100 GPUs.
15


Reviews:
[{"review_id": "eE52rAmR81", "reviewer": "Reviewer_Ykrx", "summary": "Most existing model editing methods are designed for models that only have one modality. These methods cannot be applied to VLM efficiently. In order to refine the VLM behaviors, this paper made an assumption that each modality in VLM plays a different role in knowledge editing. The analysis in this paper focuses on two perspectives: 1. Attention scores of different modalities at various layers; 2. How to maintain the reliability when editing specific modalities. Based these two findings, this paper proposes DualEdit method that considers both textual and visual modalities. The key is to propose a gating module using a learnable adapter.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper proposed a comprehensive analysis on the importances of various modalities in VLM. It provides some insights about identifying the role of modalities cross layers and within layers.\n\n- The proposed method DualEdit ensures a balanced tradeoff between reliability and locality.\n\n- This proposed method shows very reasonable performance across various VLM benchmarking datasets", "reasons_to_reject": "- These two findings have been explored in some papers before. For example, there are many papers that want to emphasize the visual modality because they found the VLMs haven’t fully captured the image information. In addition, balancing the reliability and locality is an important aspect in the model editing methods. Hence these two findings have been proposed before. The novelty of this analysis is limited.\n\n- Training the adapters needs the data. Preparing the pairs including original sample and edit sample will be a time-consuming task. This paper hasn’t explained how to prepare these edit samples clearly. This is one of the most important part this paper needs to clarify. It is better to explain if we can prepare the data for model training easily.\n\n- If we trained an adapter using one type of data, I am not sure if it can be adapted a domain that is totally difference with the training data. Providing the generalization analysis of proposed method is needed for this paper", "questions_to_authors": "Please answer the questions listed in \"reasons to reject\".", "extra_scores": {}}, {"review_id": "WgiyHJZiRd", "reviewer": "Reviewer_DZuA", "summary": "This paper addresses the challenge of model editing in Vision-Language Models (VLMs), a task largely under-explored compared to single-modal LLM editing. The authors conduct a empirical analysis to understand how visual and textual modalities contribute differently to knowledge representation and sensitivity across model layers. Based on their findings, they propose DualEdit, a novel editing framework that independently edits visual and textual features at their respective sensitive layers. A key component is a gating mechanism based on the cosine similarity of last-token representations, which helps to preserve locality while allowing targeted edits. The method is evaluated on two VLM backbones (BLIP2 and LLaVA) and two benchmarks (E-VQA and E-IC), outperforming existing baselines in reliability, generality, and locality.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provides a much-needed exploration of editing in multimodal models, an area that lags behind LLM editing research.\n- DualEdit achieves excellent performance on both accuracy and locality metrics, particularly through its gating mechanism.\n- The evaluation includes diverse datasets, lending credibility to the results.", "reasons_to_reject": "- Visual attention maps across layers show limited variation, raising questions about the necessity of editing at different V-Layers.\n- The conceptual novelty may feel limited, as the work extends existing ideas (like adapter-based editing and modality-specific routing) rather than introducing a fundamentally new editing paradigm.", "questions_to_authors": "- Given the limited difference in attention maps across V-Layers in Table 1, is the visual editing branch necessary or could simpler alternatives suffice?\n- Could the gating mechanism be adapted to perform soft gating (e.g., attention-weighted blending) rather than binary on/off activation?\n- What is the sensitivity of the method to threshold selection and layer choice across different VLMs?", "extra_scores": {}}, {"review_id": "gzld9ZWLxD", "reviewer": "Reviewer_CGGe", "summary": "The paper proposes DualEdit, a dual-modality editing framework for Vision-Language Models (VLMs) that addresses the underexplored challenge of updating knowledge in multimodal settings. Existing editing methods primarily focus on single-modal LLMs, but VLMs involve complex interactions between visual and textual modalities. Through empirical analysis, the authors identify that textual and visual representations exhibit varying sensitivity across different layers, with textual modalities generally more influential. DualEdit introduces modality-specific edits at key layers (e.g., textual layers at 16, visual layers at 19) and incorporates a gating mechanism based on cosine similarity of last-token representations to balance reliability (successful edits) and locality (preserving unrelated knowledge). Experiments on E-VQA and E-IC datasets with BLIP2-OPT and LLaVA-V1.5 backbones demonstrate DualEdit’s superiority over VLM-specific and adapted LLM editing baselines, achieving near-perfect locality preservation and high reliability.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- First comprehensive study decoupling modality-specific impacts in VLM editing, revealing critical insights (e.g., layer-wise sensitivity differences).\n\n- DualEdit’s modality-aware design and gating mechanism offer a principled solution for multimodal editing, outperforming state-of-the-art methods.\n\n-  Extensive experiments across multiple backbones (BLIP2, LLaVA) and datasets (E-VQA, E-IC) validate effectiveness, with detailed ablation studies justifying design choices.\n\n- The authors provides the various analysis of model editing data and model modules to facilitates the readers understanding.", "reasons_to_reject": "- Although the authors provide empirical evidence for the effectiveness of DualEdit, the theoretical foundations of the method are relatively weak. For instance, the selection of key layers for editing and the design of the gating mechanism lack rigorous theoretical analysis, making it difficult to fully understand the underlying principles of the method.\n\n- Experiments are restricted to two datasets and backbones; broader validation (e.g., on more VLMs like Flamingo or PaLI) would strengthen claims. Also, the data analysis of Sec. 2 only uses the LLaVA-V1.5 as the backbone to conduct the VLM's layer and metrics' charateristics. \n\n- The results of baselines are different from the original baselines' paper. For example, the results reported by VisEdit [1] are better than the metrics of Table 2.\n\n[1] Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Quality: Reviewres agree that the paper contribution is valuable.

Clarity: Reviewers agree that the paper is clear and the methods described seem replicable. 

Originality: Reviewers acknowledge that the technical novelty of the work is limited, but that the question under studying (editing VLMs) is worthwhile and underexplored.

Significance: The paper, while not presenting substantial technical novelty, does present significant and actionable findings for the community by introducing and verifying a method for effective VLM editing across multiple backbones.

Pros:
* First comprehensive study decoupling modality-specific impacts in VLM editing, revealing critical insights.
* Provides a much-needed exploration of editing in multimodal models, an area that lags behind LLM editing research.

Cons:
* The conceptual novelty may feel limited, as the work extends existing ideas (like adapter-based editing and modality-specific routing) rather than introducing a fundamentally new editing paradigm.
Decision: Accept