Learning Where to Edit Vision Transformers
Authors: Yunqiao Yang
Long-Kai Huang
Shengzhuang Chen
Kede Ma
Ying Wei
Venue: NeurIPS 2024 poster (2024)
Keywords: Model Editing
Vision Transformer
Primary area: other

Abstract: Model editing aims to data-efficiently correct predictive errors of large pre-trained models while ensuring generalization to neighboring failures and locality to minimize unintended effects on unrelated examples. While significant progress has been made in editing Transformer-based large language models, effective strategies for editing vision Transformers (ViTs) in computer vision remain largely untapped. In this paper, we take initial steps towards correcting predictive errors of ViTs, particularly those arising from subpopulation shifts. Taking a locate-then-edit approach, we first address the ``where-to-edit`` challenge by meta-learning a hypernetwork on CutMix-augmented data generated for editing reliability. This trained hypernetwork produces generalizable binary masks that identify a sparse subset of structured model parameters,  responsive to real-world failure samples. Afterward, we solve the ``how-to-edit`` problem by simply fine-tuning the identified parameters using a variant of gradient descent to achieve successful edits. To validate our method, we construct an editing benchmark that introduces subpopulation shifts towards natural underrepresented images and AI-generated images, thereby revealing the limitations of pre-trained ViTs for object recognition. Our approach not only achieves superior performance on the proposed benchmark but also allows for adjustable trade-offs between generalization and locality. Our code is available at https://github.com/hustyyq/Where-to-Edit.

Learning Where to Edit Vision Transformers
Yunqiao Yang!*
Long-Kai Huang²t
Shengzhuang Chen'
Kede Ma²
Ying Weist
City University of Hong Kong Tencent Al Lab 37hejiang University
Cygyang. cs, szchen9-c) htt.amttt.............ea..aa.ae. edu.y..eu.... kede, wwwmmtau.uu...ee edu.
alongkai@gmal com ying. Lang.weißzjp wei@zju. edu.
Abstract
Model editing aims to data-efficiently correct predictive emrors of large pre-trained models while ensuring generalization to neighboring failures and locality to min- imize unintended effects on unrelated examples. While significant progress has been made in editing Transformer-basee large language models, effective strategies for editing vision Transformers (ViTs) in computer vision remain largely untapped. In this paper, take initial steps towards correcting predictive errors of ViTs, particularly those arising from approach, we first address the subpopulation shifts. Taking a bocate-then-edit challenge by meta-learning hyper- network on CutMix -augme--e data generated for editing reliability. This trained hyperneworr produces generalizable binary masks that identify sparse subset of structured model parameters, responsive to real-world failure samples. Afterward, we solve the "how-to-edit" problem by simply fine-tuning the identified parameters using a variant of gradient descent to achieve successful edits. To validate our method, construct an editing benchmark that introduces subpopulation shifts towards natural underrepresented images and Al-generated images, thereby reveal- ing the limitations of pre- rrettrined ViT' for object recognition. Our approach not only achieves superior performance on the proposed benchmark but also allows for adjustable trade-offs between generalization and locality. Our code available htttps://github wwt.aal..oocuuuuaeeeeeee http://sssyyy///.c
1 Introduction
In many scientific and engineering disciplines, computational models serve as approximations of complex real-world phenomena. As consequence, they are inherently prone predictive errors, aptly encapsulated by George Box's 's adage: "All models are wrong, but some are useful." Model editing [55
2 40 has emerged promising technique to make (large) pre-trained models more useful by enabling targeted updates to model behavior on specific inputs or tasks in a data-efficient manner without pre-training again from scratch. An ideal model editing method should satisfy three major desiderata [ 61:: 1) reliability, ensuring the model behavior is effectively updated for the current sample; 2) generalization, so that the changes extend neighboring samples; and 3) locality, meaning the edit should have minimal impact on the model behavior on unrelated samples.
Model editing has allowed many fascinating applications, including error correction, factual knowl- edge update, bias mitigation, policy compliance, and personalization, though most of them have predominantly been within large language models (LLMMs) [5
i 12 in the natural language process- ing (NLP) community ( [40]. With the enormous and often inaccessible pre-training datasets and the ever-growing model sizes that make retraining computationally demanding, the need for effectively
*Part the work was done when the author interned Tencent Lab. *Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

editing computer vision (CV) models is also becoming urgent. Adapting model editing techniques from NLP CV is non-trivial and presents unique challenges. From the data perspective, NLP deals with one-dimensional, discrete signals that are highly semantic and information-dense, whereas CV requires processing nigh-dimensional continuous sensor data that is spatially redundant, From the model perspective, lots of model editing methods in NLP are specially designed for LLMs with unidi- rectional (i.e., autoregressive) attention, such as GPT-3 [5] and GPT-4 M. In contrast, CV models have primarily been based on convolutional networks B3 52 24,, with more recent implementations using vision Transformers (ViTs) Ii 351 that otherwise employ bidirectional attention. These differences in data formats and model structures make targeted edits more challenging to implement in CV models, and when such edits are achieved, they often result in suboptimal performance.
In this paper, we take initial steps towards editing pre-trained ViTs for object recognition [O], aiming to correct predictive errors without the need for costly and time-consuming retraining. Specifically, we take a locate-then-eee approach, which breaks down model editing into two key subproblems: where-to-edit and how-to-edit. Moreover, we prioritize learning where to edit rather than how to edit to facilitate a simpler yet better trade-off between generalization and locality, without needing to store previously trained data.
For the where-to-edit phase, we first narrow the editing scope using a greedy search- heuristic. Next, inspired by the proven effectiveness of meta-learning [4] optimizing training strategies for individual samples, we meta- meta-train hypernetwork to generate binary task, indicating which parameters are critical for the editing sample. To address the issue of limited data, the hypernetwork is trained solely using pseudo-samples, each comprising a natural image paired with its CutMix version (62 (see Fig. D The optimization objective is to align the predicted probability distribution of the CutMix sample to that of the original. By controlling the sizes of patches used in CutMix and randomly varying their locations, we simulate distribution shifts in backgrounds, contextual objects, and object attributes, creating opportunities to learn generalizable binary masks that effectively respond to real-world failures. Additionally, we apply a sparsity constraint to the binary masks, acting as indirect, data-free regularizer to promote locality. Once the where-to-edit problem is solved, the how-to-edit phase becomes straightforward: we simply fine-tune the selected parameters using a variant of gradient descent apply targeted edits.
To validate our method, we construct an editing benehmark that exposes the weaknesses of pre- trained ViTs by introducing two types of subpopulation shifts. The first is natural subpopulation shift 45 50,, with underrepresented natural images of certain categories efficiently identified by the maximum discrepancy (MAD) competition [58). The second is an artificial subpopulation shift, introduced by synthesized images from high-quality text-to-image generative models like Stable Diffusion 46)
In summary, our key contributions are as follows:
a first-of-its-king model editing method for pre-trained ViTs that leverages meta-learning to prioritize the where-to-edit phase;
an editing benchmark that provides valuable resources for future model editing research in CV;
an extensive experimental demonstration that our method achieves the best Pareto front between generalization and locality on the proposed benchmark, while offering flexible trade-offs in the how-to-edit phase.
2
Related Work
In this section, we provide a brief overview of current model editing methods in NL.P and CV.
2.1 Model Editing NLP
Memory-based Methods rely on external mechanisms, such as wrappers 41 and caches [1 to store factual updates without modifying the internal model parameters. A common theme these studies is the use of a gating mechanism to determine whether test sample falls within the editing scope; if so, the base model behavior overridden. For instance, SERAC [41] and GRACE [21] employ a scope classifier as form of hard gating, while Murty et al.
42 utilized soft gating
function, allowing for smoother integration. More recent approaches like IKE 63 and MeLLo [64
alter the input prompts of an LLM for knowledge update, where the gating mechanism is implicitly

---

embedded within the LLM itself. Generally. memory-based methods offer advantages such as non- destructive updates, modularity, and suitability for continual and few-shot learning settings. However, they face scalability issues when handling a large number of edits. Additionally, the editing success heavily depends on the accuracy of the gating mechanism.
Parameter-basee based Methods modity the internal model parameters, which offers a more fine-grained approach to editing. These methods can roughly be categorized into two subgroups: locate-then-edit approaches and hypernetwork-baase approaches. Locate-then-edit methods focus on identifying a subset of key parameters for editing. For instance, ROME 38 MEMIT 39 and MEMIT [20] leverage causal mediation analysis (i.e., representation denoising) to locate hidden states (i.e., inter- mediate representations, not model parameters) responsible for knowledge storage. The theory of associative memory [2 is then applied to transfer the state localization results to model parameters. Recent studies [22 suggest that ledge localization may not reliably inform successful edits. Furthermore, the very notion that knowledge can be localized may be inherently flawed, as factual information in L.LMs may be encoded in a distributed manner 40). Single-step integrated gradient across multiple editing samples 88 60) is another commonly used statistic for localization. Here, we adopt a more principled meta-learning strategy to locate key parameters, using multi-step gradient information that more accurately captures the changes in model behavior.
Hypernetwork- baaee methods, such KnowledgeEditor (9. MEND 40 and MALMEN [54]. train an external network to directly generate parameter updates for the editing sample, which is represented by either feedforward feature representation Localization techniques can be applied beforehand to restrict the functional space of the hypernetwork. Existing hypernetwork-basee
backward gradient decomposition 44)]
methods emphasize the how-to-edit aspect but treat the where-to- edit superficially, and often result in suboptimal performance, especially when adapting to CV applications. In contrast, our method prioritizes learning where to edit, achieving a better balance between generalization and locality.
2.2 Model Editing in CV
Limited research on model editing has been conducted in CV. Bau et al. (3 took locate- seattetteen-eee approach to rewrite generative adversarial networks. Santurkar et al. [49] adapted this method for editing image classifiers based on convolutional networks by mapping the representation of the new visual concept to that of a previously learned concept. However, this approach requires prior knowledge of the new visual concept, its location within the image, and the specific target concept for correction. In practical applications, such detailed information may not always be available. In contrast, our method relaxes all these assumptions and is one of the first applied to ViTs.
3 Learning Where to Edit ViTs
In this section, we first present the preliminaries, followed by a detailed description of the proposed method for learning where to edit ViTs. The system diagram of our method is shown in Fig.
3.1 Preliminaries
Given a base computational model f(=:0) X + V. parameterized by 0, model editing aims to modify the model behavior for specific inputs E X' (or regions of the input space, S C X') while keeping its overall performance intact. Denote the post-edited model as f (:; (())))) where g(e) represents the updated parameter vector after editing. Typically, f (::((e) is evaluated based on three main criteria: reliability, generalization, and locality.
Problem Formulation
* Reliability: For any editing sample (x, v). the edited model o (x; B(e)) y.
Generalization: For any neighboring sample (a', ./) E (xx.g) o (z':000) = y even
(x',y") is not directly used in the editing process.
We slightly abuse the notation g(e) to encompass any possible modifications, including those by memory- based methods.
'Conceptually, in high-level vision. two images are considered neighbors. if they are semantically similar, such as belonging to the same category subpopulation.
3

---

Hypernetwork
VIT festure
Leamable tokens
CutMix
extractor
VII as base model
Embedding
Transtormer encoder
1x
Inner-loop loss KL(x,pyyk<' 2 (
MSA
Projection bead
FFN
Concatenation
Outer loop loss KI. (I,,,,(y)>'' x ()) )0)) +Almille
Classification head
Binarization
Frozen paramicioos
Editablo paramoters
Figure 1: System diagram of the proposed model editing method.
Locality: For any sample (a',y/) & Wxx,, v), the model behavior should remain unchanged, i.e., (()) = f(x';0).
An ideal model editing method shall ensure reliable edits while balancing generalization and locality effectively. As initial model editing attempts in CV, limit our scope Singleexamppe editing.
Vision Transformers A VIT feature extractor, denoted by e(:: Φ) with parameter vector ø, consists of a linear embedding layer followed by L attention blocks. Each block is composed of multiheaded self-attention (MSA) layer and a feedforward neural network (FFN). The FFN, which underpins most model editing methods, including ours, comprises two fully-connected (FC) layers: FFN(2) GELU(>W + 6) + bb. Here, W E I N and W' E R N- are weight matrices, where Nn. denotes the intermediate dimension. b N.. and D' NXX are bias terms. The activation function GELU(() is the Gaussian error linear unit [26].
An input image is first partitioned into M non-overlapping. fixed-size patches, each linearly embed- ded in an N-dimensional feature space together with a class token [cls], yielding a concatenation of patch embeddings of size (M 1) x N. These embeddings are processed through the I. attention blocks for feature extraction. A linear classification head, h(-). maps the extracted features to probability distribution over classes in Y', represented as p(y c|a;; Φ) he(e(x; Φ)), where e V. For notation simplicity, we omit the parameters in the classification head fa(-), as they constitute only small fraction of the total parameters and generally frozen during model editing.
3.2 Model Editing Training Time: Where-to-edit
The simplest way of editing a VIT is through vanilla fine-tuning, which involves updating all model parameters. However, modern ViTs have millions to billions of parameters, and fine-tuning on single sample (x, y) can lead to overfitting, while incurring substantial computation costs. To overcome these, prior research g 23 first identifies subset of key parameters, followed by editing:
Φ + + Aø,
where binary mask of the same dimension Φ, Ad represents the parameter update, and is the Hadamard product.
Prevailing localization strategies in NLP rely casual mediation analysis [38] integrated gradi- ents (8), or pure heuristic methods (28), which may not be ideal for ViTs due to differences in data modalities and model architectures. In this work, follow the locate-the-ede approach, and decompose model editing into two subproblems: where-to-edit (i.e., computing m) and how-to-edit (i.e., computing Aø), with focus where-to-edit. Drawing inspiration from the demonstrated success of leta-lea 34 [4! tailoring training strategies for individual samples, we meta-train hypernetwork generate the binary mask vn for each editing sample.
Meta-learning 34 also known as learning-to-lean involves training models on collection of training episodes to enable effective generalization and adaptation to novel, unseen episodes. In

---

as the our context, a training episode corresponds to a single editing example. We employ optimization- based meta-learning approaches [4 44], framing where-to edit as a bi-level optimization problem.
In the inner loop, key parameters, indicated by mi are updated for the editing sample by optimizing reliability loss via gradient-descen over T iterations. In the outer loop, the hypernetwork g(: parameterized by sp, is refined to generate m. Mathematically, we have
L {x:
Mi v()) + Almooloo
min F
s.t.
m g 4)
(2)
At(t) = A - VV -oVo((E.): (I.
,E (1,
60 (0 + m A EE (1.2......
where (z, y) is the editing sample. ( is the updated parameter after T iterations of inner-loop optimization, and g( denotes the pre-trained parameters of the base model as initialization. The term Adtt) is the parameter update after the I-th iteration, with Agj(0) 0. The loss function L (z, ys (())) measures the reliability of the edit. To encourage sparsity in the binary mask m, add an fo-norm term in the outer-loop objective, which acts as an indirect, data-free regularizer to encourage locality. The scalar À controls the trade-off between the two terms. In our implementation, the hypernetwork takes the ltst- features corresponding to the [cls] token from the ViT feature extractor (:: o:0) as input, i.e., m = g (e (2; (z: ((0)) :4)).
3.3 Optimization Challenges
Despite mathematical elegance, solving the bi-level optimization problem in (2) presents three challenges. First, meta-training the hypernetwork necessitates a sizable of high-quality editing samples, which are expensive and time-consuming to collect in practice. To address this, we generate pscudo-samples using a data augmentation technique known as CutMix [62]. Second, identifying key parameters within the entirety of the ViT presents a vast search space. This combinatorial complexity not only introduces unacceptable computational costs but also makes the localization of key parameters challenging endeavor [36 51]. To alleviate this, we shrink the editing scope based on a greedy beacch-based heuristic. Third, generating a binary mask typically involves binarization operation in g)) which produces zero gradients almost everywhere and is thus ineffective in optimizing. To resolve this, we use a gradient-frient friendly approximation to binarization.
Pseudo-sample Generation We employ CutMix 662 to generate pseudo-samples for editing. Specif- ically, given natural image x', we apply CutMix (62 randomly overlay a small patch from another irrelevant image onto x', producing a pscudo-sample x. This patch-based perturbation tends to alter the predicted probability distribution, resulting in p (y = c/z; g(0)) * p( (y- clI': for e E Y. This motivates us to instantiate the reliability loss & (1, y; (01)) in Problem 2 Kullback-Leiblrr (KL) divergence
27 between p (((0)) and p (y/r; (b()):
(y cl2'; 2f(0))
e (1: (vlat: :)()))
Σ nEY
cl' "r(y=daa' log
Ep(r
(3)
p = c/2; 2b(1)
where {p (u/az'; ((00))} } is treated as the soft ground-truth label.
Editing Scope Shrinkage Previous studies [38 40 have suggested that modifying FFNs within Transformer is more effective for achieving successful edits 117 18). For example, MEND[40] focuses on editing the last three FFNs, while ROME [38) targets the middle FFNs. Here, we conduct similar empirical investigation to identify subset of consecutive FFNs in a ViT, by greedy search for the optimal generalization and locality trade-off. Specifically, we fine-tune ten groups of FFNs (or MSAs) in three consecutive layers [40] of pre-trained ViT/B-16, denoted as {1-3, 2-4, .... 10-12). The editing set comprises 100 predictive failures of the ViT, where volleyball is mistaken for basketball (see Fig. 2a), identified by the MAD competition [58] (see more details Sec. 1.I. The average results across the editing set are shown in Fig. 2b where we see that editing MSAs is not conducive to preserving locality. In contrast, editing the 8-th to 10-th FNNs tends to achieve the best trade-off, which are selected as the default layers for subsequent experiments.
5

---

10-12
Indox three consecutive FFNS
Index three consecutive MSA1
VIT-8/16 FENS
VIT-B/16 MSAs
85
Locality (%)
7-9
7-9
75
4-6
4-6
55 Generalization (96
75
85
(a) Editing samples.
(b) Editing results via vanilla fine-tuning.
Figure 2: The left subfigure shows representative editing examples, highlighting the predictive errors of the base ViT when predicting volleyball basketball. The right subfigure depicts the generalization and locality trade-offss when editing different groups of FFNs or MSAs in the base VIT. is evident that editing the 8-th 10-th FFNs achieves the optimal Pareto front.
To further limit the output space of the hypernetwork, employ structured tuning { selecting specific rows/columns the weight matrices in the FFNs for updating. As suggested [8. we select the weights along the intermediate dimension Nm. which further reduces the output dimension of the hypernetwork to Nm x 6 (i.e,, three FFNs with two FCs each).
Binarization Approximation As special case of quantization in signal processing, binarization can be approximated to enable gradient-bart based training through three main approaches: straight-througs estimation [4]. uniform noise addition 22, and soft-to-hard annealing [30]. Here, fixed parametric sigmoid function with favorable gradient behavior as the approximation:
m Sigmoid (k: m):
where m is a continuous map computed by the hypernetwork right before binarization, and k is hyperparameter that controls the degree which the sigmoid curve approximates the desired binarization operation. Empirically, we set A = 10. We have also experimented with soft-to-haid annealing for k, and observed comparable results. After adopting Eq. {4), we substitute with m and replace the fn-morm with the fnonr in Problem (2) to facilitate gradient-basee optimization.
3.4 Model Editing at Test Time: How-to-edit
At test time, we solve the how-to-edit problem in manner similar to the inner-loop optimization. The two minor differences lie the loss function and the binarization operation.
At test time, we are provided with the editing sample x and its ground label Therefore, the KI. divergence during training reduces the cross-entron cross-entropy loss during testing:
(z, yi 2 ((1) -
c] log (p = ci,; d
(5)
Also, directly employ the threshold-based binarization without approximation obtain
m; 2 p m: <
m; ((,))
(6)
where is the positional index, and p is hyperparameter that can be adjusted for different model editing applications. When is set to zero, all parameters in the selected FFNs are updated with improved reliability. As p increases, fewer parameters are updated, which favors locality.
3.5 Hypernetwork Architecture
Similar to the VIT feature extractor e (-: (0)), the hypernetwork g(:;+p) comprises five attention blocks, FC layer the projection head, and binarization operation. As shown in Fig. T introduce six learnable tokens, each corresponding to an FC layer within the three selected FFNs of the base ViT. These tokens are concatenated with the image features derived from e (;; (()(0)) and serve input to the hypemetwork to compute the binary mask m.

---

Samples seen during pre-training
Error samples during tasting
Natural
Volleyball ball
Baskethall
Volleyball misclassified as basketball
Shovel
Paddle
Shovel misclassified paddle
Figure 3: Visual examples seen by the base ViT/B-16 during pre pre-trainin -re-training, contrasted with visual examples the proposed editing benchmark as predictive errors the hase ViT/B-16.
4 Editing Benchmark with Subpopulation Shifts
In this section, we establish an editing benchmark that exposes failures of the base ViT in object recognition by introducing subpopulation shifts underrepresented natural and Al-generated images.
4.1 Natural Image Subset
To build the natural image subset, we first compile a large dataset of unlabeled images, denoted as U, from Flickr, by leveraging keywords relevant to the object categories in ImageNet-1k [10]. Next, we employ the MAD competition [58] to facilitate failure identification of the base ViT be edited. Under the principle of model falsification as model comparison, MAD chooses to identify images that best distinguish two classifiers, f(-) and f'()), by maximizing their prediction discrepancies. This can be mathematically formulated as
arg arrgmam max (f('', x' D'Eld
where D, f is the set of images that have been identified. d(:;:) the multi- hop distance defined over the WordNet [13] measure prediction discrepancy at a semantic level. Intuitively, if one classifier is weaker, the identified image set 2,, is more likely to include its predictive failures, thereby substantially reducing the human effort for failure identification. Moreover, the "ground-truth" labels for these failures can be first suggested by the stronger model and then verified by two of the authors. To leverage this intuition, we pair our base model (i.e., a VIT/B-16 pre- trained on ImageNet- 1k) with stronger one (i.e., the same VIT/B-16 16 pre-trained using CLIP 43) and fine- tuned on ImageNet), which generally exhibits better generalization unseen data. In total. we collect 354 HAAD-searche natural images, which are partitioned into 16 groups, i.e., D, = (8 1 Ji=1+ based on the predictions by the two models. Each group is named according to the format "prediction of the stronger "odel"""preddcee" of the base model," with the statisties and visual examples given in the Appendix.
4.2 A Al-generated Image Subset
Classifiers pre-trained on natural images often struggle to generalize to Al-generated images [56] 59 To exploit this, we construct an Al-generated image subset containing two groups of images, denoted as D. {S())1 11 17-17- The 17-th group includes 860 images with an art style shift (i.e., oil painting) generated by Textural Inversion [56]. while the 18-th group comprises 1, 092 images with a lighting condition shift (i.e., stage light) produced by PUG (5). Both Textural Inversion
PUG are text-
to-image generators, wherein the "ground-truth" label is embedded in the input text prompt and subsequently verified by two of the authors. Additional details of the Al-generated image subset can be found in the Appendix.
7

---

Natural image subset
Al oil painting subset
stage light subset
85
80
75
80
70
70
65
60
50
40
45
30
70
75
90
95
60
75
80
85
65
70
75
90
95
LR (%)
LR (%6
LR (16)
FT-42
T-Patcher
KN
SPT
ROME
LoRA
KB
MEND
Ouns
Figure 4: Editing results for V the proposed benchmark.
Experiments
In this section, we first describe the experimental setups and then present comparison results on the proposed editing benchmark.
5.1 Experiment Setups
Evaluation Metries Following 29), evaluate all model editing methods on the single-example editing task and compare their performance using three evaluation metrics. The first is the success rate (SR), which indicates the reliability (i.e., accuracy) of the edited model f (::())
SR(f.D,)
1
Σ
I - (2; 0(0) ( (0))0) (x.)))]
(8)
[Drl
(2,))ED,
where D, = Dr UD consists of all MAD-searched and Al-generated images, and we make it explicit the dependence of the updated parameters ((e) on the editing sample (x, v). The second metric is the gemeralization rate (GR), which assesses the accuracy of the edited model on neighboring samples that fall within the editing scope:
GR(f.S) SI(SS) - 1)
Σ (x's')ES (1"))))(("
E
I
I|u- (
(9)
where S denotes of the 18 groups in the proposed editing benchmark. We further average the GR values across all groups as an overall indicator generalization. The third metric is the locality rate (LR), which examines whether the edited model maintains its predictions on unrelated samples outside the editing scope:
Σ Σ (
LR(f.Dr, Dri (x' (x',))ED, (z,0)ED;
(10)
where D; includes out-of-scope images. Using the validation from ImageNet- DD does not adequately examine locality, as the majority are easy samples that lie far from the decision boundary [16]. To more closely examine the adverse effects of model editing, we have carefully curated 2, 071 images near the decision boundary of the base model from the validation sets of ImageNet- 1k (47). ImageNet-R (25], and ImageNet-Sketch (57), whose predictions are more suscep- tible to change. Our selection criteria rely on the predicted probabilities of the pre-trained ViT/B-16 model as follows: 1) the predicted probability for the true label is the highest, and 2) the difference between the top two predicted probabilities is less than 0.05, suggesting a highly ambiguous class. We also employ the GR-LR curve to delineate the generalization and locality trade-off.
Base Models For all model editing methods, we experiment with two ViT backbones, ViT-B/16 and ViT/S-16, 16, both pre-trained on ImageNet-21k and ImageNet-Ik [53 47).

---

Natural image subset
Natural image subset
95.0
90
92.5
e
90.0
GR
FT-4
87.5
70
FT-4
85.0
sample
Random masking
samples
82.5
60
Ours
samples
80.0
78
82
84
88
92
LR (%)
LR (%)
(a) Localization effectiveness.
(h) More editing samples.
Figure 5: Ablation results of the hypernetwork for ViT/B-16.
Competing Methods We compare method with several recent model editing approaches as follows. 1) tinnettnnigg& (FT) updates the 8-th to 10-th FFNs, which have been identified as the most effective layers using greedy search (see Fig. 2) FT-1z [39 incorporates f--norm regularization during fine-tuning. 3) T-Patcher [29] adds and tunes a single neuron in the last FFN. 4) KN 8 and 5) SPT [23] select key parameters based on integrated gradient information. 6) ROME [38 is implemented to adjust the second FC layer of the last FFN by solving a constrained least squares problem. 7) LoRA (28] introduces trainable low-rank matrices update the queries and values of all MSAs. 8) KE (9) and 9) MEND [4)] employ hypernetworks to generate parameter updates for the
last three FFNs. In line with previous work [40] 39 early stopping is applied when the training loss drops below 0.01 or the maximum of 100 editing steps is reached. Detailed implementations of the
competing methods and additional training configurations are provided in the Appendix.
5.2 Main Results
Fig. 4 shows the GR-LR curves for different editing methods applied to ViT-B/16, averaged across 18 groups in the proposed benchmark. We highlight several interesting observations. First, correcting single predictive eтror is generally feasible, as evidenced by a nearly 100% SR for most methods. Second, achieving high levels of generalization and locality simultancously proves to be significant challenge. T-Patcher and ROME utilize previously seen data to maintain locality. Nevertheless, T-Patcher, which relies on an editing scope classifier, exhibits noticeable generalization variability across different editing samples. ROME, being specifically designed for language- based GPT [15]. shows limited promise in generalizing to ViTs. LoRA manages to maintain locality because of its low-rank updates but struggles to generalize. Both KE and MEND exhibit low locality on the MAD-searched natural images and poor generalization to the Al-generated images. Third, our method achieves the new state-of-the-art without relying previously trained data to explicitly enforce locality. Similar conclusions can be drawn for ViT-S/16, shown in the Appendix.
We then evaluate our method across different parameter sparsity levels in the three FFNs from (0.25, 0.50, 0.75, 0.90, 0.95]. corresponding (12.4%, 8.25%, 4. 4.13%, 1.65%, 0.83%) parameters of the entire model, by adjusting in Eq. (6. The competing methoos-FFT- adjusted to comparable levels of parameter sparsity by tuning their respective hyperparameters. Note that our method reduces to FT when p O. The resulting GR-LR curves are shown in Fig. 4 As expected, increasing the parameter sparsity KN, SPT, and our method improves locality at the expense of generalization. Notably, our method achieves the best Pareto front among all methods, which we believe arises from our proposed strategy of learning where to edit towards editing success.
KN, and SPT-are
5.3 Ablation Studies
Localization Effectiveness To substantiate that the effectiveness of our method is indeed due to the successful localization of specific subset of key parameters, rather than merely due to sparsity, we compare the binary masks produced by our hypernetorr to random masks at the same sparsity levels, together with FT-6, and FT-{2- As depicted various regularization levels as fn-mrrm is more effective in zeroing out less important parameters. Applying random masks shows effects akin to FT-,, When the ratio of editing parameters falls below 1.65%, the performance of random masking becomes significantly inferior to method.
Fig. 5a FT-6; generally surpasses FT-&, at

---

1.0
0.9
Image index
2
0.7
0.6
0.5
-0.4
0.3
3 Image index
2
5
(a) Six representative editing examples from three different groups.
(b) Binary mask IoU results hetween pairs samples in (a), indexed column-majje order.
Figure 6: Mask specificity results.
Mask Specificity To confirm the specificity of the parameters identified by the hypernetwork for different editing samples, compute the intersection over union (IoU) of the corresponding binary masks at 0.95 sparsity level for samples within and outside the same groups in the natural image subset. Fig. 6b illustrates that the identified parameters demonstrate substantial overlaps for images within the same group and much lower overlaps between images from different groups. These findings support that the hypemettwrk successfully pinpoints key parameters necessary to correct specific errors while effectively excluding parameters associated with other unrelated samples. This learned mask specificity allows method to balance effectively between generalization and locality.
More Editing Samples We further evaluate method when multiple editing samples in the same group (i.e., with similar failure causes) are available. As a straightforward extension, we compute the average of the continuous masks generated from each sample, followed by binarization using Eq. (6). Fig. (ae.55hppp presents the results of using one, two, and three samples for model editing. Remarkably. the editing performance improves with more editing samples, which can be attributed more precise parameter localization as a result of the ensemble of masks.
More Ablation Studies More ablation studies (e.g., the alternative pseudo-sample generation strategy, the sparsity regularization in the outer loop, the gradient step and learning in the inner loop, and the number of attention blocks in the hypernetwork) are in the Appendix.
6 Conclusion and Discussion
We have introduced a model editing method to correct predictive errors in ViTs. Our method prioritizes where-to-edit over how-to-edit by meta-training a hypernetwork identify subset of structured parameters for editing. By applying f--nrrm regularization, our method promotes sparsity in the generated mask, thereby indirectly ensuring locality without needing to retrain on previously used data. Comprehensive tests on the proposed editing benchmark confirm that our method effectively corrects predictive errors in ViT'. Moreover, the introduced edits are not only reliable but also generalize well to neighboring samples, while maintaining high rate of locality.
Our work is among the early endeavors in CV model editing, and it raises several intriguing questions for future research. First, our approach utilizes the CutMix technique [62] to generate cost-cffective pseudo-samples for training, but its effectiveness has only been confirmed empirically. The reasons why the hypernewoor trained on such synthetic data achieves reasonable generalization and the identification of optimal synthetic data generation techniques remain wide open. Second, it would be beneficial to adapt our method to other vision architectures, such convolutional networks or Swin Transformers (35), and extend its application to other vision areas like dense prediction, generative modeling, and multimodal L.Mss. Third, exploring how to apply our method in batch-editing setting represents a promising avenue. In such scenarios, the use of decoupling trick (see more details in the Appendix) may prove essential for effectively reducing computational and memory demands.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.0877 087744 2023.
[2] Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compres- sion. In International Conference on Learning Representationss 2017.
[3] David Bau, Steven Liu, Tongzhou Wang, Jan Zhu, and Antonio Torralba. Rewriting deep generative model. In European Conference on Computer Vision, pages 351-369, 2020.
[4] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:/308. 3432,, 2013.
[5] Florian Bordes, Shashank Shekhar, Mark Ibrahim, Diane Bouchacourt, Pascal Vincent, and Ari Morcos. PUG: Photorealistic and semantically controllable synthetic data for representation learning. In Advances in Neural Information Processing Systems, pages 8072-8081, 2023.
[6] Wray Buntine. Theory refinement Bayesian networks. In Conference Uncertainty in Artificial Intelligence, pages 52-60, 1991.
[7] Jiaxin Chen, Xiao-Ming Wu, Yanke Li, Qimai Li, Li-Ming Zhan, and Fu-lai Chung. A closer look the training strategy for modern meta-learning. In Advances in Neural Information Processing Systems, pages 396-406, 2020.
[8] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained Transformers. In Annual Meeting of the Association for Computational Linguistics, pages 8493-8502, 2022.
[9] Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Empirical Methods in Natural Language Processing, pages 6491-6506, 2021.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[12] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[13] Christiane Fellbaum. WordNet: An Electronic Lexical Database. MIT Press, 1998.
[14] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning, pages 1126-113 1135, 2017.
[15] Luciano Floridi and Massimo Chiriatti. GPT-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681-694, 2020.
[16] Irena Gao, Gabriel Ilharco, Scott Lundberg, and Marco Ribeiro. Adaptive testing of computer vision models. In IEEE International Conference on Computer Vision, pages 4003-401 2023.
[17] Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed- beed-forward layers build predictions by promoting concepts in the vocabulary space. In Empirical Methods in Natural Language Processing, pages 40, 2022.
[18] Mor Geva, Roei Schuster, Jonashann Berant, and Oner Levy. Transformer feed-forwer feedwrd layers are key-value memories. In Empirical Methods Natural Language Processing, pages 5484-5495, 2021.

---

[19] Chong Guo, Michael Lee. Guillaume Leclere, Joel Dapello, Yug Rao, Aleksander Madry, and James Dicarlo. Adversarially trained neural representations are already as robust as biological neural representations. In International Conference on Machine Learning, pages 8072-8081, 2022.
[20] Anshita Gupta, Debanjan Mondal, Akshay Sheshadri, Wenlong Zhao, Xiang Li, Sarah Wiegreffe, and Niket Tandon. Editing common sense in transformers. In Empirical Methods in Natural Language Processing, pages 8214-8232 2023.
[21] Thomas Hartvigsen, Swami Sankaranarayanae Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with grace: Lifelong model editing with discrete key-value adaptors. In Advances in Neural Information Processing Systems, pages 47934-47957 2023.
[22] Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? Surprising differences in causality-basedd localization vs. knowledge editing in language models. In Advances Neural Information Processing Systems, pages 17643-17668, 2023.
[23] Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Sensitivity-aware visual parameter-effcieet fine-tuning. In IEEE International Conference on Computer Vision, pages 1185--1 1835, 2023.
[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[25] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robusmess: A critical analysis of out-of-distribution generalization. In IEEE International Conference on Computer Vision, pages 8340-8344 8349, 2021.
[26] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv preprint 16Xiv:1606.006 2016.
[27] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. ArXiv preprint wxxiv:/503.02533 2015.
[28] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[29] Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. Transformer-patcher. One mistake worth one neuron. In International Conference on Learning Representations, 2023.
[30] Jeong-Hum Hun Jang and Ki-Sang Hong. Binarization of noisy gray-scale character images by thin line modeling, Pattern Recognition, 32(5):743-752, 1999.
[31] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.
[32] Teuvo Kohonen. Correlation matrix memories. IEEE Transactions on Computers, 100(4):353- 359, 1972.
[33] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097-1105, 2012.
[34] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 35006266) 2015.
[35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical vision Transformer using shifted windows. In IEEE International Conference Computer Vision, pages 10012-10022, 2021.
12

---

[36] Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Tine Cheng, and Jian Sun. MetaPruning: Meta learning for automatic neural network channel pruning. In IEEE International Conference on Computer Vision, pages 3296-3305, 2019.
[37] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018
[38] Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations GPT. Advances Neural Information Processing Systems, pages 17359- 17372, 2022.
[39] Kevin Meng, Arna $ Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a Transformer. In International Conference on Learning Representations, 2023.
[40] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher Manning. Fast model editing at scale. In International Conference on Learning Representations, 2022.
[41] Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory-based model editing at scale. In International Conference on Machine Learning, pages 15817-15831, 2022.
[42] Shikhar Murty, Christopher D Manning, Scott Lundberg, and Marco T Ribeiro. Fixing model bugs with natural language patches. In Empirical Methods Natural Language Processing, pages 11600-11611 11613, 2022.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 8748-877 8763, 2021.
[44] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Advances in Neural Information Processing Systems, pages 113-124, 2019.
[45] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize ImageNet? International Conference on Machine Learning, pages 5389-5400, 2019.
[46] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEEE Conference Computer Vision and Puttern Recognition, pages 10684-10695, 2022.
[47] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-252, 2015.
[48] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust ImageNet models transfer better? In Advances in Neural Information Processing Systems, pages 3533-3545, 2020.
[49] Shibani Santurkar, Dimitris Tsipras, Mahalaxmi Elango, David Bau, Antonio Torralba, and Aleksander Madry. Editing a classifier by rewriting its prediction rules. In Advances in Neural Information Processing Systems, pages 23359, 2021.
[50] Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. BREEDS; Benchmarks for sub- population shift. In International Conference on Learning Representations, 2021.
[51] Haopu Shang, Jia-Liang Wu, Wenjing Hong, and Chao Qian. Neural network pruning by cooperative coevolution. In International Joint Conference on Artificial Intelligence, pages 4814-4820, 2022.
13

---

[52] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
[53] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your VIT? Data, augmentation, and regularization in vision Transformers. Transactions on Machine Learning Research, 2022.
[54] Chenmien Tan, Ge Zhang, and Jie Fu. Massive editing for large language models via meta learning. In International Conference Learning Representations, 2024.
[55] Sebastian Thrun and Tom M Mitchell. Learning one more thing. In International Joint Conference on Artificial Intelligence, pages 1217-1223, 1995.
[56] Joshua Vendrow, Saachi Jain, Logan Engstrom, and Aleksander Madry. Dataset interfaces: Diagnosing model failures using controllable counterfactual generation. ArXiv preprint arXiv:230.007865, 2023.
[57] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global represen- tations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages 10506-10518, 2019.
[58] Haotao Wang, Tianlong Chen, Zhangyang Wang, and Kede Ma. I am going mad: Maximum discrepancy competition for comparing classifiers adaptively. In International Conference on Learning Representations, 2020.
[59] Olivia Wiles, Isabela Albuquerque, and Sven Gowal. Discovering bugs in vision models using off-the-shelf image generation and captioning. In Advances in Neural Information Processing Systems Workshop on Machine Learning Safery, 2022.
[60] Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. DEPN: Detecting and editing privacy neurons in pretrained language models. In Empirical Methods in Natural Language Processing, pages 2875-2886, 2023.
[61] Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. In Empirical Methods in Natural Language Processing, pages 10222-10240, 2023.
[62] Sangdoo Yun, Dongyoon Han, S Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization strategy train strong classifiers with localizable features. In International Conference on Computer Vision, pages 6023-6032, 2019.
[63] Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjin Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? Empirical Methods in Natural Language Processing, pages
2023.
[64] Zexuan Zhong. Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. MQuAKE: Assessing knowledge editing in language models via multi-hop hop questions. In Empirical Methods in Natural Language Processing, pages 15686-15702, 2023.
14

---

A More Details about the Editing Benchmark
A.1 Natural Image Subset
Table A: Statistics of the natural image subset. The first column lists identifiers for each object category in ImageNet-1k, Ik. The "Class Name" in the second column is in the format as "prediction by the stronger "preel"-"rrediction by the base model."
Group Identifier
Class Name
Sample Number
890-430
volleeyball-basketbaall cheeseburger-pplte candle-matchstick
123
933-923
133
470-644
113
900-437
water tower-beacon
159
609-586
jeep-half track
410
543-422
dumbbell-barbell
240
879-762
umbrella-restaurant
49
417-865
balloon-toyshop
75
573-751
go-kart-racer
172
880-671
micycle-mouutain bike
149
954-582
banana-grocery store
75
752-890
racket-volleyball
137
640-539
manhole cover-doormat
80
407-654
ambulance-minibur
155
562-975
fountain-lakeside
155
888-718
viaduct-pier
129
We divide the MAD-searcher natural image subset into 16 groups. whose statistics are listed in Table Visual examples in each group are shown in Figs. and These images are sourced from Flickr, prior to the advent Stable Diffusion, and are licensed under creative commons.
A.2 Al-generated Image Subset
We adopt Textural Inversion [56] and PUG [5] to construct the Al-generated image subset, encom- passing the oil painting and stage light shifts, respectively. The statistics are given in Table B
Specific classes in the oil painting subset include stingray, bullfrog, box turtle, garter snake, harvestman, crayfish, hermit crab, mongoose, rhinoceros beetle, weevil, wood rabbit, capuchin, african elephant, breastplate, drumstick, envelope, hand blower, shovel, spatula, syringe, wine bottle, and corn.
Specific classes in the stage light subset include barrel, cofee mug, washer, jack lantern, vase, throne, soccer ball, basketball, car wheel, vacuum, birdhouse, laptop. piano, pool table, carousel, jellyfish, convertible, motor scooter, mask, sewing machine, hay, gasmask, bell pepper, drum, table lamb, backpack, chicken hen, tennis ball, safe, pay phone, cabbage, and pineapple.
Visual examples of the oil painting and stage light images shown in Fig. Cad Fig. respectively.
A.3 Potential Dataset Filtering
Recall that the editing benchmark is designed to challenge the ViT/B-16 model. Thus, it is likely that some images might not induce predictive errors in other base models, which vary terms of training data, model architecture, loss function, and optimization pipeline. For the ViT/S-16 model, the benchmark is subject to an additional filtering process based on its predictions. Consequently, 65% of the natural images and 100% of the Al Al-generated images retained.
15

---

Group 890-430: volleyball-baskee -bakeebball
Group 933-923: cheeseburger-platee
Group 470-644:candle e masthii
Group 900-437: water tower-beacon
Group 609-586: jeep-half track
Group 543-422: Humbbell-barbell
Group 879-762: umbrella .eresturrnt
Group 417-865: balloon-toyshop
Figure A: Visual examples in each group of the natural image subset. Part 1/2.
Table B: Statistics of the Al-generated image subset.
Group
Class Number
Sample Number
oil painting
22
860
stage light 32
1,092
B More Experimental Details
In this section, give more implementation details of the proposed and competing model editing methods. Algorithm presents the pseudo-code of our method.
B.1 More Details of Our Method
Decoupling Trick In meta-lcarning, optimization of the hypernetwork entails differentiating the outer-loop loss with respect to the output of the inner loop (()), and propagating the gradient through the inner-loop optimization to the output of the hypernetwork m (approximated by Eq. 4 and finally to the parameters of the hypernetwork, sp. This extended chain computation not only demands substantial computational resources but also hampers efficient optimization. To mitigate these, we decouple the pathway of hypernetwork optimization from the meta-learning gradient. Specifically, we introduce auxiliary variable m, matching the dimensionality of m to substitute
16

---

Group 573-751: go-kart-racer
Group 880-671: unicycle-mountain bike
Group 954-582: banana banana-grocery store
Group 752-890: racket-volleyball
Group 640-539; manhole cover-doormat
Group 407-654: ambulance-minibus
FIY
Group 562-975;
Group 888-718: fountain-lakesion
Figure B: Visual examples in each group of the natural image subset. Part 2/2.
for the hypernetook's output during bi-level optimization. As result, ((T) is now dependent on m,
rather than m. We first optimize the auxiliary variable:
m
= arg min (z. (r.vi (T) + A||m//1
(11)
Subsequently, n directs the parameter optimization the hypernetwork using the element-wise KL divergence averaged across all positions:
Σ KL (g: (c(2; :40 .m²))
4 arg min 1
(12)
dim(m=)
where is the positional index and dim(m") = Nm x 6 in our implementation.
Pseudo-sample Generation When applying CutMix, we vary the sizes of the pasted patches from 48 48 128 128, ensuring the preservation of the primary structural and textural details in the original images, which are 224 x 224 in size.
Hypernetwork Architecture We design the hypernetwork to mirror the architecture of its corre- sponding base model (i.e., V6T/B-16 or ViT/S-16), with the same input and intermediate dimensions. Nevertheless, reduce the number of attention blocks to five.
17

---

corn
rhinoceros beetle
capuchin
Vais
breastplate
shovel
wine bottle
Figure C: Visual examples the Al-generated oil painting images.
coffee mug
washer
Figure Visual examples of the Al-generated stage light images.
Hyperparameter Configuration We set the learning rate in the inner loop 0.001, and perform gradient descent for five steps (i.e., T= 5). In the outer loop, we apply the Adam optimizer with learning rate of 0.1 to optimize m from random initialization for total of ten steps. For the hypernetwork optimization, RMSProF is utilized with a learning rate of 10-4, minibatch size of eight, and a maximum iteration number of 7, 000. Training a hypernetwork for the base ViT/B-16 takes approximately 9 hours on single RTX A6000 GPU (48G).
slidessssssse_eeeeeeeeee wec.._ec.ppp... .df
18

---

Algorithm Hypernetwork Meta-training via Standard Implementation Decoupling Trick
Require:
Hypernetwork, g(: y)); ViT feature extractor, e (+ (00)); CutMix dataset, B; training itera- tion, MaxIter; inner-loop learning rate, inner-loop step, 7: outer-loop hypernetwork learning rate, 33 outer-loop nn learning rate, 7 outer-loop step, MaxOuterIter; trade-off parameter, À
1: Randomly initialize
2: for Mainlter = to Maxiter do
3: Create a CutMix dataset B from ImageNet-Ik
4: for (x', x) do
5: Calculate = g (e (z; ;4p)
II Approximated by Eq.
6: Set (g =
7: fort
do
8:
Ad = Ao(t-) a Vo (z, (vlæ'; (v) :
9:
6 = x( m @ Ap(t)
10:
end for
o. [e 12: Randomly initialize mn.
11:
o 4- 1 - v.
(x., (ulz's (f ) A||m|l
for Outerlter
MaxOuterlter do
14:
for t t= 1 to l" do
(o Ad a Vol (z, (y)z' ((0) (((-1))
16:
(( = 0 + m @ Ae(t)
17:
end for
18:
i f- m 7 vi [ (x, (y|z'; (000)); ()(2) + {|[mi|li]
19:
end for
20:
end for
ạ 4- p- BV KL (m, m)
22:
end for
B.2 Implementation Details of Competing Methods
For methods that involve updating the base model parameters through backpropagationn-nccudingg FT, FT-C2, KN [ SPT [23], and our method we follow 99] and adopt RMSProp as the optimizer,
where the learning rate is set to 2 x 10 for ViT/B-16 and 10 for ViT/S- 16, respectively.
T-Patcher 29 adds one neuron in the last FFN, together with a trainable multiplier initialized as 10. The new parameters are optimized using Adam with learning rate of 5 x 10-3.
ROME 38
employs Adam with learning rate of 0.01 to obtain the target hidden representations of the last FFN, and then solves a constrained least squares problem to update the second FC layer.
We follow the default setting in LoRA (28), adding learnable matrices with rank of eight. These low-rank matrices are optimized by Adam with a learning rate of 10-4
For KE [9 and MEND 19. we adhere to their training protocols to edit the six FC layers within the last three FFNs. The hypernetworks are treia-trained on editing samples sourced from ImageNet- 1k to alter the base model's 's predictions to match the top-k randomly selected classes. The optimizer is Adam with learning rate of 10-5
C More Experimental Results
C.1 More Editing Results for ViT/B-16
In the main paper, we report the averaged editing results for ViT/B-16 across the sixteen groups in the natural image subset. Here, we further report the editing results on each group in Fig, E
19

---

Goaa
Ghoug
Croup
190
108
199
CR (1)
1R
Coopp 685-590
Group 545-422
Croup 879-332
Croup 417-395
CDR 130
82
25
(N)
LR 1711
12 (50
333-551
Goop 204-522
132-000
190
100
100
LR
LR
Ohapp 64B-399
Churp 4714
Creup 360-875
Creug 365-718
5
(5)
1..
Figure E: Editing results for V1 on the sixteen groups in the natural image subset.
Natural image subset
Al oil painting subset
Al stage light subset
100
90
90
90
80
GR (%)
70
70
60
50
GR
60
50
60
40
40
30
30
20
20
60
70
80
90
100
50
60
70
90
100
60
90
100
LR (%)
LR (%)
LR (%)
FT-42
T-Patcher
KN
SPT
ROME
LoRA
KE
MEND
Ours
Figure F: Editing results for ViT/S- 16 on the proposed benchmark.
C.2 Editing Results for VIT/S-16
Fig. presents the editing outcomes for ViT/S-16, where our method continues to exhibit the opti- mal cineration-localityy trade-off, demonstrating its adaptability across various model architectures.
Meanwhile, Fig. G presents the editing results on each group in the natural image subset.
C.3 More Analysis
We present the training curves the hypernetwork Fig. H We find that the mask sparsity increases rapidly at the beginning of training from 0.0 to 0.86, which poses challenges for successful edits. As training progresses, the mask sparsity stabilizes while the KI. divergence decreases. This suggests that the hypernetwork has effectively located key parameters relevant to successful edits.
20

---

Gooe
Dreay 533423
Grea
CRL
(8)
18,05)
18,50
18.050
Greay S25.223
Oney 411.855
(5U
(R.9))
Lit
Greg 575-55
Crong 195-571
Group Croap 954-882
Creap 753 800
401
CR.P0
300
80
100
300
LK,09)
437-654
Greap 502-975
75
300
100
300
Figure G: Editing results for ViT/S-16 on the sixteen groups in the natural image subset.
890-490
0.55
0.8
0.23
933 923
0.22
045
0.6
900-437
0.40
Sparsir
Mask spunity
609-386
0.35
0.0
Outer loop KL divergenco
573-751
00.25
407 07-654
0.20
590-430
478444
900-477
379-663
373,281
404454
0.0
0.16
2k lierations
Grops
Figure Binary mask IoU results for samples among eight groups of the natural image subset.
Figure H: Training curves of the hypernetwork.
C.4 Ablation Studies
Mask Specificity We further compute the averaged IoU results of the binary masks the 0.95 sparsity level for editing samples among eight groups in the natural image subset. The results in Fig. [ show that the identified parameters exhibit substantial overlaps for samples within the same group and much lower overlaps for samples from different groups
Alternative Strategy for Pseudo-sample Generation We examine another more computationally expensive pseudo-sample generation strategy, i.e., PGD [37], which has been validated to capture diverse distribution variations [9 48 Given natural image x' with the label y' in the pre-training set, we apply PGD 37 on x' to obtain the pseudo-sample x with the prediction different from g'. We set the number of attack steps 10 with step size of 2/255, under the feasible set of
21

---

Natural image subset
Al oil painting subset
Al stage light subset
95.0
77.5
92.3
73.0
72.5
87.5
85.0
82.5
65.0
80.0
62.5
77.5
60.0
80
82
86
88
90
76
78
80
65.0 67.5 70.0 72.5 75.0 77.5 80.0
(96)
(%)
LRR (16)
CutMix
PGD
Figure J: Editing results for ViT/B-16 on the proposed benchmark, using the hypernetworks meta- trained by two different sample generation approaches.
V6 on 954-582
VIT.B- 933 933-923
97.5
95.0
92.5
85
90.0
GR
87.5
0.2
0.2
85.0
1.0
1.0
82.5
5.0
5.0
75
80.0
80
84
86
90
LR (%)
(%)
Figure Ablation results of the hyperparameter in the outer-loop optimization of Problem
ViT/B-16 890-430
VIT/B-16 752-890
97.5
97.5
95.0
92.5
92.5
90.0
RR
90.0
87.5
87.5
85.0
85.0
10
82.5
10
77.5
80.0
82.5
85.0
87.5
90.0
92.5
78
80
82
84
86
88
90
LR (%))
LR
(%)
Figure L: Ablation results the gradient step T in the inner loop.
VTT/B- 16 890-430
752-890
97.5
95.0
92.5
GR (%)
92.5
90.0
90.0
10
87.5
10-4
87.5
10
85.0
10
85.0
10:2
82.5
10
77.5
80.0
82.5
85.0
87.5
92.5
78
80
82
84
86
88
LR (%))
LR (%)
Figure M: Ablation results of the learning rate in the inner loop.
(,, 8/255.. During training, we employ the cross-entropy loss & ( ((1) to correct the prediction of x.
22

---

VTT/B-16 890-430
VIT/B-16 on
98
96
GR (%)
94
92
GR
One block
94
One block
90
Three blocks
Three blocks
88
Five blocks
Five blocks
82
86
90
92
88
90
92
LR (%)
LR (%)
Figure N: Ablation results of the number of attention blocks in the hypernetwork.
Fig. shows the editing results of two hypernetworks meta-trained using the two different pseudo- sample generation approaches. Remarkably, the simple CutMix rivals PGD in simulating distribution shifts, even in the two Al-generated image subsets.
Sparsity Regularization in the Outer Loop In the outer loop, we introduce trade-off hyperpa- rameter, to balance the reliability objective with the sparsity regularizer. Here, we explore the impact of and observe that the sensitivity of hypernetwork to this trade-off parameter is minimal, as shown in Fig.
Gradient Step in the Inner Loop For the gradient step, T. in the inner loop, we test values of (1,5, 103. The performance of ViT/B-16 for each setting is illustrated in Fig. L where we find that one gradient step yields slightly inferior results compared more steps. Five and ten steps perform similarly, yet ten steps have greater training costs. Thus, we opt for five gradient steps as the default.
Learning Rate in the Inner Loop We explore the impact of the learning rate in the inner loop with values from {10-1, 1003 10- The editing results shown in Fig. Mindicate that a lower learning rate (i.e., 10 4) exhibits slightly inferior performance than larger learning rate. This may arise because a lower learning rate results in minimal updates to the base model within five gradient steps, thereby ineffective guiding the hypernetwork training-
We additionally conduct ablative experiments to evaluate the impact of the number of attention blocks in the hypernetwork. We test values of {1,3, 5), and the editing performance for ViT/B-16 is illustrated in Fig. N where we find that small hypernetwork can achieve comparable performance to larger hypernetworks. Decreasing the number of attention blocks in the hypernetwork from five to three, and to one, does not incur noticeable performance drop.
Number of Attention Blocks
D Limitations
See the Conclusion and Discussion section the main text.
E Broader Impact
Model editing has a broad impact by accelerating innovation in AI development through rapid iterations and refinements without extensive retraining, thus conserving resources and reducing environmental impact. The proposed method enables error correction of CV models, thereby enhanc- ing adaptability and accessibility. We believe our method has great potential in addressing ethical concerns by mitigating biases and improving fairness CV applications, while also increasing the robustness of CV systems against security threats like adversarial attacks.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes],
Justification:
Limitations are discussed in the Appendix D.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Implementation details are provided in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
Our code is available at
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee 7/nips.com http://uall//uuuuddooooo//aaaaaa wwssinicccccyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Details are provided the Appendix A and B. Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Fine-tuning specific example is determinative when additional ran- domly initialized parameters are introduced. In this case, there no associated error bar.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computer resources are provided in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://pp.....eo....oo//oo////aa
Answer: [Yes]
Justification: The authors conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: appendix E. Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system Iearns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: Yes]
Justification: For the 2354 images scraped from the Internet, check the images manually to ensure the safety.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The authors cite the original paper that produced the dataset used in this paper. The Internet-searched images from the website are used under the copyright, which is stated in Appendix A.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
28

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: The paper does not release new assets before acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [No]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "riOeIkKXgU", "reviewer": "Reviewer_wreN", "summary": "The paper investigates a novel method for editing Vision Transformers (ViTs) to enhance their performance by rectifying predictive errors. Specifically, it proposes training an additional ViT to locate specific rows in the weight matrices of several feedforward network (FFN) layers for efficient fine-tuning. The authors evaluate their approach using benchmarks such as MAD-searched natural images and AI-generated image datasets. Compared with several state-of-the-art (SOTA) baselines, this work achieves the best locality/generalization tradeoff, offering flexible editing choices and accurate localization of where to edit.", "questions": "1. For the comparison with FT-L2, is it possible to compare with FT-L1 or apply L2 in the proposed method to have a clearer comparison on the regularization term?\n2. Can the authors provide more detail on the computational complexity and resource requirements of their method? How does the additional complexity of the hypernet compare to the saving in fine-tuning?\n3. In line 150, is there a detailed description of the challenge that the authors focus on? The flow seems to be interrupted here.", "limitations": "Limitations are addressed in the originality and quality in the weakness section.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality: \n1. The paper introduces a novel approach to model editing in vision tasks, which is an emerging area with significant potential.\n2. It combines existing techniques in a new way to address specific challenges in model performance enhancement.\n3. The method's ability to identify and edit specific parts of models is innovative and could be valuable for various applications.\n\nQuality: \nThe submission is technically sound with comprehensive question-raising and experimental support.\n\nClarity: \nThe paper is well-written and organized, making it easy to follow the methodology and results.\n\nSignificance:\n1. The results are important as they demonstrate a new way to enhance vision models, which could have significant implications for both research and practical applications.\n2. The method addresses a challenging task and advances the state of the art in model editing and vision tasks.", "weaknesses": "Originality: \nThe main objective, learning where to edit, is not a brand-new idea in NLP. The novelty could be better emphasized by comparing more thoroughly with recent advancements in model editing and vision tasks.\n\nQuality:\n1. The application of the proposed method is somewhat limited to plain ViTs. It would be interesting to see the application to hierarchical ViTs like Swin and PVT.\n2. The evaluation is somewhat limited to specific datasets. It would be better to explore other datasets like CUB-200-2011, NABirds, Oxford Flowers, Stanford Dogs, and Stanford Cars\n\nClarity: \nNo further issues.\n\nSignificance:\nThe impact of the work would be more convincing with broader validation across diverse datasets and tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive comments on this paper. We detail our response below point by point. Please kindly let us know if our response addresses the issues you raised in this paper. \n\n##### [W1] Application to hierarchical ViTs\n> We greatly appreciate the reviewer's suggestion to test our method with different ViT architectures beyond the plain ViTs. We note that our approach is designed to be **broadly applicable across various ViT architectures**, including hierarchical ViTs. This is because our method meta-learns the optimal locations for editing within the parameter space of any given pre-trained ViT, **without imposing strict assumptions on the specific architecture of the models**. To verify our claim:\n> - We follow the reviewer's suggestion and apply our method to a 1-billion-parameter SwinV2-g, which is self-supervised pre-trained via SimMIM [1]. \n> - As shown in **Fig. r1** of the rebuttal PDF, **our method achieves the best Pareto front** when editing two groups of the natural dataset. \n> - Due to the time limit and the model's size, we only included the most recent editing methods as baselines for comparison, but we will include all baseline methods and their results in our revised manuscript.\n\n##### [W2] Application to more datasets\n\n> We follow the reviewer's suggestion and conduct more editing experiments on the Stanford Cars dataset. Experimental results in **Fig. r4** of the rebuttal PDF **show the superiority of our method over baselines** on this fine-grained classification dataset. The **details of the experiment setup** are as follow:\n> \n> - Following [2], we first train a classification head by linear probing on the Stanford Cars training set and evaluate it on the Stanford Cars testing set, achieving an accuracy of 50.69%. \n> - To construct the editing dataset, we collect incorrectly predicted images from the Stanford Cars testing set and group them by their labels. For each error group, one image is used to edit the model, while the remaining images in the same group are used to evaluate the generalization. Additionally, correctly predicted images from the Stanford Cars testing set are used to assess the locality.\n\n##### [Q1] Comparison of the regularization term\n> We follow the reviewer's suggestion and conduct additional ablation studies to provide a detailed comparison of the regularization term. Our observations are as follows:\n> - **FT-L1 generally outperforms FT-L2**: As shown in **Fig. r2(a)** of the rebuttal PDF, FT-L1 generally outperforms FT-L2 across varying levels of regularization strength. This is because **L1 regularization encourages sparse updates**, allowing some pre-trained parameters to remain unchanged. Consequently, it better preserves the locality of the pre-trained model and updates only the most crucial parameters linked to editing success.\n> - **Our method outperforms both FT-L1 and FT-L2**: Despite the effectiveness of L1 regularization, our proposed method outperforms FT-L1 **thanks to our meta-learned hypernetworks which can leverage transferable knowledge (inductive bias) from related training editing episodes** to enhance the editing performance on each test editing sample (see Fig. 5b and lines 361-363). In contrast, FT-L1 performs each edit solely based on that particular test editing sample.\n> - **Locality vs generalization trade-offs for ours + L2 regularization**: To further illustrate the effect of L2 regularization, we conducted editing experiments using our proposed method with an additional L2 term. We explored two sets of balancing ratios between the cross-entropy (CE) loss and L2 regularization: 1:1 and 1:10. As shown in **Fig. r2(b-c)** of the rebuttal PDF, **increasing the strength of L2 regularization enhances locality but compromises generalization performance**.\n\n##### [Q2] Computational efficiency concerns regarding the hypernetwork \n> - Please kindly refer to **[R2] of the global response**.\n\n##### [Q3] Clarification on line 150\n> We would like to clarify that the **central challenge** we wish to address in line 150 pertains to the current localization strategies in model editing, which are **predominantly designed for large language models (LLMs)**, such as GPTs. The strategies such as Knowledge Neurons (KN)and causal tracing (ROME) are **not readily transferable to editing Vision Transformers** due to the inherent differences between LLMs and ViTs: \n> - (i) **Input tokenizations**. LLMs use word embeddings as input tokens, whereas ViTs use cropped patches from images as input tokens.; \n> - (ii) **Attention mechanisms**. Each token in a sequence in GPT only attends to subsequent tokens in the sequence after it, while in ViTs every token attends to all tokens. \n> - (iii) **Hierarchical structures**. Structures of ViT-variants, such as Swin and PVT, possess hierarchical structures not presented in LLMs. \n> \n> As a result, prior localization methods for LLMs yield suboptimal editing results when applied to vision transformers, **[(c.f. Fig. 4, Ours outperforms ROME and KN)]**. Thus, we are highly motivated to design a new localization strategy for pinpointing where to edit in ViTs.\n\n[1] On data scaling in masked image modeling, CVPR 2023.\n\n[2] Visual prompt tuning. ECCV 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z0PsJAexcQ", "reviewer": "Reviewer_91WR", "summary": "The paper explores a model-editing task for ViT that is similar to model-editing in LLMs. For ViT model-editing, the paper proposes a meta-learning-based approach to selecting parameters for fine-tuning updates. The proposed selecting method trains a hyper-network that selects learning parameters using episodes made by CutMix augmentation. The proposed method exhibits superior performance to similar methods on newly made Editing Benchmark.", "questions": "- The authors argue the importance of model editing based on LLM scenarios. But, I think vision domains are different from LLM scenarios. What are the benefits of general model editing in the vision domain? Without performance improvement made by the proposed method, does model editing have a unique role in the vision domain? It is important to evaluate contribution as a new editing benchmark and the first model editing in vision. \n\n- The proposed method requires training a hyper-network with two-loop optimization, which seems to be a high-cost computation. Does this method offer any computational cost advantages compared to other existing methods?\n\n- Can hyper-network be used to select parameters for another pre-trained network?\n\n- I think hyper-network training with multiple episodes is a significant advantage for a fair comparison. Do other methods also use similar processes?\n\n- The proposed method uses 5 transformer blocks to select just 3 FFNs. It is not parameter efficient and can't be expanded to large-scale networks over a billion scales. Is there any way to expand the proposed method for larger select space?", "limitations": "I didn't find a limitation section in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Exploring model editing for ViT with a meta-learning approach is new and interesting.\n- The paper proposes a new benchmark for model editing, including data set and evaluation metric.", "weaknesses": "- Explanation and justification for the necessity of model editing in the vision domain are not enough.\n  - Although model editing is a powerful and important tool for LLM tuning, it doesn't explain the need for model editing in vision models.\n  - Vision models have relatively small parameters, and FLOPs are considered to be an important cost rather than memory and the number of parameters. Model editing might be effective in reducing parameters and memory, but I'm not sure it is also beneficial in the vision domain.\n- Hypernetwork requires too high cost to select parameters for model editing.\n  - To the best of my knowledge, the major goal of model editing is to reduce costs for full fine-tuning like LoRA. But, the proposed method requires huge computation, and the training process might be larger than full finetuning.\n  - For 12 blocks ViT-B, the proposed method trains a hyper-network with 5 transformer blocks using multiple training episodes with inner- and outer-loop optimization, which might require huge computation costs compared to the original network.\n  - The hyper-network is trained only for a single pre-trained network. Thus, it is required to train other hyper-networks for other pre-trained networks. This whole process doesn't look efficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your comments on our paper. You may find our responses below for your concerns. If you have any further concerns, we would be grateful if you could let us know.\n\n##### [W1 & Q1] Explanation and justification for the necessity of model editing in the vision domain\n> - We would like to humbly clarify that the motivation for model editing in the visual domain **aligns with** that in LLMs, i.e., **enabling data-efficient updates of a pre-trained model to correct errors** (c.f. the last sentence of the first paragraph in the introduction in [2] and [3]).\n>    - **Necessity to correct errors**: analogous to outdated knowledge in LLMs, **vision models also** encounter errors over time, such as failing to recognize images with subpopulation shifts. Correcting these errors, particularly in safety-critical applications like self-driving, is urgent. \n>    - **Necessity for data-efficient updates**: the optimal strategy to correct errors without sacrificing generalization capabilities is to re-train a foundation model (**including LLMs and vision foundation models**) with both pre-training and error data. However, pre-training data, due to its huge volume, is often inaccessible [1] or computationally exhaustive to re-train.\n> - The motivation of being **data-efficient**, instead of parameter/memory-efficient, has led to several lines of model editing methods that **do not necessarily reduce** parameter or memory. For example,\n>    - fine-tuning all parameters [5] and hypernetwork-based methods [6] update **all** parameters;\n>    - memory-based [2] and T-patcher [7] even introduce **additional memory and parameters**, respectively.\n> - Our proposed method falls under the locate-then-edit line [8],\n>    - whose primary motivation is to address **the generalization-locality trade-off, which vision models also face as shown in Fig. 4**, by precisely targeting a minimal number of parameters to update; In Fig. 4, ours shows **impressive improvement on the balance** between generalization and locality.\n>    - which meantime enjoys the benefits of **reducing** parameters, memory, and also **FLOPS of vision models**. Compared to full fine-tuning which consumes around 52.8G FLOPs per editing iteration, ours consumes 35.5G FLOPs for the first itertation and 26.4G FLOPs for subsequent iterations. Please refer to [R2] of the general response for calculation details.\n\n##### [W2 & Q2 & Q3] Computational efficiency concerns regarding the hypernetwork \n> - Please kindly refer to **[R2] of the global response**.\n\n##### [Q4] Clarification on the fair comparison\n> For fair comparison, we have indeed compared against the state-of-the-art hypernetwork-based editing baselines, including KE [6] and MEND [4], in our experiments. The two baselines **also involve the training of a hypernetwork with multiple episodes**, although their objective and therefore inputs/outputs of hypernetworks differ from ours.\n>    - Hypernetworks in the two baselines learn how to edit, while ours first learns where to edit.\n>    - Both inputs and outputs are parameter gradients, while ours takes an editing sample as input to output a binary mask.\n> \n##### [Q5] Extention to large-scale networks\n>  We humbly clarify that our method generalizes well to larger-scale networks.\n>  - First, **even as the size of pre-trained networks grows, our method via greedy search reduces the optimal layers to edit** (i.e., the hypernetwork output space) to a minimal number (c.f. Line 201). During the response period, we found that editing the 19th to 21st layers of **SwinV2-g (1B)** is sufficient to achieve a good balance between generalization and locality.\n>  - Second, **at the hypernetwork side, its size does not necessarily increase with the size of the pre-trained model**:\n>    - For the same pre-trained model of ViT/B-16, a small hypernetwork can achieve almost comparable performance to larger hypernetworks. Decreasing the number of blocks in the hypernetwork from 5, to 3, to 1, did not result in a pronounced performance drop (c.f. Fig. r3 of the rebuttal PDF).\n>    - Utilizing the same hypernetwork consisting of 5 transformer blocks still helps precisely locate edits for even billion-scale models like SwinV2-g, evidenced in Fig. r1(b-c) of the rebuttal PDF. \n> - Third, even if more layers are identified to edit, the size of the hypernetwork only marginally increases. This is achieved by including one more trainable input token for each FC layer without changing the architecture of the hypernetwork.\n\n\n[1] Transferring pre-trained multimodal representations with cross-modal similarity matching, NeurIPS 2022.\n\n[2] Memory-based model editing at scale, ICML 2022.\n\n[3] Editing large language models: Problems, methods, and opportunities, EMNLP 2023.\n\n[4] Fast model editing at scale, ICLR 2022.\n\n[5] Modifying memories in Transformer models, ArXiv 2020.\n\n[6] Editing factual knowledge in language models, EMNLP 2021.\n\n[7] Transformer-patcher: one mistake worth one neuron, ICLR 2023.\n\n[8] Knowledge neurons in pretrained transformers, ACL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Td3pbE553M", "reviewer": "Reviewer_sCvZ", "summary": "The paper explores and discussed the VIT editing, which is the first work in this field. The paper proposes a learning-to-learn approach that identifies a small set of critical parameters to edit to correct the prediction of an error example. The paper considers the reliability and generalization during the ViT editing. In experiments, the paper curates two benchmarks to validate that the framework can correct the error and provide customization.", "questions": "Please refer to the weakness part.", "limitations": "The author address the limitations.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tThe problem formulation and method are clearly defined and explained, and the method is easy to follow.\n2.\tThe problem of ViT and the proposed method based on bi-level optimization are new in this field and meaningful.\n3.\tThe paper contributes the benchmark for the problem including the natural data and generated data which is beneficial to later research.\n4.\tThe experiments are intensive and validated on both real data and generated data. The ablation and analysis cover various metrics that form strong evidence.", "weaknesses": "1.\tThe motivation of CutMix and its influence are not clearly discussed and demonstrated. For example, there are also other data augmentation methods, are they the same as CutMix? Can they generate the same \n2.\tThe paper mention that the editing should not influence the irrelevant examples. However, this is not clearly discussed in the paper. I have questions about the metric Locality. Specifically, if the predicted probability of the irrelevant examples changed, does this indicate that the model change?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. We address your concerns below point by point. Please kindly let us know whether you have any further concerns.\n\n##### [W1] Motivation and influence of CutMix\n> We appreciate the reviewer's insightful question regarding the motivation and influence of CutMix, as well as the comparison with other data augmentation methods. To answer these questions:\n> \n> **Motivation and influence**: As stated in lines 190-191, the primary motivation for using CutMix is to efficiently generate pseudo-editing training episodes that **emulate the actual editing process**, thereby **enabling the hypernetwork to learn-to-localize** the most crucial parameters for editing during meta-training. Specifically:\n> - Given an original image, we generate an associated **pre-editing image** through CutMix by pasting a small patch of another image (ranging in size from 48 × 48 to 128 × 128) onto the original image at a random position. \n> - This newly introduced patch by **CutMix simulates distribution shifts** in backgrounds, contextual objects, and novel attributes of an object, **resulting in a difference in the pre-trained model's predictive distribution** between the pre-editing and original samples. \n> - **By aligning the model's predictive distributions** on the CutMix-generated pre-editing image with that of the original image **through fine-tuning only a subset of pre-trained parameters identified by the hypernetwork**, as stated in **our editing objective in Eq. (2)**, the hypernetwork learns to locate **the most crucial parameters** in the pre-trained models that account for the patch introduced by CutMix. This achieves our goal of learning where to edit in vision Transformers.\n\n> **Comparison to other data augmentations**: To address the reviewer's query, we refer to our ablation study in our paper where we have compared CutMix with PGD, a gradient-based augmentation strategy known for capturing diverse distribution variations (see line 381). As shown in **Fig. 6(a)**, the performance of hypernetworks meta-trained with CutMix rivals that of more computationally expensive gradient-based augmentation strategies. This indicates that CutMix is not only effective but also efficient in simulating the necessary distribution shifts for our model editing tasks.\n\n\n##### [W2] Clarification on the locality metric\n> Due to space constraints, we have included the discussion on the locality metric in **Appendix B.1** of our manuscript. To further improve the clarity of the paper, we will explicitly highlight this in the revised manuscript. Specifically, we would like to emphasize the following points from our discussion:\n\n> - **Our definition and evaluation of the locality metric are strictly consistent with the ones in prior works**. \n>   - As in [1, 2], the locality metric (see Eq. (6)) measures whether the edited model can maintain the prediction accuracy on a set of irrelevant data at the level of the pre-trained model (refer to the Metrics section in [1] and the evaluation of specificity, which is equivalent to locality, in [2]).\n>   - That said, **to address the reviewer's question**: In line with the definition of locality metric in prior works, a change in the edited model's predicted probability on an irrelevant sample, **as long as it does not affect the accuracy of the predicted label**, reflects a change in the edited model but does not impact the value of the locality metric.\n> - **We have designed our evaluation benchmark to better reflect changes in the edited model, even with the above definition of locality metric.**\n>   - To better capture model changes under slight predicted probability variations, we collect **sensitive validation images** from ImageNet, ImageNet-R, and ImageNet-Sketch to evaluate locality. \n>   - These sensitive images are characterized by: **1)** being **correctly predicted by the pre-trained model**, meaning that they have the highest predicted probability on the label class, and **2)** having **a difference less than 0.05 in predicted probability** between the label class and the class with the second highest predicted probability.\n>   - As a result, prediction accuracy, **hence locality, on these sensitive images is affected even by small changes in predicted probability**. \n\n[1] Fast model editing at scale, ICLR 2022.\n\n[2] Locating and editing factual associations in GPT, NeurIPS 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kwabqD110I", "reviewer": "Reviewer_MKpn", "summary": "The paper presents a novel approach for editing Vision Transformers (ViTs) to correct predictive errors, specifically addressing subpopulation shifts. It introduces a learning-to-learn methodology utilizing a hypernetwork that identifies and modifies a small set of critical parameters in response to erroneous samples. This method is distinguished by its focus on localized edits, ensuring minimal disruption to unrelated model functions while generalizing corrections to similar errors.", "questions": "Could a strategy similar to the one proposed by [1] be adapted to apply a low-rank update to the weight matrix for memory editing in ViTs?\n\n[1] Meng, Kevin, et al. \"Locating and editing factual associations in GPT.\" Advances in Neural Information Processing Systems 35 (2022): 17359-17372.s:", "limitations": "The CutMix technique employed may not adequately represent real-world shifts, which could limit the applicability of the findings to practical scenarios.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The concept of utilizing a hypernetwork to identify editing locations in ViTs for image processing tasks is innovative, addressing a gap in the literature concerning efficient and localized model editing. The proposed method is rigorously validated through the introduction of new benchmarks, showing significant improvements over existing techniques. The paper is well-organized and clearly written, with technical details and methodologies elaborately explained, making it accessible to readers familiar with the field. This research is highly significant as it enhances the practical utility of ViTs in real-world applications, particularly where predictive reliability is crucial, such as in autonomous driving and medical imaging.", "weaknesses": "The success of the proposed method heavily relies on the hypernetwork's ability to accurately predict edit locations. Misidentifications could lead to suboptimal edits, affecting model reliability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive comments on improving our paper. We detail our response below point by point. Please kindly let us know if our response addresses the questions you had for this paper.\n\n##### [W1] Reliability of the hypernetwork\n> \n> We argue that identifying the 'correct' parameters to edit (i.e., editing locations) is a central challenge in any localization-based editing model, not just a particular challenge for our hypernetwork-based localization method.\n>\n> - **To tackle this common challenge, we have proposed the following two strategies to improve the reliability of our hypernetwork for localization**:\n>   1. **Meta-learn to edit**: Our meta-trained hypernetworks leverage transferable knowledge (inductive bias) from related training editing episodes to enhance the editing performance on each test editing sample. As shown in Fig. 5b and lines 361-363, the similarity between editing regions output by the hypernetworks correlates with the similarity in the input data. In contrast, previous editing baselines, such as FT-L2, determine the editing region for each image solely based on that particular test editing sample, which is prone to overfitting (resulting in low generalization) or overwriting irrelevant concepts (resulting in low locality).\n> \n>   2. **Hypernetwork Optimization Decoupling Trick**: As described in Appendix B.3.2, by introducing intermediate variables $\\tilde{m}$ as the target editing region for the hypernetwork's prediction, this technique successfully ensures optimization success and provides extra supervision signals to guide the hypernetworks in learning to output optimal editing masks.\n> \n> - **On the empirical evaluation side, we conduct extensive experiments to demonstrate the reliability and superiority of our methods across various scenarios**.\n>   1. We evaluate the editing methods **on various datasets**: the natural dataset, which consists of sixteen types of errors; the AI-generated datasets, which include two types of shifts (c.f. Fig. 4); and the fine-grained classification dataset (c.f. Fig. r4 of the rebuttal PDF).\n>   2. We apply our methods **to various pre-trained models**: ViT/S-16 (c.f. Fig. K in appendix), ViT/B-16 (c.f. Fig. 4), and SwinV2-g (c.f. Fig. r1 of the rebuttal PDF), under different pre-training strategies, including supervised pre-training for ViT/S-16 and ViT/B-16, and SimMIM self-supervised pre-training [1] for SwinV2-g.\n> \n\n##### [Q1] Adaption of ROME with LoRA\n> Following the reviewer's suggestion, we adapt ROME with LoRA for model editing experiments on two groups of the natural dataset using ViT/B-16. The results are presented in the table below.\n> \n> Although the combination of ROME and LoRA demonstrates an improved trade-off between generalization and locality compared to using LoRA individually, **our method still outperforms this combined approach by 13.58% and 15.31% in generalization** at similar locality rates in the 609-586 and 890-430 groups, respectively, even ROME and ROME+LoRA require access to pre-training data.\n>\n >    609-586 group in the natural dataset | Generalization | Locality\n    >    ---- | --- | ---\n    >    ROME | 78.36 | 96.56\n    >    LoRA |  76.07 | 91.27\n    >    ROME+LoRA |  81.27 |  91.13\n    >    Ours |  94.85 | 90.79\n >    890-430 group in the natural dataset | Generalization | Locality\n    >    ROME | 66.10  |  97.70\n    >    LoRA | 65.21  |  91.30\n    >    ROME+LoRA | 71.75  |  91.84 \n    >    Ours |    87.06 | 92.83\n\n[1] On data scaling in masked image modeling, CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers and ACs for your diligent efforts and high-quality reviews.  If there are any additional questions or if further clarification is needed, please feel free to let us know. Your insights are highly valued.

We are delighted to note that reviewers find that:

- our method is novel (`Reviewers wreN`), innovative (`Reviewers MKpn`), with clear and easy-to-follow writing (`Reviewers wreN, sCvZ, MKpn`)
- we  contribute a new benchmark for model editing (`Reviewers 91WR, sCvZ, and MKpn`), which is beneficial  to later research (`Reviewers sCvZ`)

In response to your valuable suggestions, we conduct additional experiments and the supplementary rebuttal PDF includes the new results for your convenience:

- **Fig. r1**: We add experiments on SwinV2-g [1], a large-scale (suggested by `Reviewer 91WR`) hierarchical (suggested by `Reviewer wreN`) ViT.
- **Fig. r2**: Ablation study for a clear comprision on different regularization terms (suggested by `Reviewer wreN`).
- **Fig. r3**: Ablation study of the number of blocks in the hypernetwork (suggested by `Reviewer 91WR`).
- **Fig. r4**: We add experiments on the Stanford Cars to futher evaluate our methods (suggested by `Reviewer wreN`).

Finally, due to character limits, we put responses to commonly raised questions below.

##### [R1] Application to large-scale hierarchical ViTs
> We note that our approach is designed to be broadly applicable across various ViTs, including **large-scale** and  **hierarchical** ViTs. To verify our claim:
> - First, our method meta-learns the optimal locations for editing within the parameter space of any given pre-trained ViT, without imposing strict assumptions on the specific architecture of the models. 
> - Second, even as the size of pre-trained networks grows, our method via greedy search reduces the optimal layers to edit (i.e., the hypernetwork output space) to a minimal number (c.f. Line 201). The results in Fig. r1(a) of the rebuttal PDF indicate that editing the 19th to 21st layers of SwinV2-g (1B) is sufficient to achieve a good balance between generalization and locality.
> - We compare our method with the most recent editing methods for editing SwinV2-g in Fig.r1(b-c) of the rebuttal PDF.  Our proposed method demonstrates **superior performance** compared to all baselines, achieving **the best balance** between generalization and locality. 
> - Due to the time limit and the large scale of the model, we only included the most recent editing methods and will include all methods.

##### [R2] Computational efficiency regarding the hypernetwork (To Reviewer wreN and 91WR)
> First, we would like to outline the major computational processes of the proposed method in the following table.
> 
> | (a) Before Editing (Section 3.3) | (b) Test-time Editing (Section 3.4) | (c\) Test-time Editing (Section 3.4)|
> | -------- | -------- | -------- |
> | Meta-learning the hypernetwork with Eqn. (2)   | Forward pass of the hypernetwork to obtain the mask $m^e$ in Eqn. (3)     | Training only the parameters activated by $m^e$ (c.f. Line 238-242) |
> - We humbly clarify that, in line with hypernetwork-based model editing methods [2,3], the additional computational costs incurred by hypernetwork training (i.e., (a)) are acceptable and worthwhile for several reasons:
>   - Training the 5-block hypernetwork (31.9M) in our paper for a pre-trained ViT/B-16 takes approximately 9 hours on a single RTX A6000 GPU (48G).
>   - Given a target pre-trained network, the hypernetwork is trained **prior to model editing (i.e., (b) + (c\)) and this training only needs to be done once**.
>   - The number of available popular vision pre-trained models (or even LLMs) is limited [4].
>   - The effectiveness of the learned hypernetwork in locating "where to edit" has been proved in subsequent editing tasks (c.f. Fig. 4 and Appendix C.1/C.2) and the comparison with random masks (c.f. Fig. 5(c\)).
> - During test-time editing (i.e., (b) + (c\)), our method **even reduces the computation compared to the editing method of full fine-tuning** which consumes around 52.8G FLOPs per editing iteration;
>   - In (b), one-shot inference with the hypernetwork to generate the mask takes only 277.1M FLOPs;
>   - In (c\),  updating the parameters activated by the mask consumes 35.5G FLOPs in the first iteration and 26.4G FLOPs in subsequent iterations, either of which is significantly less than 52.8G FLOPS. The difference between iterations arises because the first iteration requires a one-time full forward pass through the pre-trained ViT to obtain features of the error samples, whereas subsequent iterations only update the optimal layers identified by our method (c.f. Line 201).
>   
>

[1] On data scaling in masked image modeling, CVPR 2023.

[2] Editing factual knowledge in language models, EMNLP 2021.

[3] Fast model editing at scale, ICLR 2022.

[4] Battle of the backbones: A large-scale comparison of pretrained models across computer vision tasks, NeurIPS 2023.

Meta-review: This paper proposes a new method to edit vision transformers using a hypernetwork. It introduces a learning-to-learn methodology utilizing a hypernetwork that identifies and modifies a small set of critical parameters in response to erroneous samples. 

The submission received scores of 7,5,7,6 and the rebuttal and discussion between reviewers and authors has cleared the remaining points. The AC recommends acceptance.
Decision: Accept (poster)