Editing a classifier by rewriting its prediction rules
Authors: Shibani Santurkar
Dimitris Tsipras
Mahalaxmi Elango
David Bau
Antonio Torralba
Aleksander Madry
Venue: NeurIPS 2021 Poster (2021)
Keywords: model debugging
spurious correlations
robustness

Abstract: We propose a methodology for modifying the behavior of a classifier by directly rewriting its prediction rules. Our method requires virtually no additional data collection and can be applied to a variety of settings, including adapting a model to new environments, and modifying it to ignore spurious features.

Editing a Classifier by Rewriting Its Prediction Rules
Shibani Santurkar' MIT shi Shibaniie @mit .eiit.edu
Dimitris Tsipras" MIT tsipras@mit.ent edu
Mahalaxmi Elango MIT
melango@mit.. edu
David Bau MIT davidbau@mit.edd
Antonio Torralba MIT torralbammit.edd
Aleksander Mądry
MIT madry@mit edu
Abstract
We present methodology for modifying the behavior of classifier by directly rewriting its prediction rules [1] Our approach requires virtually no additional data collection and be applied to variety of settings, including adapting model to new environments, and modifying it to ignore spurious features.
Introduction
At the core of machine learning the ability to automatically discover prediction rules from raw However, there is mounting evidence that not all of these rules are reliable [Torral ba and
data.
Efros
2011
Beery et al. 2018 Shetty et al., 2019, Agarwal et al., 2020 Xiao et al. 2020 Bissoto Geirhos et al. 2020]. In particular, some rules could be based on biases in the training
et al.
2020
data: e-g..
learning associate cows with grass since they are typically depicted on pastures (Beery
While such prediction rules may be useful in some scenarios, they will be irrelevant or misleading in others. This raises the question:
al.
2018
How most effectively modify the way in which given model makes its predictions?
The canonical approach for performing such post hoc modifications is to intervene at the data level. For example, by gathering additional data that better reflects the real world (e-g.. images of cows on the beach) and then using it to further train the model. Unfortunately, collecting such data can be challenging: how do we get cows to pose for in a variety of environments? Furthermore, data collection is ultimately very indirect way of specifying the intended model behavior. After all, even when data has been carefully curated to reflect given real-world task, models still end up learning unintended prediction rules from it [Ponce et al. 2006 Torralba and Efros 2011 Tsipras
ct al. 2020 Beyer et al. 2020
Our contributions
The goal of our work is develop toolkit that enables users directly modify the prediction rules learned by an (image) classifier, as opposed to doing so implicitly via the data. Concretely:
Editing prediction rules. We build on the recent work of Bau al. 2020a for modifying classifier's prediction rules with essentially no additional data collection (Section 2p. At high level, our method enables the user to modify the weight of layer so that the latent
develop method
"Equal contribution.
OOr code is available http://///ghtta.oooo//aaaaaaaaar
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

POLLIC
SOALICE
(b)
VGG16
"now"
"road"
Pre: amphibian racer
snowmobile motor scooter
snowplow car wheel
snowplow traffic light
Post:
(c)
(d)
CLIP
"iPod letters"
"blank"
Pre:
iPod
iPod
iPod
iPod toilet
Post:
teapot
mug
pot
Test
Figure 1: Editing prediction rules in pre-trained classifiers using a single exemplar. (a) We edit VGG16 ImageNet classifier map the representation of the concept "snow" to that of "asphalt road". (b) This edit corrects classification errors on snowy scenes corresponding to various classes. (c) We edit CL.IP [Radford al., 202T] model such that the text "iPod" maps to blank area. (d) This change makes the model robust to the typographic attacks from Goh et al. [202T].
representations of a specific concept (e.g., snow) map to the representations of another (e.g., road). Crucially, this allows us change the behavior of the classifier on all occurrences of that concept, beyond the specific examples (and the corresponding classes) used in the editing process.
Real-world scenarios. We demonstrate our approach in two scenarios motivated by real-world applications (Section 3). First, we focus on adapting an ImageNet classifier to new environment: recognizing vehicles on snowy roads. Second, we consider the recent "typographic attack" Goh et al. [021] on zero-shot CLIP [Radford et al. [021]] classifier: attaching piece of paper with "PPdd" written on to various household items causes them to be incorrectly classified as "iPod." In both settings. we find that approach enables to significantly improve model performance, using only single, synthetic example to perform the edit-cf. Figurel)
Large-scale synthetic evaluation. To evaluate our method scale, develop an automated pipeline to generate suite of varied test cases (Section 4.. Our pipeline revolves around identi- fying specific concepts (e.g.. "road" or "pasture") an existing dataset using pre-trained instance segmentation models and then modifying them using style transfer [Gatys et al.) 2016) (c.g.. to create "snowy road"). We find that our editing methodology is able consistently correct signif- icant fraction of model failures induced by these transformations. In contrast, standard fine-tuning approaches are unable to do so given the same data, often causing more errors than they are fixing.
Probing model behavior with counterfactuals. Beyond model editing, transformation pipeline can also be viewed as a scalable way of generating image counterfactuals. In Section [5] demonstrate how such counterfactuals can be useful to gain insights into how given model makes its predictions and pinpoint certain spurious correlations that it has picked up.
concept-
2 A toolkit for editing prediction rules
has been widely observed that models pick up various context-specific correlations in the data- e.g., using the presence of "road" "wheel" to predict "car" (cf. Section 5). Such unreliable prediction rules (dependencies of predictions on specific input concepts) could hinder models when they encounter novel environments (e.g., snow-covered roads), and confusing adversarial test conditions (e.g.. cars with wooden wheels). Thus, a model designer might want to modify these rules before deploying their model.
The canonical approach to modify classifier post hoc to collect additional data that captures the desired deployment scenario, and use it to retrain the model. However, even setting aside the

---

challenges of data collection, it is not obvious a priori how much of an effect such retraining (e.g., via fine-tuning) will have. For instance, if we fine-tune our model on "cars" with wooden wheels, will it now recognize "scooters" or "trucks" with such wheels?
The goal of this work is to instead develop more direct way to modify model's behavior: rewriting its prediction rules in a targeted manner. For instance, in our previous example, we would ideally be able to modify the classifier correctly recognize all vehicles with wooden wheels by simply teaching it to treat any wooden wheel as it would a standard one. Our approach is able to do exactly this. However, before describing this approach (Section 2.2p. we first provide a brief overview of recent work by Bau et al. [2020a] which forms its basis.
2.1 Background: Rewriting generative models
Bau et al. 2020a] developed an approach for rewriting deep generative model: specifically, en- abling a user to replace all occurrences of one selected object (say, "dome") in the generated images with another (say. "tree"), without changing the model's 's behavior in other contexts. Their approach motivated by the observation that, using handful of example images, we can identify vector in the model's s representation space that encodes a specific high-level concept [Kim et al. 2018 Bau al. 2020a]. Building on this, Bau et al.] [2200]] treat each layer of the model as an associative memory, which maps such a concept vector at each spatial location in its input (which we will refer to as the key) to another concept vector in its output (which we will call the value).
In the simplest case, one can think of a linear layer with weights W E R transforming the key k IR" to the value modifying the layer weights from W to W" so that o² W where k" corresponds to the old concept that we want to replace, and v* to the new concept. For instance, to replace "domes" with "trees" in the generated images, we would modify the layer so that the key k* for "dome" maps to the value v" for "tree". Consequently, when this value fed into the downstream layers of the network it would result in a tree in the final image. Crucially, this update should change the model's s behavior for every instance of the concept encoded in --- all "domes" in the images should now be "trees".
Rm In this setting. Bau al. [2020a] formulate the rewrite operation as
To extend this approach to typical deep generative models, two challenges remain: (1) handling non-linear layers, and (2) ensuring that the edit doesn't I significantly hurt model behavior on other concepts. With these considerations in mind, Bau et al. [2020a] propose making the following rank- one updates to the parameters W of an arbitrary non-linear layer J::
min A
Σ El - S W'll (i,))S
s.t.
W' = W + A(C
(1)
Here, S denotes the set of spatial locations in representation space for a single image corresponding to the concept of interest, d is the top eigenvector of the keys kij at locations (i, E S and C
Ea kaka captures the second-order statistics for other keys hy. Intuitively, the goal of this update modify the layer parameters rewrite the desired key-value mapping in the most minimal way. We refer the reader to Bau et al. [2020a] for further details.
2.2 Editing classifiers
We now shift our attention to the focus of this work: editing classifiers. To describe our approach, we use the task of enabling classifiers to detect vehicles with "wooden wheels" as a running example. At high level, we would like to apply the approach described in Section 2. to modify a chosen (potentially non-linear) layer of the network to rewrite the relevant key-value association. But, we need to first determine what these relevant keys and values are.
Let us start with single image x, say, from class "car", that contains the concept "wheel". Let the location of the "wheel" in the image be denoted by a binary mask m 2 Then, say we have access to a transformed version of x-namely, wh-where the "car" has "wooden wheel" (cf. Figure 2). This x' could be created by manually replacing the wheel, or by applying an automated procedure such as that in Section 4 In the rest of our study, we refer to a single (a, x') pair as an exemplar.
? Such mask can either be obtained manually or automatically via instance segmentation (cf. Section
3

---

Classifier
Layer
Original (x)
Label
Mask
w
(m)
Downsample
Modified
(4)
Label
(a) Edit
(b) Test
Figure 2: Overview of our pipeline for directly editing the prediction-rule of classifier. The edit in (a) seeks to modify the network to perceive wooden wheels as standard ones, using a small set of exemplar images (say from class "car"). To achieve this, we first obtain the keys ki corresponding to the new concept (here, "wooden wheel"), and the values ti corresponding to the original concept (here, "standard wheel") in the input and output representation space of a layer L respectively. We then update the weights W of the layer enforce this new key-value association ( (b) To test our method, we measure the improvement snncppthhere images of other vehicles with "wooden wheels".
model performance on test instances containing the new
Intuitively, we want the classifier to perceive the "wooden wheel" in the transformed image x' does the standard wheel in the original image z. To achieve this, we must map the keys for wooden wheels the value corresponding to their standard counterparts, In other words, the relevant keys (k*) correspond to the network's representation of the concept in the transformed image (x') directly before layer T.. Similarly, the relevant values (v") that we want map these keys correspond to the network's representation of the concept in the original images (z) directly after layer L. (The pertinent spatial regions the representation space are simply determined by downsampling the mask to the appropriate dimensions.) Finally, the model edit is performed by feeding the resulting key-value pairs into the optimization problem (I) to determine the updated layer weights W'-cf. Figure [ for an illustration of the overall process. Note that this approach can be easily extended to use multiple exemplars (Ix, xf{)) by simply expanding S to include the union relevant spatial locations (corresponding the concept of interest) across these exemplars.
3 Does editing work in practice?
To evaluate of our approach, we start by considering two scenarios motivated by real-world con- cerns: (i) adapting classifiers to handle novel weather conditions, and (ii) making models robust to typographic attacks Goh et al. [2021]. In both cases, we edit the model using single exemplar, i.e., a single image that we manually annotate and modify. For comparison, we also consider two vari- ants of fine-tuning using the same exemplar: (i) local fine-tuning, where we only train the weights of a single layer (similar to our editing approach); and (ii) global fine-tuning, where we also train all other layers between layer L and the output of the model. It is worth noting that unlike fine-tuning, editing does not utilize class labels in any way. See Appendix for experimental details.
3.1 Tackling new environments: Vehicles on snow
Our first use-case is adapting pre-trained classifiers to image subpopulations that are under- represented the training data. Specifically, we focus on the task of recognizing vehicles under heavy snow conditions- setting that could be pertinent to self-driving cars using a VGG16 clas- sifier trained on ImageNet- To study this problem, we collect a set of real photographs from road-related ImageNet classes using Flickr (details in Appendix A.5). We then rewrite the model's prediction rules to map "snowy roads" to "road". To do so, we create an exemplar by manually an- notating the concept "road" in an ImageNet image from different class (here, "police van"), and the

---

Correct predictions "
Accuracy
original
20
fine-tuning
global fine nan-tunning fine-tuning editing
editing
racing car
amy tank (retruck
wheel
traftic right
schoal bus
sconter
Typed
Handwritten
(a) Vehicles in snowy weather
(b) Typographic attacks (6 images)
Figure 3: (a) Adapting a pre-trained ImageNet VGG16 classifier to images of vehicles on snowy roads with single exemplar. Fine-tuning (both local and global) does not improve accuracy, while editing to map "snowy "oad"">rroan leads to consistent improvement across multiple classes. (b) Improving the robustness of CIIP-ResNet-550 [Radford et al. 202T] models to typographic attacks [Goh et al., 2021]. Editing the model to map the text "iPod" + ""blank" using a single exemplar- either based on hand- wrrtten text on a physical teapot or from pasting typed text on an image of a "can opener"-come completely corrects this vulnerability. While global fine-tuning can also improve model performance in this setting, requires more careful hyperparameter tuning and typically hurts model performance in other contexts (Appendix Figure (9). Here, hyperparameters (cf. Appendix Table are chosen based on the large-scale synthetic study in Section 4.1
manually replace it with texture obtained from Flickr. We then apply our editing methodology (cf. Section 2). using this single synthetic snow-to-road exemplar-ses Figure
In Figure 3a measure the error rate of the model on the new test set (vehicles in snow) before and after performing the rewrite. We find that our edits significantly improve the model's error rate on these images, despite the fact that we only road" photograph). Moreover, Figure Ba demonstrates that our method indeed changes the way that the model processes concept (here "snow") in way that generalizes beyond the specific class used during editing (here, the exemplar was "police van"). In contrast, fine-tuning the model under the same setup does not improve its performance on these inputs.
single syntheric exemplar (i.e., not real "snowy
One potential concern is the impact of this process on the model's accuracy on other ImageNet classes that contain snow (e.g., "ski"). On the 246 (of 50k) ImageNet test images that contain snow (identified using an MS- MS--OCC-tr instance segmentation model [Chen et al. (017]), ), the model's accuracy pre-edit is 92.27% and post-edit is 91.05%-i... only 3/246 images rendered incorrect by the edit. This indicates that the classifier not disproportionately affected by the edit.
3.2 Ignoring a spurious feature: Typographic attacks
Our second use-case is modifying model to ignore spurious feature. We focus the recently- discovered typographic attacks from Goh et al. [021]: simply attaching a piece of paper with the text "iPod" on it is enough make zero-shot CLIP [Radford et al.) 2021] classifier incorrectly classify an assortment of objects to be iPods. We reproduce these attacks on the ResNet-50 variant of the model see Appendix Figure for an illustration.
To correct this behavior, we rewrite the model's prediction rules map the text "iPod" to "blank". For the choice of our transformed input x', we consider two variants: either real photograph of "teapot" with the typographic attack (Appendix Figure 7): or an ImageNet image of a "can opener" (randomly-chosen) with the typed text "iPod" pasted on (Figure D The original image for approach is obtained by replacing the handwritten/typed text with a white mask Figure We then use this single training exemplar to perform the model edit.
In both cases, we find that editing is able fix all the errors caused by the typographic attacks, see Figure (3b] Interestingly, global fine-tuning also helps to correct many of these errors (potentially by adjusting class biases), albeit less reliably (for specific hyperparameters) However, unlike editing, fine-tuning also ends up damaging the model behavior in other scenarios- e.g. the model now spuriously associates the text "iPod" with the target class used for fine-tuning and/or has lower accuracy on normal "iPod" images from the test set (Appendix Figure
5

---

Transform concepts
Benchmarks
Dataset
Identify concepts (via instance segmentation)
II. Intervene concepts (via style transfer)
Sea
"tree"
gravel
Figure 4: Creating large-scale test sets for model rewriting. Given standard dataset, first identify salient concepts within the corresponding images using instance segmentation, and then apply realistic transformation to each of these concepts using style transfer [Gatys et al. [016]] We can then evaluate the effectiveness of a model rewriting technique based on the extent to which it can alleviate the model's sensitivity (i.e., drop in accuracy) such concept-level transformations.
Large-scale synthetic evaluation
The analysis of the previous section demonstrates that editing improve model performance in realistic settings. However, due to the practical constraints of real-world data collection, this analysis was restricted relatively small test To corroborate the generality of our approach, we now develop pipeline to automatically construct diverse rule-editing test cases. We then perform large-scale evaluation and ablation of editing method on this testbed.
4.1 Synthesizing concept-level transformations
At a high level, our goal is to automatically create a test set(s) in which a specific concept- possibly relevant the detection of multiple dataset undsege-udeereess realistic transformation. To achieve this without additional data collection, we transform all instances of the concept of in- terest within existing datasets. For example, the "vehicles-on-snov" scenario of Section 3.1 can be synthetically reproduced by identifying the images within a standard dataset (say ImageNet (Deng al. 2009 Russakovsky al. 2015) that contain segments of road and transforming these make them resemble snow. Concretely, our pipeline (see Figure 4 for an illustration and Appendix A.
for details), which takes input an existing dataset, consists of the following two steps:
1. Concept identification: In order identify concepts within the dataset images in scal- able manner, we leverage pre-trained instance segmentation models. In particular, us- ing state-of-the-art segmentation models-trained
MS-COCO [Lin al.] 2014 and LVIS [Gupta et al. 2019]-we are able to automatically generate concept segmentations for range of high-level concepts (e.g., "grass", "sea", "tree").
2. Concept transformation: We then transform the detected concept (within dataset images) in a consistent manner using existing methods for style transfer [Gatys et al.) 2016 Ghiasi et 2017). This allows preserve fine-gra ainiindi image features and realism, while still exploring a range of potential transformations for a single concept. For our analysis, we manually curate of realistic transformations (e.g., "snow" and "graffiti")..
Note that this concept-transformatio pipeline does not require any additional training data an- notation. Thus it can be directly applied to new datasets, as long as we have access to pre- pre-trained segmentation model for the concepts of interest.
4.2 Creating suite of editing tasks
We now utilize the concept transformations described above to create benchmark for evaluating model rewriting methods. Intuitively, these transformations can capture invariances that the model should ideally have-e.g. recognizing vehicles correctly even when they have wooden wheels. In practice however, model accuracy on one or more classes (e.g., "car", "scooter") may degrade under

---

these transformations. The goal of rewriting the model would thus be to fix these failure modes in a data-efficient manner. In this section, we evaluate our editing methodology as well as the fine-tuning approaches discussed in Section 3 along this axis.
VGG (Simonyan and Zisserman 2015 and ResNet [He et al. 2015) models trained on the ImageNet (Deng et al. 2009) Russakovsky et al, 2015) and Places-363 365 [Zhou al. 2017] datasets (cf. Appendix A. Each test set constructed using the tonnepttrrnnsformation pipeline discussed above, based on a chosen concept-style pair (say "wheel"""wooden" ""Wooden") [ It consists of N exemplars (pairs of original and transformed images, (r)=²)) that belong to single (randomly-chosen) target class in the dataset. All other transformed images containing the concept, including those from classes other than the target one, are used for validation and testing (30-70 split). We create two variants of the test set: one using the same style image as the exemplars (i.e., same wooden texture) for the transformation; and another using held-out style images (i.e., other wooden textures).
Concretely, we focus vision
To evaluate the impact of a method, we measure the change in model accuracy on the transformed examples (e.g., vehicles with "wooden wheel's in Figure 2b). If the method is effective, then it should recover some of the incorrect predictions caused by the transformations. We only focus on the subset of examples D that were correctly classified before the transformation, since we cannot expect to correct mistakes that do not stem from the transformation itself. Concretely, we measure the change in the number of mistakes made by the model on the transformed examples:
% errors corrected := Nyre (D- - Npost(D)
(2)
(D)
where N pore/ppst( (D) denotes the number of transformed examples misclassified by the model before and after the rewrite, respectively. Note that this metric can range from 100% when rewriting leads to perfect classification on the transformed examples, to even negative value when the rewriting process causes more mistakes that it fixes.
In each case, we select the best hyperparameters inncuuuu the choice of the layer to modify based on the validation set performance (ef. Appendix (..6... To quantify the effect of the modifi- cation on overall model behavior, we also measure the change in its (standard) test set performance. Since are interested in rewrites that do not significantly hurt the overall model performance, we only consider hyperparameters that do not cause a large accuracy drop (50.25%) We found that the exact accuracy threshold did not have significant impact on the results-see see Appendix Figures 15 for a demonstration of the full -ccuracy-eflectiveness tradc-off.
4.3
The effectiveness of editing
Recall that key desideratum of our prediction-rule edits is that they should generalize. That is, if we modify the way that our model treats a specific concept, we want this modification to apply to every occurrence of that concept. For instance, if we edit a model to enforce that "wooden wheels" should be treated the same as regular "wheels" in the context of "car" images, we want the model to do the same when encountering other vehicles with "wooden wheels". Thus, when analyzing performance in Figure 5 we consider inputs belonging to the class used to perform the edit separately.
Editing. We find that editing is able to consistently correct mistakes in a manner that generalizes across classes. That is, editing is able reduce errors in non-target classes, often by more than 20 percentage points, even when performed using only three exemplars from the target class. Moreover, this improvement extends to transformations using different variants of the style (e.g., textures of "woodd"), other than those present in exemplars used to perform the modification.
In Appendix B.1.3]
conduct ablation studies get better sense of the key algorithmic factors driving performance. Notably, we find that imposing the editing constraints (T) on the entirety of the image-as opposed to only focusing on key-value pairs that correspond to the concept of interest as proposed in [au et al. [2020a] leads to even better performance (cf. "-mask' in Figure ().. We hypothesize that this has regularizing effect as it constrains the weights to preserve the original mapping between keys and values in regions that do not contain the concept.
"ote that some of these cases might not be suitable for editing, i.e., when the transformed concept is critical for recognizing the label of an input image (e.g.. transforming concept "dog" in images of class "poodle"). We thus manually exclude such concept-class pairs from our analysis cf. Appendix A.6.2
7

---

100
160
% Emors corrected
editing
% Errors corrected
editing
editing (-mask) rre-tuning
editing (-mask)
fine tuning
global fine-tuning
global fine-tuning
Target
Ter get
Other classes Mtyl
Other
Target class
Target class, styles
Other classes style
Other classes styles
style
sty
styles
(a) ImageNet-trainet VGGI6 (concepts from COCO)
(b) Places-trainek ResNet- (concepts from LVIS)
Figure 5: Editing vs. fine-tuning, averaged over concept-style pairs. We find that both methods (and their variants) are fairly successful at correcting errors on the target class (examples of which are used to perform the rewrite). This holds even when the transformation applied during testing is different from the one present in the train exemplars (c.g., a different texture of "wood"). However, crucially, only the improvements induced by editing generalize other classes where the trans- formed concept is present. Fine-tuning fails this sosting-typicallly causing more errors than fixes. See Appendix Figures 10 13 for other experimental settings.
Fine-tuning. Our baseline is the canonical fine-tuning approach, i.e., directly minimizing the model loss on the new data (here the transformed images) with respect to their label. Similar to Section 3 we consider both the local and global variants of fine-tuning. We find that while these approaches are able correct model errors on transformed inputs from the class used to perform the modification, they typically decrease performance on other classes-i.e., they cause more errors than they fix. Moreover, even when we allow larger drop in the model's accuracy, or use more training exemplars, their performance often becomes worse such inputs (Appendix Figures 15 18
We present examples of errors corrected (or not) by editing and fine-tuning in Appendix Figure 14 and provide a per-concept/style break down in Appendix Figures 19 and 20
Beyond editing: Probing model behavior via counterfactuals
In the previous section, developed scalable pipeline for creating concept-level level transformations which we used to evaluate model rewriting methods. Here, we put forth another related use-case of that pipeline: debugging models to identify their (leamed) prediction rules.
In particular, observe that the resulting transformed inputs (cf. Figure 4 can be viewed as counter- factuals-2 primitive commonly used in causal inference [Pearl] 2010 and interpretability [Goyal jet al. b Bau et al. 20206). Counterfactuals can be used to identify the features that a model uses to make its prediction on the presence a "wheel" to recognize a "car" image, we can evaluate how the its prediction changes when just the wheel in the image is transformed. Based this view, now repurpose concept transformations from Section to discover a given classifier's prediction rules.
given input. For example, to understand whether the model relies
The effect of specific concepts. As a point of start, we study how sensitive the model's prediction is to a given concept in terms of the accuracy drop caused by transformation of said concept. For instance, in Figure 64, we find that the accuracy of VGG16 ImageNet classifier drops by 25% on images of "croquet ball" when "grass" is transformed, whereas its accuracy on "collie" does not change. In line with previous studies [Zhang et al.) 2007 Ribeiro al. 2016 Rosenfeld al. 2018 Barbu et al.) 2019 [iao et al.) 20200] we also find that background concepts. such as "grass". "sea" and "sand". have large effect on model performance. We contrast this measure of influence across concepts for single model (Appendix Figure 27). and across architectures for single concept (Appendix B.2). Finally, we examine the effect of each concept stylization F6g
Per-class prediction rules. If instead we restrict our attention to a single class, we can pinpoint the set of concepts that the model relies on detect this class. It turns out that aside from the main image object, ImageNet classifiers also heavily depend on co-occurring objects SSock and Cisse
8

---

(a) Classes sensitive to the concept "grass"
(b) Style impact on accuracy
croquet
kit fox
Accuracy (96)
25
Accraa
20
(c) Concept sensitivity, class "graom"
water buffs
collie
craquet Ball
Kit medil prakie chucker
Pecaeettto lawn foo
baistand
Bur groii
Scatesh
Stafforde deriediiiin gna
Figure 6: Model sensitivities diagnosed using our pipeline in VGG16 ImageNet classifier. (a) Classes for which the model relies on "grass": e.g., a "croquet ball" is not accurately recognized if "grass" is transformed, while "collie"s are not affected. (b) Applying different transformations to visual concepts reduces accuracy by varying amounts. (c) Transformations that hamper accuracy can highlight prediction rules: e.g., the class "groom" is sensitive the presence of "dress."
2018 Tsipras et al. 2020 Beyer et al. 2020 in the image-eg. "dress" for the class "groom". "person" for the class "tench" (sic), and "road" for the class "race car" (cf. Figure 6 and Appendix Figure 29). We can also examine which styles hurt performance the most-e.g we find that making "plants" "floral" hurts accuracy the class "damselfly" 15% more than making them "snowy".
Overall, this pipeline provides scalable path for model designers to analyze the invariances (and sensitivities) of their models with respect to various natural transformations of interest. We thus believe that this primitive holds promise for future interpretability and robustness studies.
6 Related work
High-level concepts in latent representations. There has been increasing interest in explaining
the inner workings deep classifiers through high-level concepts: c.g., by identifying individual
neurons [Erhan et al.) 2009 Zeiler and Fergus, 2014 Olah et al. [017 Bau et al, 2017 Engstrom
al. 2019a| or activation vectors [Kim al.) 2018 Zhou et 2018 [hen et al.] 2020 that
correspond to human- unnrannnnndrttadaablll features. The effect of these features on model predictions can
be analyzed by either inspecting the downstream weights of the model OOla et al. 2018 Wong et al.) 2021] or through counterfactual tests: e.g., via synthetic data [Goyal et al. 2019a] by swapping features between individual images [Goyal et al. 2019b]; or by silencing sets neurons [Bau al., 2020b]. A parallel line of work aims to learn models that operate on data representations that explicitly encode high-level concepts: either by learning to predict a set of attributes [Lampert et al.) 2009 Koh al. [020a]] by learning to segment inputs [Losch et al., 2019). In our work, identify concepts by manually selecting the relevant pixels a handful of images and measure the impact of manipulating these features model performance on test set.
Model interventions. Direct manipulations of latent representations inside generative models have been used to create human-understandabll changes in synthesized images [Bau et al. 2019 Jahanian et al., 2019 Goetschalckx et al.) 2019 Shen al., 2020 Härkönen et al. 2020 Wu al. 2020]. Our work is inspired by that line of work as well as recent finding that parameters of generative model can be directly changed to alter generalized behavior [Bau et al. 2020a). Unlike previous work, we edit classification models, changing rules that govern predictions rather than im- age synthesis. Concurrently with our work, there has been a series of methods proposed for editing factual knowledge in language models MMtchell al. 202 I
De Cao et al. 202
Dai et al. 202
Ignoring spurious features. Prior work on preventing models from relying on spurious correla- tions is based on constraining model predictions to satisfy certain invariances. Examples include: training on counterfactuals (either by adding or removing objects from scenes (Shetty et al. 2018
2019 Agarwal et al. 2020 or having human annotators edit text input (Kaushik et al. 2019

---

learning representations that are simultaneously optimal across domains (Arjovsky et al., 2019), ensuring comparable performance across subpopulations [Sagawa et al., 2020] or enforcing consis- tency across inputs that depict the same entity [Heinze-Deml and Meinshausen 2017]. In this work, we focus on a setring where the model designer is aware of undesirable correlations learned by the model and we provide the tools to rewrite them directly.
Model robustness. A long line of work has been devoted to discovering and correcting failure modes of models. These studies focus on simulating variations in testing conditions that can arise during deployment, including: adversarial or natural input corruptions [Szegedy et al. [0114] Fawzi and Frossard 2015, Fawzi et al., 2016 Engstrom et al., 2019h, Ford et al.) 2019] Hendrycks and
Dietterich] 2019 Kang ct al. 2019], changes in the data collection process [Saenko et al. 2010
Torralba and Eiros 2011 Khosla et al. 2012 Tommasi and Tuytelaars 2014 Recht et al. 2019]
or variations in the data subpopulations present [eerry et al. 2018 Oren et al. 2019 Sagawa et al.
2020 Santurkar al.
2020b]. Typical approaches for improving robustness in 202T Koh et al.
these contexts include robust optimization [Madry et al. 2018 Yin et al. 2019 Sagawa et al. 2020)
and data augmentation schemes [Lopes al. 2019 Hendrycks et al. |211 Zhang al. |022 Our
rule-discovery and editing pipelines can be viewed as complementary to this work as they allows us to preemptively adjust the model's prediction rules in anticipation of deployment conditions.
Domain adaptation. The goal of domain adaptation is to adapt a model to a specific deployment environment using (potentially unlabeled) samples from it. This is typically achieved by either fine-tuning the model on the new domain [Donahue et al.) 2014 [hharif Razavian et al. [0144] Kumar
2020 learning the correspondence between the source and target domain, often a latent rep- et al.
resentation space [Ben-David et al. 2007 (aenko et al. 2010 Ganin and Lempitsky [0015] Courty
let al. 2016 Gong et al.] 2016]. or updating the model's batch normalization statisties [Li et al.)
2016 Burns and Steinhardi 202T]. These approaches all require a non-trivial amount of data from the target domain. The question of adaptation from a handful of samples has been explored [Motiaan iet al. 2017]. but in setting that requires samples across all target classes. In contrast, our method allows for generalization new (potentially unknown) classes with even a single example.
7 Conclusion
We developed general toolkit for performing targeted post hoc modifications to vision classifiers. Crucially, instead of specifying the desired behavior implicitly via the training data, our method al- lows users to directly edit the model's prediction rules. By doing so, our approach makes it easier for users to encode their prior knowledge and preferences during the model debugging process. Ad- ditionally, a key benefit of this technique is that it fundamentally changes how the model processes a given concept-thus making it possible to edit its behavior beyond the specific class(es) used for editing. Finally, our edits do not require any additional data collection: they can be guided by few as a single (synthetically-cree -reatted))) exemplar. We believe that this primitive opens up new avenues to interact with and correct our models before or during deployment.
Limitations and broader impact.
Even though our methodology provides a general tool for model editing, performing such edits does require manual intervention and domain expertise. After all, the choice of what concept to edit- and its implications on the robustness of the model-lies with the model designer. For instance, in the vehicles-on-snow example, our objective was to have the model recognize any vehicle on snow the same way it would on a regular road-c.... to adapt a system to different weather conditions. However, if our dataset contains classes for which the presence snow is absolutely essential for recognition, this might not be an appropriate edit to perform.
Moreover, direct model editing is a departure from the standard way in which models are trained, and may have broader implications. While we have shown how it can be used cause beneficial changes in pre-trained models, direct control of prediction rules could also make it easier for adver- saries to introduce vulnerabilities into the model (e.g., by manipulating model behavior on a specific population demographic). Overall, direct model editing makes it clearer than ever that our models are a reflection of the goals and biases of we who create them-not only through the training tasks we choose, but now also through the rules that we rewrite.
10

---

Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers for their helpful comments and feedback.
Work supported in part by the NSF grants CCF-1553428 and CNS-1815221, the DARPA SAIL- ON 1-2001-200 grant, Open Philanthropy, Google PhD fellowship, and Facebook PhD fellowship. This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001 20C00115
Research was sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1 1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed im- plied, of the United States Air Force or the U.S. Govermment. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstandding any copyright notation herein.
References
V. Agarwal, R. Shetty, and M. Fritz. Towards causal vqa: Revealing and reducing spurious correlations by invariant and covariant semantic editing. In Proceedings of the IEFF/CVF Conference Computer Vision and Puttern Recognition, pages 9690-9699 9698, 2020.
M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. 19Xiv:9907.02893 2019.
Invariant risk minimization. arXiv preprint
A. Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang. D. Gutfreund, J. Tenenbaum, and B. Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Neural Information Processing Systems (NeurIPS), 2019.
D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Computer Vision and Pattern Recognition (CVPR), 2017.
D. Bau. J.-Y. Zhu, Strobelt, B. Zhou, B. Tenenbaum, W. Freeman, and A. Torralba. Gan dissection: Visualizing and understanding generative adversarial networks. In International Conference on Learning Representations (ICLR), 2019.
D. Bau, S. Liu, T. Wang, J.-Y. Zhu, and A. Torralba. Rewriting a deep generative model. In European Confer- ence Computer Vision (ECCV), 2020a.
D. Bau, J.-Y Zhu, H. Strobelt, A. Lapedriza, B. Zhou, and A. Torralba. Understanding the role of individual units in deep neural network. Proceedings of the National Academy of Sciences (PNAS), 2020b.
S. Beery, G. Van Horn, and Perona. Recognition in terra incognita. In European Conference Computer Vision (ECCV), 2018.
Ben-David, J. Blitzer, K. Crammer, and Pereira. Analysis of representations for domain adaptation. In Neural Information Processing Systems (NeurIPS), 2007."
L. Beyer, O. J. Hénaff, A. Kolesnikov, X. Zhai, and A. v. d. Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07159 2020.
A. Bissoto, E. Valle, and S. Avila. Debiasing skin lesion datasets and models? not so fast. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pages 740-741, 2020.
C. Bums and J. Steinhardt. Limitations of post-hoe feature alignment for robustness. 2021.
L.-C. Chen, G. Papandreou, I. Kokkinos. K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE transactions on pattern analysis and machine intelligence, 2017.
Chen, Bei, and C. Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2020.
N. Courty, R. Flamary, D. Tuia, and A. Rakotomamonjy. Optimal transport for domain adaptation. In Transac- tions Pattern Analysis and Machine Intelligence, 2016.

---

D. Dai. L. Dong, Y. Hao. Z. Sui, and F. Wei. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2104.0866, 2021.
N. De Cao. W. Aziz, and L. Tiov.. Editing factual knowledge in language models. arX1::104.088644 2021.
arXiv preprint
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hicrarchical image databasc. In Computer Vision and Pattern Recognition (CVPR), 2009.
J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference machine learning (ICML), 2014.
I. Engstrom, A. Nyas, S. Santurkar, D. Tsipras, B. Tran, and A. Madry. Adversarial robustness a prior for learned representations. In ArXiv preprint arXiv: 190.00095, 2019a.
L. Engstrom, B. Tran, D. Tsipras, L. Schmidt, and A. Madry. Exploring the landscape of spatial robustness. In International Conference on Machine Learning (ICML), 2019b.
201L),
D. Erhan, Y. Bengio, A. Courville, and P. Vincent. Visualizing higher-layer features of a deep network. 2009.
A. Fawzi and P. Frossard. Manitest: Are classifiers really invariant? In British Machine Vision Conference (BMVC), 2015,
A. Fawzi, S.-M. Moosavi-Dezfooli, and P. Frossard. Robustness of classifiers: from adversarial to random noise. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
N. Ford, J. Gilmer, N. Carlini, and D. Cubuk. Adversarial examples are a natural consequence of test eroor in noise. In arXiv preprint arXiv: 1901. 10513, 2019.
Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. 2015,
L. A. Gatys, Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In computer vision and pattern recognition (CVPR), 2016.
R. Geirhos, J.-H. Jacobsen, C. Michaelis, Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2020.
G. Ghiasi, H. Lee, M. Kudlur, V. Dumoulin, and J. Shlens. Exploring the structure of real-time, arbitrary neural artistic stylization network, arXiv preprint arXiv: 0660060000 2017.
R. Girshick, 1. Radosavovic, G. Gkioxari, P. Dollár, and K. He. Detectron. htttp:///githbb com/
facebooiresearcdd/tteeetemm 2018.
L. Goetschalekx, A. Andonian, A. Oliva, and P. Isola. Ganalyze: Toward visual definitions of cognitive image properties. In Proceedings of IEEE/CVF International Conference on Computer Vision, pages 5744- 5753, 2019.
G. Goh, Cammarata, Voss, S. Carter, M. Petrov, L. Schubert, A. Radford, and C. Olah. Multimodal neurons in artificial neural networks. Distill, 2021.
M. Gong, Zhang, T. Liu, D. Tao, C. Glymour, and B. Schölkopf. Domain adaptation with conditional transferable components. In International conference on machine learning (ICML., 2016.
Y. Goyal, A. Feder, U. Shalit, and B. Kim. Explaining classifiers with causal concept effect (cace). arXiv preprint 1907.079 2019a.
Y. Goyal, Z. Wu, J. Emst, D. Batra, D. Parikh, and S. Lee. Counterfactual visual explanations. arXiv preprint arXiv:1904.07-55 2019b.
A. Gupta, P. Dollar, and Girshick. LVIS: A dataset for large vocabulary instance segmentation. In Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
E. Härkönen, A. Hertzmann, J. Lehtinen, and S. Paris. Ganspace: Discovering interpretable gan controls.
Advances in Neural Information Processing Systems, 33, 2020.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual lcarning for image recognition, 2015.
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
12

---

C. Heinze-Deae Deme and N. Meinshausen. Conditional variance penalties and domain shift robustness. arXiv preprint arXiv: 1469, 2017.
D. Hendrycks and T. G. Dietterich. Benchmarking neural network robustmess common corruptions and surface variations. In International Conference on Learning Representations (ICLR), 2019.
Hendrycks, Mu, E. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanaa. Augmix: A simple data processing method to improve robustness and uncertainty, arXiv preprint 1711v:1112.02781, 2019.
A. Jahanian, L. Chai, and Isola. On the" steerability" of generative adversarial networks. In International Conference on Learning Representations, 2019.
D. Kang, Y. Sun, D. Hendrycks, T. Brown, and J. Steinhardt. Testing robustness against unforeseen adversaries. In ArXiv preprint arxiv: / 190800006, 2019.
D. Kaushik, E. Hovy, and Z. Lipton. Learning the difference that makes difference with counterfactually- augmented data. In International Conference Learning Representations (ICLR), 2019.
A. Khosla, T. Zhou, T. Malisiewicz, European Conference on Computer Vision (ECCV), 2012.
Efros, and Torralba. Undoing the damage of dataset bias. In
B. Kim, M. Wattenberg, Gilmer, C. Cai, Wexler, Viegas, Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning (ICML), 2018.
P. W. Koh, T. Nguyen, Y. Tang, S. Mussmann, E. Pierson. B. Kim, and P. Liang. Concept bottleneck models. In International Conference Machine Learning (ICML), 2020a.
W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. Phillips, S. Beery, al. Wilds: benchmark of in in-the-wild distribution shifts. arXiv preprint arXiv:2012.0742.1 2020b.
A. Kumar. T. Ma, and P. Liang. Understanding self-training for gradual domain adaptation. In International Conference on Machine Learning (ICML), 2020.
C. H. Lampert, H. Nickisch, and S. Harmeling. Learning to detect unseen object classes by between between-class attribute transfer. In Computer Vision and Pattern Recognition (CVPR), 2009.
Y. Li, N. Wang, J. Shi, J. Liu, and X. Hou. Revisiting batch normalization for practical domain adaptation. arXiv preprint arXiv:1603.04799 2016.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitickk. Microsoft coco: Common objects in context. European conference on computer vision (ECCV), 2014.
R. G. Lopes, D. Yin, Poole, Gilmer, and F. D. Cubuk. Improving robustness without sacrificing accuracy with patch gaussian augmentation. arXiv preprini 196.01196 2019.
M. Losch, M. Fritz, and B. Schiele. Interpretability beyond classification output: Semantic bottleneck networks. arXiv preprint arXiv: /907. 100.79907...... 10882, 2019.
A. Madry, A. Makelov, L. Schmidt, D. Tsipras. and A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference Learning Representations (ICLR), 2018.
E. Mitchell, C. Lin, A. Bosselut, C. Finn, and C. Manning. Fast model editing scale. arXiv preprint arXIv:2110.11399, 2021.
Motiaa, Q. Jones, M. Iranmanesh, and Doretto. Few-shot adversarial domain adaptation. NIPS, 2017.
C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. In Distill, 2017.
C. Olah, A. Satyanarayan, Johnson, S. Carter, L. Schubert, K. Ye, and A. Mordvintsev. The building blocks of interpretability. In Distill, 2018.
Y. Oren, S. Sagawa, T. Hashimoto, and P. Liang. Distributionally robust language modeling. In Empirical Methods in Natural Language Processing (EMNLP), 2019.
J. Pearl. Causal inference. Causality: Objectives and Assensment, 2010.
13

---

J. Ponce, T. L. Berg. M. Everingham, D. A. Forsyth, M. Hebert, S. Lazebnik, M. Marszalek, C. Schmid. B, C. Russell, A. Torralba, al. Dataset issues in object recognition. In Toward category-level object recognition, 2006.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, al. Learning transferable visual models from natural language supervision. arXiv preprint wow:2103.00000 2021.
B. Recht, R. Roctofs. L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? In Interna- tional Conference on Machine Learning (ICML), 2019.
M. T. Ribeiro, S. Singh, and C. Guestrin. "why should trust you?": Explaining the predictions of any classifier. In International Conference on Knowledge Discovery and Data Mining (KDD), 2016.
A. Rosenfeld, R. Zemel, and J. K. Tsotsos. The elephant in the room. In arXiv preprint urXiv:1808.03305. 2018.
O. Russakovsky. J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma. Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. In International Journal of Computer Vision (IJCV), 2015.
K. Saenko, B. Kulis, M. Fritz, and T. Darrell. Adapting visual category models to new domains. In European conference computer vision (ECCV), 2010.
S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang, Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference Learning Representations, 2020.
S. Santurkar, D. Tsipras, and A. Madry. Breeds: Benchmarks for subpopulation shift. In International Confer- on Learning Representations (ICLR), 2021.
A. Sharif Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. Can features off shelf: an astounding baseline for recognition. In conference computer vision and pattern recognition ( (CVPR) workshops, 2014.
Y. Shen, J. Gu, X. Tang, and B. Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IÉEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9243-9252, 2020.
R. Shetty, M. Fritz, and B. Schiele. Adversarial scene editing: Automatic object removal from weak supervi- sion. In NeurlPS, 2018.
R. Shetty, B. Schiele, and M. Fritz. Not using the car to see the sidewalk--qantiffing and controlling the effects of context in classification and segmentation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 8218-8226, 2019.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
P. Stock and M. Cisse. Convnets and imagenet beyond accuracy: Understanding mistakes and uncovering biases. In European Conference on Computer Vision (ECCV), 2018.
C. Szegedy, W. Zaremba, Sutskever, Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
Tommasi and T. Tuytelaars. testbed for cross-dataset analysis. In European Conference on Computer Vision (ECCV), 2014.
A. Torralaa and A. A. Efros. Unbiased look at dataset bias. In CVPR 2011, 2011.
D. Tsipras. S. Santurkar, L. Engstrom, A. Ilyas, and A. Madry. From imagenet image classification: Contex-
tualizing progress on benchmarks. International Conference on Machine Learning (ICML), 2020.
E. Wong, S. Santurkar, and A. Madry. Leveraging sparse lincar layers for debuggable deep networks. In International Conference on Machine Learning (ICML), 2021.
Z. Wu, D. Lischinski, and E. Shechtman. Stylespace analysis: Disentangled controls for stylegan image gener- ation. In Computer Vision and Pattern Recognition (CVPR). 2020.
K. Xiao, L. Engstrom, A. lyas, and A. Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint 20Xiv:2006.090 09994, 2020.
14

---

D. Yin, R. G. Lopes. J. Shlens, E. D. Cubuk, and J. Gilmer. A fourier perspective on model robustness in computer vision. In Neural Information Processing Systems (NeurlPS), 2019.
M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. 2014.
J. Zhang, M. Marszalek, S. Lazebnik, and C. Schmid. Local features and kernels for classification of texture and object categories: A comprehensive study. In International journal of computer vision, 2007.
L. Zhang, 7. Deng, K. Kawaguchi, A. Ghorbani, and J. Zou. How does mixup help with robustness and generalization? In International Conference on Learning Representations (ICIR), 2021.
B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 2017.
B. Zhou, Y. Sun, D. Bau, and A. Torralba. Interpretable basis dccomposition for visual explanation. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pages 119-134, 2018.
15


Reviews:
[{"review_id": "loSuhw1PqrC", "reviewer": "Reviewer_V8H7", "summary": "This paper proposes a method to directly edit the predictions of a trained neural network by (1) building “counterfactual” images by replacing objects in a scene, then (2) mapping activations from pre-transformed images to post-transformed images through the algorithm proposed in Bau et. al. The method is applied to well-known image datasets to show improvements in accuracy on images where transformed objects are present. In particular, the proposed approach generalizes better than naive baselines of fine-tuning on new data. The authors then apply the method to two real world contexts: driving scenes and typographic errors. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Although none of the individual pieces of the method are new (there are works that replace objects in an image esp. for data augmentation, and much of the method relies on Bau et. al.), I find the method as a whole novel as it enables a new kind of interpretability: users can now audit and correct neural network predictions in a way they could not before. \n\nQuality: I found the ImageNet and Places results to be compelling but the real world experiments less so. I think self-driving cars to be a compelling application of the method but the experiments on vehicles-in-snow represents one small sub-problem. It would be interesting to show more use-cases for self-driving cars. Similarly, I think mitigating spurious correlations is an important problem and one of the motivations for this method (in the introduction). But I found the experiments to be rather simple, and focus on typographic errors. It would be nice to see a larger-scale and more systematic evaluation e.g. Waterbirds dataset. \n\nClarity: The writing quality was good and I found it easy to understand most of the ideas presented. There were areas I got confused as a reader (see questions below for details). Also, the presentation of Bau et. al.’s method could be more clear, especially to readers (like myself) who are not familiar with the original method. \n\nSignificance: Yes, I find the lack of interpretability in neural networks one of the big bottlenecks to widespread adoption. Although the method does not provide an explanation of what neural networks are doing, it does provide a compelling method to correct wrong predictions: a step in the right direction. \n\nQuestions on Section 2:\n- In generating counterfactuals, how do you choose on what axis to modify an input? For instance, in Figure 2, the tree is modified from a snowy forest to a fall forest. But you could have modified it to replace trees with buildings, or something else altogether. How does the user choose the correct (or best) counterfactual?\n- I also wonder how generalizable prediction rules are. A rule like “cars have wheels” is likely a loose approximation of the highly nonlinear decision boundary that the neural network has learned. For some images of cars, perhaps the network relies on wheel features whereas on others, it does not (could be texture / style features). Is there a way to measure how “faithful” my prediction rule is to the neural network?\n- I imagine the success of probing model behavior relies heavily on the quality of the instance segmentation. Is it somewhat circular to rely on pretrained segmentation models to identify concepts, which themselves have spurious correlations (that need to be edited out). Is this a chicken and egg problem? \n- Similarly, how important is it for the data distribution used to train a segmentation model to be “aligned” with the training data distribution? I imagine if there is a large mismatch, poor segmentation affects the rest of the pipeline. Would be nice to see how sensitive this algorithm is to distribution differences. \n- In Figure 3 experiments, how do you pick what a class is transformed to? Specifically, in the example that transformations of “grass” affect accuracy of predicting “croquet ball”, what does “grass” get replaced by? A random object from a random image? \n- In Figure 3, what does style mean? Why does changing style affect accuracy much more (~20% vs ~5%) than changing concept?\n- How much of the drop in classification accuracy in Figure 3 is due to counterfactuals being out of domain (e.g. looking very different from training distribution), versus detecting model sensitivities? For instance, replacing “grass” with “megalith” likely creates an image that is fairly out of domain, and which the model should not do well on.  How do you differentiate between these two kinds of errors? Similarly, when replacing  objects, do you have to be conscious of lightning, camera angle, resolution, object size, etc. Otherwise, you may run into OOD problems again. How do you filter out cases in which the counterfactual is not (visually) reasonable?\n- It makes sense that croquet ball and grass are correlated and hence performance suffers. But I might also expect this for “wood rabbit” (e.g. rabbits usually appear on grass) where it is affected much less. Why is this? What is the relationship between feature correlation and accuracy drop? \n\nQuestions on Section 3:\n- Is S in Equation 1 derived from the segmentation mask? \n- I like the use of Bau et. al. [5] for editing classifiers by key-value replacement. Clever!\n- Personally, I am not extremely familiar with Bau et. al. and would love some exposition on why it is that the method can replace objectives without changing the model’s behaviors in other contexts. How is this latter statement guaranteed? \n- How do you optimize Equations 1 & 2? In particular, how do you compute k*_ij and how do you enforce the constraint (e.g. Lagrangian?). Some discussion on the optimization complexities would be useful. \n- In figure 4’s caption, it is stated that the weights W are updated to enforce a new key-value association. This makes sense but when adding multiple key-value associations, do you have to optimize W to enforce them all at once? I imagine you cannot do this sequentially as W may no longer enforce previous associations when optimized to satisfy a new association. If this is true, does it get very difficult to enforce more and more associations? Are there sets of associations in which a single weight W cannot satisfy? \n\nQuestions on Section 4+5:\n- Results in Figure 5 are very cool. It makes sense that the finetuning outperforms editing on the target class. It was nice to see that editing generalizes better to unseen examples with the transformed concept whereas fine-tuning overfits. The observation that methods generalize more easily to different styles is interesting. \n- I wasn’t clear on the single synthetic exemplar description: does this mean editing was applied in a one-shot scenario? Does this mean fine-tuning is done on a single example as well? If so, it would be interesting to finetune on more examples. For example, is it possible to show that editing is as good as fine-tuning on N new data points? \n- I think the iPod example is really cool but would like to see a larger study on a more well-known dataset of spurious correlations. (In particular, Figure 6b suggests that there are only ~6 images?). Would be compelling and interesting to apply this method to the Waterbirds dataset, which has more subtle spurious correlations. \n\nNit: \n- Typo on page 9 line 270 (“neurons” duplicated). \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I found the limitations and broad impact section to be vague and should be expanded upon. The amount of “manual intervention” and “domain expertise” should be quantified. I mentioned this above but I would like to see effects from the (1) quality of the segmentation model, (2) quality of the counterfactual images. The “broader implications” of direct model editing should also be elaborated upon. For example, what does this enable a malicious actor to do? What kinds of vulnerabilities or new adversarial examples are now possible? ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "9Uu-ev0_7C2", "reviewer": "Reviewer_4Ajg", "summary": "The paper presents a method to improve robustness of object classifiers to style-changes in components, termed “concepts” in the paper, which can include both background elements as well as parts of the object of interest. This is done by first localizing confounding concepts using existing instance segmentation methods, performing style-transfer on the localized concept using existing methods, and then using an existing method which performs a rank-1 update to (a layer of) network parameters to map the transformed-concept-region to the same value as the original concept-region was mapped to by the layer. Experiments illustrate that this pipeline is effective at making the network robust to style-changes for concepts identified to be potentially misleading.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The objective of the proposed toolkit seems potentially useful: as the paper says, “after identifying undesirable prediction rules, a user might want to modify them before deploying their model in the wild”. Synthetic experiments indicate that it is possible to significantly improve robustness to style-changes, both for the trained transformations as well as held-out styles.\n\nIt seems that the setup being considered is one where one does not have the budget to re-train (otherwise a baseline that emerges naturally is to augment the training set, perhaps on the fly, with the style-transferred examples). Perhaps this point should be motivated and made clearer somewhere in the paper.\n\nTwo realistic examples of the toolkit-in-action are demonstrated: using a synthetic exemplar of a vehicle on snowy roads, improved generalization to vehicles on actual snowy roads is obtained; a possible application to avoid typographic attacks is shown, by editing out the specific typographic attack using counterfactual images. While these are interesting demonstrations, the test sets seem to be on the smaller side - the snowy roads test set seems to contain around 120 examples, and the typography-attack set seems to consist of 6 (x2) pictures. One way to make the toolkit developed in the paper more compelling might be to showcase improvements in challenging test-sets such as Imagenet-A [1], where it is suggested that at least some of the images exacerbate generalization failure due to over-reliance on specific features.\n\nFor the fine-tuning baseline, was there a reason the number of steps was capped at 800? Could performance improve with more steps? (Especially given that the number of optimization steps for the proposed method seems capped at a much higher 80000.)\n\nMinor suggestions:\nPerhaps “concept localization” might be a better term instead of “concept identification” in L69; at first glance I interpreted it to mean the process of discovering misleading concepts.\n\nOn the whole, I think the paper presents a technique that can potentially be inspirational, or of use to practitioners in some areas. However, some of the underlying assumptions (i.e. no re-training budget, but continued access to data and ability to perform style-transfers) might be too narrow to be realistic. I encourage the authors to add a comment or two describing the specifics of situations where such a pipeline would be a natural choice, to help the paper find its right audience.\n\n\n[1] Natural Adversarial Examples, Hendrycks et al., 2019\n\n------------\nI have read the rebuttal, and stand by my rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and potential societal impact have been adequately discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "6vNWW_HFIWf", "reviewer": "Reviewer_VmAm", "summary": "The authors propose a model editing method that proceeds in two steps. First, they rely on off-the-shelf segmentation models and style transfer models to learn a data transformation $x \\to x'$ that preserves semantic meaning (e.g., in a vehicle recognition task, they might convert a snowy road into a non-snowy road, which preserves the label of the vehicle). Second, they build on recent work by Bau et al. to use these transformed pairs $(x,  x')$ to learn an edit to the model $f$ such that, roughly speaking, $f(x)$ and $f(x')$ agree in their representations at some particular layer and above. They show that their procedure generalizes (astonishingly) well from edits learned on just a single exemplar. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the paper interesting and well-written. Thank you to the authors for this submission! While each component of their pipeline is not \"new\", in the sense that they adopt the model editing method from recent work and use off-the-shelf models otherwise, the overall idea and their analysis is still interesting and novel. I think this will be a positive contribution to the research community.\n\n1. My main feedback is around the framing of the problem setting and how the method requires \"essentially no additional data collection\" (L30) or \"any annotation effort\" (L81). Related to this concern is that there are potentially stronger baselines that could be run against the authors' model editing procedure. \n\nConcretely, it seems that the authors consider two related but distinct settings:\n\na) In Sections 2 and 4, the authors describe an automated setting that relies on off-the-shelf segmentation models to segment different concepts (\"grass\", \"sea\") and then off-the-shelf style transfer models to transform images according to these concepts (e.g., converting grass in one image to sea). Once the library of concepts and the set of acceptable transformations are established, this is an essentially automatic process that can be scaled to multiple images.\n\nb) In Sections 5, the authors describe a manual setting where they use prior knowledge of the test distribution to construct a single example (pair) that can then be used to learn a model edit.\n\nThese settings are a bit conflated in the paper, but they seem distinct. In particular, if we are in the scalable setting (a), where we're learning transformations through style transfer, then it is a bit unclear to me why we are restricting ourselves to experiments in Section 4 with a small number of exemplars ($N$ is not specified in the main text, I think, but from the appendix it seems to be $N=3$ for Figure 5) and with exemplars from only one class. The authors show that in this setting, model editing outperforms finetuning (for example, the performance gains from model editing extend to other classes), but it's unclear to me why we're in this setting in the first place. If we can just generate more examples from more classes using the same automated rules, then from Figure 5, it seems like finetuning could perform significantly better than model editing. \n\nBesides finetuning, another natural baseline would also be to use the learned data transformation (e.g., grass -> sea) and then plug it in to any number of contemporary methods that use these kinds of data transformations, such as contrastive learning, consistency regularization, or even plain data augmentation etc. It'd be a bit surprising if those methods didn't outperform model editing given sufficient data. If there's a reason why it's hard to get \"sufficient data\" in setting (a), for example because it's computationally too intensive to construct these transformed images or something like that, that would be good to discuss.\n\nSetting (b) seems like a more compelling demonstration of model editing, since it only relies on a single exemplar. But this setting is introduced only later in the paper, and doesn't seem to be the focus of the first few sections. Even in Section 5, it was unclear why we were in setting (b), since if we come up with the rewrite rule snow -> road, it seems like it'd be straightforward to generate many more than one example. What happens when we have more than one example?\n\nOverall, could the authors clarify precisely what constraints and problem setting we're operating in, the motivation for it, and the reasoning for the baselines chosen?\n\n---\n\nOther minor comments: \n\n2. The \"no manual annotation\" claims could be more nuanced. First, the off-the-shelf models were trained on extensive manual annotations, so this method only works \"without manual annotations\" on datasets where good off-the-shelf models have already been trained (with manual annotations). Second, identifying a reasonable set of concepts (e.g., disallowing the dog concept from being transformed, as the authors note) is also a form of manual annotation, albeit much less labor intensive. \n\n3. It might be worth discussing work in the interpretability literature that learn models which explicitly encode concepts. Examples of this include \nrecent papers on concept whitening models (https://arxiv.org/abs/2002.01650) and concept bottleneck models (https://arxiv.org/abs/2007.04612), which also discuss intervening on concepts but in a different way from the proposed method. Another example is the work on semantic bottleneck models (https://arxiv.org/abs/1907.10882), which consider semantic segmentations as well, and study the effect of intervening on those segmentations. Broadly speaking, these are all similar in the sense of using an existing library of concepts to understand and intervene on model behavior, though the authors' proposed method here is still distinct and novel.\n\n4. The two components of the pipeline (segmentation + style transfer, and then model editing) seem like they are somewhat independent. For example, one could do model editing to remove spurious correlations using external metadata (e.g., if you know the race of the person in a photo and you want to edit the model to map people of different races into the same representation; or if perhaps you know that you want to be robust to backgrounds as in https://arxiv.org/abs/2006.09994). As a suggestion, discussing this might be helpful for broadening the scope of the proposed method. \n\n---\nUpdate: Thanks to the authors for the response. I would like to keep my score. I think the paper contains interesting ideas, though I agree with the other reviewers that the setting could be clarified and the experiments made more rigorous.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "-z6szAPI9Ag", "reviewer": "Reviewer_MsmY", "summary": "The paper proposes an algorithm to change the behaviour of a trained model in the presence of some external interference in the presence of an external concept. The paper does it without using extra data in two stages. First, the algorithm utilises a rule detection framework using counterfactuals and then changes it using the technique from bau et. al. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality\n* The paper is largely based on existing works including Bau et. al. [5]. Bau et. al [5] work with generative models whereas the authors here look at classification models. But, the main technique of how to insert a new prediction rule is almost exactly similar.\n\n## Significance\n\n* There is very little systematic quantification of the success of the algorithm. For example, when style editing is done on a large scale on all images of a class for one concept, does it make sense to expect a correct classification of the transformed images ? Does the style/concept editing actually produce proper images for all ? I am not sure what a systematic way to measure success would be but this lack of a proper way to do a systematic study probably originates from the absence of a proper formulation of the problem. While there doesn't need to be a theoretical understanding, I think there needs to be a rigorous problem formulation regarding what are _concepts_ what are _rules_ and what this method is supposed to achieve. \n* There are also no explanations for some of the observations eg. In Line 664 in the supplementary the authors state that \" the accuracy of a ResNet-50 ImageNet classifier 664 drops by more than 30% on the class “three-toed sloth” when “tree”s in the image are modified, while the accuracy of a VGG16 model drops by less than 5% under the same setup.\" What makes resnets so vulnerable  ? This seems like a weird phenomenon and should be explored further.\n* There is no discussion of the limitations of the algorithm. For example, how many rules can be changed simultaneously ? What kind of rules are easier to change and why ? The _application to real world_ is also a very particular example of using a _snowy_ weather.\n* I think some of the claims of the paper should be downplayed a bit. For example in the abstract, the authors state _\"including adapting a model to new environments, and modifying it to ignore spurious features\"_. But the paper doesn't show how to detect and remove spurious features (in addition to not actually defining what a spurious feature is\". Also, there is no example of adapting the algorithm to new environments (again no definition of what an environment actually is). The abstract makes it sound like the paper actually achieves all of this, which it doesn't, at this state.\n \n\n\nOverall, there are a lot of different observations of various scales in the paper which can pose interesting questions. But the efficacy of the algorithm has mainly been demonstrated in the main text through anecdotal examples  and the problem lacks a proper formulation, which I think is necessary to distinguish between success and failure of the algorithm. Thus, I am unwilling to vote for acceptance of this paper at  the level of NeurIPS.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a model to edit a classifier's decision boundary, in order to enforce invariance of the classifier to regions in the image that are known a priori to not change the semantic label (i.e. a car on a road should still be classified as a car on a snowy road). They proceed in two steps: first they rely on segmentation models to isolate concepts (e.g. road) and style transfer to transform such concepts (e.g. road -> snowy road) to obtain augmentations of the input image. Taking inspiration from previous work, they learn a "translation" between inner features of the base classifier corresponding to the concept in the transformed image, to the corresponding features of the original image, effectively making the transformed image behave like the original one. Experiments show that the procedure generalizes well, even when just learning from very few transformed exemplars.

--

Reviewers recognized the potential impact of the proposed approach in real-world scenarios and the strength of the shown results in a harder few-shot generalization setting (the method is robust on modifications for classes held out during training). A reviewer found this work "thought-provoking" and seen "as a start of a line of research that could be fruitful". All reviewers also recognized the main criticisms of this paper, which turn around: (a) the overall lack of rigor in the problem definition; (b) it's rather unclear what are the limitations of this approach and when the approach may fail; (c) the somewhat anecdotal nature of the experiments, which test for specific concept-rule pairs rather than a more systematic evaluation.

Regarding (a), reviewers found the paper a bit hand-wavy at times, for example, there's no definition of what is meant by "rule". Regarding (b), one reviewer found particularly critical that the paper didn't address the question "how many rules can be changed simultaneously?", given that the paper only deals with changing one rule at a time. Authors are strongly encouraged to think about adding an experiment illustrating this point in the final revision, for example, by verifying performance when training multiple rule translations. During rebuttal, I discussed with the authors a specific experimental scenario that can illustrate a clear failure of this model. I hope the results of our discussion will be included in the paper. Regarding (c), a reviewer acknowledged that the lack of large scale systematic evaluation might also be "due to lack of datasets and the nature of the problem being studied". To make evaluation stronger in terms of baselines, I suggest the authors to incorporate also additional standard contrastive/constitency-based regularization methods, as suggested by a reviewer.

Overall, even if reviewers were not unanimous about acceptance, I believe that the results and use cases showcased in this paper can bring excitement and inspire future work and outweigh the criticisms expressed above. I recommend this paper for acceptance. I suggest the authors do their best to address the criticisms of the reviewers (with a special effort for those who were the most negative) and incorporate the aforementioned suggestions in their final version.
Decision: Accept (Poster)