Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a model editing method that proceeds in two steps. First, they rely on off-the-shelf segmentation models and style transfer models to learn a data transformation $x \to x'$ that preserves semantic meaning (e.g., in a vehicle recognition task, they might convert a snowy road into a non-snowy road, which preserves the label of the vehicle). Second, they build on recent work by Bau et al. to use these transformed pairs $(x,  x')$ to learn an edit to the model $f$ such that, roughly speaking, $f(x)$ and $f(x')$ agree in their representations at some particular layer and above. They show that their procedure generalizes (astonishingly) well from edits learned on just a single exemplar. 
main_review: I found the paper interesting and well-written. Thank you to the authors for this submission! While each component of their pipeline is not "new", in the sense that they adopt the model editing method from recent work and use off-the-shelf models otherwise, the overall idea and their analysis is still interesting and novel. I think this will be a positive contribution to the research community.

1. My main feedback is around the framing of the problem setting and how the method requires "essentially no additional data collection" (L30) or "any annotation effort" (L81). Related to this concern is that there are potentially stronger baselines that could be run against the authors' model editing procedure. 

Concretely, it seems that the authors consider two related but distinct settings:

a) In Sections 2 and 4, the authors describe an automated setting that relies on off-the-shelf segmentation models to segment different concepts ("grass", "sea") and then off-the-shelf style transfer models to transform images according to these concepts (e.g., converting grass in one image to sea). Once the library of concepts and the set of acceptable transformations are established, this is an essentially automatic process that can be scaled to multiple images.

b) In Sections 5, the authors describe a manual setting where they use prior knowledge of the test distribution to construct a single example (pair) that can then be used to learn a model edit.

These settings are a bit conflated in the paper, but they seem distinct. In particular, if we are in the scalable setting (a), where we're learning transformations through style transfer, then it is a bit unclear to me why we are restricting ourselves to experiments in Section 4 with a small number of exemplars ($N$ is not specified in the main text, I think, but from the appendix it seems to be $N=3$ for Figure 5) and with exemplars from only one class. The authors show that in this setting, model editing outperforms finetuning (for example, the performance gains from model editing extend to other classes), but it's unclear to me why we're in this setting in the first place. If we can just generate more examples from more classes using the same automated rules, then from Figure 5, it seems like finetuning could perform significantly better than model editing. 

Besides finetuning, another natural baseline would also be to use the learned data transformation (e.g., grass -> sea) and then plug it in to any number of contemporary methods that use these kinds of data transformations, such as contrastive learning, consistency regularization, or even plain data augmentation etc. It'd be a bit surprising if those methods didn't outperform model editing given sufficient data. If there's a reason why it's hard to get "sufficient data" in setting (a), for example because it's computationally too intensive to construct these transformed images or something like that, that would be good to discuss.

Setting (b) seems like a more compelling demonstration of model editing, since it only relies on a single exemplar. But this setting is introduced only later in the paper, and doesn't seem to be the focus of the first few sections. Even in Section 5, it was unclear why we were in setting (b), since if we come up with the rewrite rule snow -> road, it seems like it'd be straightforward to generate many more than one example. What happens when we have more than one example?

Overall, could the authors clarify precisely what constraints and problem setting we're operating in, the motivation for it, and the reasoning for the baselines chosen?

---

Other minor comments: 

2. The "no manual annotation" claims could be more nuanced. First, the off-the-shelf models were trained on extensive manual annotations, so this method only works "without manual annotations" on datasets where good off-the-shelf models have already been trained (with manual annotations). Second, identifying a reasonable set of concepts (e.g., disallowing the dog concept from being transformed, as the authors note) is also a form of manual annotation, albeit much less labor intensive. 

3. It might be worth discussing work in the interpretability literature that learn models which explicitly encode concepts. Examples of this include 
recent papers on concept whitening models (https://arxiv.org/abs/2002.01650) and concept bottleneck models (https://arxiv.org/abs/2007.04612), which also discuss intervening on concepts but in a different way from the proposed method. Another example is the work on semantic bottleneck models (https://arxiv.org/abs/1907.10882), which consider semantic segmentations as well, and study the effect of intervening on those segmentations. Broadly speaking, these are all similar in the sense of using an existing library of concepts to understand and intervene on model behavior, though the authors' proposed method here is still distinct and novel.

4. The two components of the pipeline (segmentation + style transfer, and then model editing) seem like they are somewhat independent. For example, one could do model editing to remove spurious correlations using external metadata (e.g., if you know the race of the person in a photo and you want to edit the model to map people of different races into the same representation; or if perhaps you know that you want to be robust to backgrounds as in https://arxiv.org/abs/2006.09994). As a suggestion, discussing this might be helpful for broadening the scope of the proposed method. 

---
Update: Thanks to the authors for the response. I would like to keep my score. I think the paper contains interesting ideas, though I agree with the other reviewers that the setting could be clarified and the experiments made more rigorous.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposes an algorithm to change the behaviour of a trained model in the presence of some external interference in the presence of an external concept. The paper does it without using extra data in two stages. First, the algorithm utilises a rule detection framework using counterfactuals and then changes it using the technique from bau et. al. 
main_review: ## Originality
* The paper is largely based on existing works including Bau et. al. [5]. Bau et. al [5] work with generative models whereas the authors here look at classification models. But, the main technique of how to insert a new prediction rule is almost exactly similar.

## Significance

* There is very little systematic quantification of the success of the algorithm. For example, when style editing is done on a large scale on all images of a class for one concept, does it make sense to expect a correct classification of the transformed images ? Does the style/concept editing actually produce proper images for all ? I am not sure what a systematic way to measure success would be but this lack of a proper way to do a systematic study probably originates from the absence of a proper formulation of the problem. While there doesn't need to be a theoretical understanding, I think there needs to be a rigorous problem formulation regarding what are _concepts_ what are _rules_ and what this method is supposed to achieve. 
* There are also no explanations for some of the observations eg. In Line 664 in the supplementary the authors state that " the accuracy of a ResNet-50 ImageNet classifier 664 drops by more than 30% on the class “three-toed sloth” when “tree”s in the image are modified, while the accuracy of a VGG16 model drops by less than 5% under the same setup." What makes resnets so vulnerable  ? This seems like a weird phenomenon and should be explored further.
* There is no discussion of the limitations of the algorithm. For example, how many rules can be changed simultaneously ? What kind of rules are easier to change and why ? The _application to real world_ is also a very particular example of using a _snowy_ weather.
* I think some of the claims of the paper should be downplayed a bit. For example in the abstract, the authors state _"including adapting a model to new environments, and modifying it to ignore spurious features"_. But the paper doesn't show how to detect and remove spurious features (in addition to not actually defining what a spurious feature is". Also, there is no example of adapting the algorithm to new environments (again no definition of what an environment actually is). The abstract makes it sound like the paper actually achieves all of this, which it doesn't, at this state.
 


Overall, there are a lot of different observations of various scales in the paper which can pose interesting questions. But the efficacy of the algorithm has mainly been demonstrated in the main text through anecdotal examples  and the problem lacks a proper formulation, which I think is necessary to distinguish between success and failure of the algorithm. Thus, I am unwilling to vote for acceptance of this paper at  the level of NeurIPS.
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a method to improve robustness of object classifiers to style-changes in components, termed “concepts” in the paper, which can include both background elements as well as parts of the object of interest. This is done by first localizing confounding concepts using existing instance segmentation methods, performing style-transfer on the localized concept using existing methods, and then using an existing method which performs a rank-1 update to (a layer of) network parameters to map the transformed-concept-region to the same value as the original concept-region was mapped to by the layer. Experiments illustrate that this pipeline is effective at making the network robust to style-changes for concepts identified to be potentially misleading.
main_review: The objective of the proposed toolkit seems potentially useful: as the paper says, “after identifying undesirable prediction rules, a user might want to modify them before deploying their model in the wild”. Synthetic experiments indicate that it is possible to significantly improve robustness to style-changes, both for the trained transformations as well as held-out styles.

It seems that the setup being considered is one where one does not have the budget to re-train (otherwise a baseline that emerges naturally is to augment the training set, perhaps on the fly, with the style-transferred examples). Perhaps this point should be motivated and made clearer somewhere in the paper.

Two realistic examples of the toolkit-in-action are demonstrated: using a synthetic exemplar of a vehicle on snowy roads, improved generalization to vehicles on actual snowy roads is obtained; a possible application to avoid typographic attacks is shown, by editing out the specific typographic attack using counterfactual images. While these are interesting demonstrations, the test sets seem to be on the smaller side - the snowy roads test set seems to contain around 120 examples, and the typography-attack set seems to consist of 6 (x2) pictures. One way to make the toolkit developed in the paper more compelling might be to showcase improvements in challenging test-sets such as Imagenet-A [1], where it is suggested that at least some of the images exacerbate generalization failure due to over-reliance on specific features.

For the fine-tuning baseline, was there a reason the number of steps was capped at 800? Could performance improve with more steps? (Especially given that the number of optimization steps for the proposed method seems capped at a much higher 80000.)

Minor suggestions:
Perhaps “concept localization” might be a better term instead of “concept identification” in L69; at first glance I interpreted it to mean the process of discovering misleading concepts.

On the whole, I think the paper presents a technique that can potentially be inspirational, or of use to practitioners in some areas. However, some of the underlying assumptions (i.e. no re-training budget, but continued access to data and ability to perform style-transfers) might be too narrow to be realistic. I encourage the authors to add a comment or two describing the specifics of situations where such a pipeline would be a natural choice, to help the paper find its right audience.


[1] Natural Adversarial Examples, Hendrycks et al., 2019

------------
I have read the rebuttal, and stand by my rating.
limitations_and_societal_impact: Limitations and potential societal impact have been adequately discussed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a method to directly edit the predictions of a trained neural network by (1) building “counterfactual” images by replacing objects in a scene, then (2) mapping activations from pre-transformed images to post-transformed images through the algorithm proposed in Bau et. al. The method is applied to well-known image datasets to show improvements in accuracy on images where transformed objects are present. In particular, the proposed approach generalizes better than naive baselines of fine-tuning on new data. The authors then apply the method to two real world contexts: driving scenes and typographic errors. 
main_review: Originality: Although none of the individual pieces of the method are new (there are works that replace objects in an image esp. for data augmentation, and much of the method relies on Bau et. al.), I find the method as a whole novel as it enables a new kind of interpretability: users can now audit and correct neural network predictions in a way they could not before. 

Quality: I found the ImageNet and Places results to be compelling but the real world experiments less so. I think self-driving cars to be a compelling application of the method but the experiments on vehicles-in-snow represents one small sub-problem. It would be interesting to show more use-cases for self-driving cars. Similarly, I think mitigating spurious correlations is an important problem and one of the motivations for this method (in the introduction). But I found the experiments to be rather simple, and focus on typographic errors. It would be nice to see a larger-scale and more systematic evaluation e.g. Waterbirds dataset. 

Clarity: The writing quality was good and I found it easy to understand most of the ideas presented. There were areas I got confused as a reader (see questions below for details). Also, the presentation of Bau et. al.’s method could be more clear, especially to readers (like myself) who are not familiar with the original method. 

Significance: Yes, I find the lack of interpretability in neural networks one of the big bottlenecks to widespread adoption. Although the method does not provide an explanation of what neural networks are doing, it does provide a compelling method to correct wrong predictions: a step in the right direction. 

Questions on Section 2:
- In generating counterfactuals, how do you choose on what axis to modify an input? For instance, in Figure 2, the tree is modified from a snowy forest to a fall forest. But you could have modified it to replace trees with buildings, or something else altogether. How does the user choose the correct (or best) counterfactual?
- I also wonder how generalizable prediction rules are. A rule like “cars have wheels” is likely a loose approximation of the highly nonlinear decision boundary that the neural network has learned. For some images of cars, perhaps the network relies on wheel features whereas on others, it does not (could be texture / style features). Is there a way to measure how “faithful” my prediction rule is to the neural network?
- I imagine the success of probing model behavior relies heavily on the quality of the instance segmentation. Is it somewhat circular to rely on pretrained segmentation models to identify concepts, which themselves have spurious correlations (that need to be edited out). Is this a chicken and egg problem? 
- Similarly, how important is it for the data distribution used to train a segmentation model to be “aligned” with the training data distribution? I imagine if there is a large mismatch, poor segmentation affects the rest of the pipeline. Would be nice to see how sensitive this algorithm is to distribution differences. 
- In Figure 3 experiments, how do you pick what a class is transformed to? Specifically, in the example that transformations of “grass” affect accuracy of predicting “croquet ball”, what does “grass” get replaced by? A random object from a random image? 
- In Figure 3, what does style mean? Why does changing style affect accuracy much more (~20% vs ~5%) than changing concept?
- How much of the drop in classification accuracy in Figure 3 is due to counterfactuals being out of domain (e.g. looking very different from training distribution), versus detecting model sensitivities? For instance, replacing “grass” with “megalith” likely creates an image that is fairly out of domain, and which the model should not do well on.  How do you differentiate between these two kinds of errors? Similarly, when replacing  objects, do you have to be conscious of lightning, camera angle, resolution, object size, etc. Otherwise, you may run into OOD problems again. How do you filter out cases in which the counterfactual is not (visually) reasonable?
- It makes sense that croquet ball and grass are correlated and hence performance suffers. But I might also expect this for “wood rabbit” (e.g. rabbits usually appear on grass) where it is affected much less. Why is this? What is the relationship between feature correlation and accuracy drop? 

Questions on Section 3:
- Is S in Equation 1 derived from the segmentation mask? 
- I like the use of Bau et. al. [5] for editing classifiers by key-value replacement. Clever!
- Personally, I am not extremely familiar with Bau et. al. and would love some exposition on why it is that the method can replace objectives without changing the model’s behaviors in other contexts. How is this latter statement guaranteed? 
- How do you optimize Equations 1 & 2? In particular, how do you compute k*_ij and how do you enforce the constraint (e.g. Lagrangian?). Some discussion on the optimization complexities would be useful. 
- In figure 4’s caption, it is stated that the weights W are updated to enforce a new key-value association. This makes sense but when adding multiple key-value associations, do you have to optimize W to enforce them all at once? I imagine you cannot do this sequentially as W may no longer enforce previous associations when optimized to satisfy a new association. If this is true, does it get very difficult to enforce more and more associations? Are there sets of associations in which a single weight W cannot satisfy? 

Questions on Section 4+5:
- Results in Figure 5 are very cool. It makes sense that the finetuning outperforms editing on the target class. It was nice to see that editing generalizes better to unseen examples with the transformed concept whereas fine-tuning overfits. The observation that methods generalize more easily to different styles is interesting. 
- I wasn’t clear on the single synthetic exemplar description: does this mean editing was applied in a one-shot scenario? Does this mean fine-tuning is done on a single example as well? If so, it would be interesting to finetune on more examples. For example, is it possible to show that editing is as good as fine-tuning on N new data points? 
- I think the iPod example is really cool but would like to see a larger study on a more well-known dataset of spurious correlations. (In particular, Figure 6b suggests that there are only ~6 images?). Would be compelling and interesting to apply this method to the Waterbirds dataset, which has more subtle spurious correlations. 

Nit: 
- Typo on page 9 line 270 (“neurons” duplicated). 

limitations_and_societal_impact: I found the limitations and broad impact section to be vague and should be expanded upon. The amount of “manual intervention” and “domain expertise” should be quantified. I mentioned this above but I would like to see effects from the (1) quality of the segmentation model, (2) quality of the counterfactual images. The “broader implications” of direct model editing should also be elaborated upon. For example, what does this enable a malicious actor to do? What kinds of vulnerabilities or new adversarial examples are now possible?
review_title: null

Meta-review: The authors propose a model to edit a classifier's decision boundary, in order to enforce invariance of the classifier to regions in the image that are known a priori to not change the semantic label (i.e. a car on a road should still be classified as a car on a snowy road). They proceed in two steps: first they rely on segmentation models to isolate concepts (e.g. road) and style transfer to transform such concepts (e.g. road -> snowy road) to obtain augmentations of the input image. Taking inspiration from previous work, they learn a "translation" between inner features of the base classifier corresponding to the concept in the transformed image, to the corresponding features of the original image, effectively making the transformed image behave like the original one. Experiments show that the procedure generalizes well, even when just learning from very few transformed exemplars.

--

Reviewers recognized the potential impact of the proposed approach in real-world scenarios and the strength of the shown results in a harder few-shot generalization setting (the method is robust on modifications for classes held out during training). A reviewer found this work "thought-provoking" and seen "as a start of a line of research that could be fruitful". All reviewers also recognized the main criticisms of this paper, which turn around: (a) the overall lack of rigor in the problem definition; (b) it's rather unclear what are the limitations of this approach and when the approach may fail; (c) the somewhat anecdotal nature of the experiments, which test for specific concept-rule pairs rather than a more systematic evaluation.

Regarding (a), reviewers found the paper a bit hand-wavy at times, for example, there's no definition of what is meant by "rule". Regarding (b), one reviewer found particularly critical that the paper didn't address the question "how many rules can be changed simultaneously?", given that the paper only deals with changing one rule at a time. Authors are strongly encouraged to think about adding an experiment illustrating this point in the final revision, for example, by verifying performance when training multiple rule translations. During rebuttal, I discussed with the authors a specific experimental scenario that can illustrate a clear failure of this model. I hope the results of our discussion will be included in the paper. Regarding (c), a reviewer acknowledged that the lack of large scale systematic evaluation might also be "due to lack of datasets and the nature of the problem being studied". To make evaluation stronger in terms of baselines, I suggest the authors to incorporate also additional standard contrastive/constitency-based regularization methods, as suggested by a reviewer.

Overall, even if reviewers were not unanimous about acceptance, I believe that the results and use cases showcased in this paper can bring excitement and inspire future work and outweigh the criticisms expressed above. I recommend this paper for acceptance. I suggest the authors do their best to address the criticisms of the reviewers (with a special effort for those who were the most negative) and incorporate the aforementioned suggestions in their final version.
Decision: Accept (Poster)