Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This submission aims to improve the robustness of vision transformers (ViT) by leveraging interpretability methods during training. The proposed approach relies on a recent method for computing pixel-wise relevance maps for ViT models. The relevance map for a pre-trained ViT does not necessarily coincide with the area occupied by the foreground object. Accordingly, a pre-trained model is fine-tuned with cross-entropy loss and two regularisation terms aimed at improving the relevance map. These two terms respectively encourage positive and negative agreement of the relevance map with a foreground segmentation map. This method is evaluated on several ImageNet-adjacent robustness benchmarks, and results in either noticeable improvements (>1pts: ObjNet, INet-A, INet-R) or performance on par with the baseline (+/-1pt: INet, INet-v2, Sketch).


strengths_and_weaknesses: This is a nice simple idea and it appears to improve robustness without much effort. If I understand correctly, it relies on fine-tuning a model for 50 epochs with just 1500 images per epoch (500 classes, 3 images each -- half of the total classes). It thus requires very few annotated segmentation masks relative to the full dataset. However, the method also results in decent performance with automatic segmentations obtained by a recent method called TokenCut (CVPR '22). 

The experiments make sense, particularly the choice of evaluation benchmarks as well as the selection of competing methods. I like that several architectural variants & differently pre-trained instances thereof were considered, as well as repeated runs with different random seeds (see Appendix). I do have some concerns/questions with the experiments however which I list in the next section of the review. I think an opportunity was missed to conduct a more detailed ablation, and I have concerns about the hyperparameter selection procedure for the competing methods. I am surprised that increasing the amount of fine-tuning data (beyond 3*500 images) can harm performance (see Fig. 8).

In terms of clarity, I think the description of the relevance map extraction procedure (mostly relegated to the appendix) could be improved. No need to reproduce the full description from the original paper which describes it, but some clarification is necessary, e.g. I am not sure how this is initialised to an identity map? Are there any relevant knobs and dials to tweak the results?

Overall, I am leaning towards tentative acceptance because I like the method and the direction. The paper makes an important point (made elsewhere too), namely that focusing on a "single measuring stick: [accuracy]" can only get us so far. Instead -- and this is my read -- we have to look into encouraging better model behaviour through other means, e.g. in this case making sure the model attends to relevant parts of the object. Before reaching a final rating, I would appreciate responses to the questions I raise in the rebuttal regarding the experiments.

One unstated assumption that could perhaps be interrogated a bit more is that reliance on background is by default something to be discouraged. While this probably is the case for most classes, there are other classes with more ambiguous appearance where the background is presumably useful. A more fine-grained analysis of the impact on various classes would have been quite useful. This issue is alluded to in section 5 but without any corresponding qualitative or quantitative analysis as far as I can tell.

There is partially such an analysis in the appendix (section H), which compares performance on the 500 training classes. vs the 500 remaining classes. These are interesting results which are briefly discussed in the main paper (section 5). There is an accuracy difference between the two sets, but it is not consistent across models and datasets. Accuracy for the training classes is on average better, but in some cases it is the other way around. Are there any ideas as to why this is the case? 

questions: I have a few questions about the experimental evaluation:

I would have liked to see a baseline which fine-tunes the model only using the classification loss. The reported results labelled "original" appear to refer to performance for the pre-trained model without fine-tuning. Is this correct? As such, we don't quite have a proper comparison between methods.

I am also very confused by the choice of hyperparameters for the competing methods (RRR, GradCAM) -- see Appendix. I don't understand why the weight for the classification loss was not kept static for all methods together with the learning rate. What happens if you conduct a grid search for the regularisation term weights by fixing the aforementioned hyperparameters. Could you also describe the "difficulties" you faced while choosing the appropriate hyperparam values?

While it is interesting that the method is effective given very little data, it should however be mentioned that (based on Fig. 8 in the appendix) the number of classes (500) and number of samples per class (3) were selected based on accuracy. Increasing the number of classes and/or samples beyond the chosen values can have a slightly negative impact, especially visible for the ImageNet-A curve. 

This is a little surprising to me and I wonder why that is. Was the number of fine-tuning steps adapted to a change in the number of samples/classes? Is then the improvement a function of then number of training steps, or does the diversity of fine-tuning data not matter all? This is not immediately clear based on Fig. 8. It thus seems like a relatively minor adaptation of the network weights is enough to get better relevance maps and with it improved robustness. On which layers does the fine-tuning have the largest impact?

I also think that an opportunity was missed when it comes to examining the impact of different loss terms. I could not find an ablation study that considers different combinations of the regularisation terms that were considered (e.g. fg only, bg only).

What is the difference between AugReg and the corresponding vanilla ViT in terms of training?


limitations: Negative societal impact of the work is not addressed ([N/A] in the check list).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper focuses on improving the robustness of Vision Transformers by monitoring the relevancy map of models. Acted as a fine-tuning step, the proposed method contains three losses to suppress relevance on background regions, force the model to predict using foreground information, and learn from its own predictions. Experiments on several datasets show the effectiveness of the method.
strengths_and_weaknesses: Strengths:
The motivations in the paper were well established. It is a relatively simple idea, but the main claims are validated by experimental results and visualizations.

Weakness:
1. The paper should consider some self-supervised learning (SSL) methods using ViTs as a baseline, as SSL methods are believed to work well on out-of-distribution than supervised methods. Besides, DINO [1] also shows the property to identify the foregrounds, which may, to a certain extent, remedies the shortcoming of relying on image backgrounds to classify.  Providing the performance of SSL ViTs will make the paper more convincing. 
2. Adding the $L_{classification}$ on the fine-tuning phase, the method is limited to pre-trained classes. What if the out-of-distribution datasets have some semantic-different classes (iNaturalist or other natural datasets with different classes), would the K-nearest neighbor testing of these novel classes still show better performance? 
3. Also, the organization of the paper is not ideal. For instance, in Section 2, the description of validation datasets and related works are mixed together.
questions: 1. Provide a baseline experiment using DINO pretraining checkpoints.
2. What if the out-of-distribution datasets have some semantic-different classes, would the K-nearest neighbor testing on these novel classes still show better performance than the baseline? As mentioned in the paper, only parts of the classes used during fine-tuning improve the performance of the other classes. Therefore, the method should also work well in unseen classes?

limitations: Yes, the authors have addressed the limitation. The potential negative societal impact is not mentioned, which is okay as the paper focuses on improving the robustness of general models.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a method for robustifying Transformers-based image classifiers against different image distributions, assuming that better attention improves the generalization performance. The main idea is to force the relevance map (or an aggregation of attention maps in Transformers) to focus on foreground objects. The method generates a relevance map with [8] and gives a manual or unsupervised segementation map as a preferred relevance map. The advantage of the method is experimentally demonstrated.
strengths_and_weaknesses: ## Strengths

1. The method is simple but actually improves the performance. 
2. A good set of comparisons and ablation are presented. 

## Weaknesses

1. According to Eqs. (6) and (7), a relevance map is based on gradients. I guess the classifier (or the model) is trained through relevance maps, and if this is the case, more details on how the model is trained with these gradients. 
2. There are some methods that try to debias a model, like [Burns et al., "Women also Snowboard: Overcoming Bias in Captioning Models," ECCV 2018], which also uses manual segmentation for debiasing the model. I think the proposed method shares the basic ideas with such methods, and they may be easily adapted to classification tasks. I would like to see how the proposed method differs from these methods. Experimentally comparing some of these methods (for the manual segmentation variant) or at least providing discussion on them could be beneficial if they are comparable. 
3. The paper could discuss the data distribution used for training TokenCut in the context of generalization of the method to severely different data distributions. 
questions: 1. I would like to see some discussions and explanations on Weaknesses 1-3.
1. For the unsupervised segmentation case, how many images are used for training with segmentation results? 
2. I'm also curious about the performance of TokenCut over the datasets used in the paper. 
limitations: I'm not very sure if this is a limitation, but the paper does not mention the distribution used for training the unsupervised segmentation model. At least segmentation may not work when the data distribution is completely different (e.g., X-ray images). I think this point is not discussed in the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: When models are trained without extra guidance, they often extract spurious features that are causally irrelevant to the target task. The paper proposes to regularise "where the model looks at" with human-annotated segmentation masks (or with automatically-generated foreground masks). More precisely, the proposed method regularises Vision Transformers (ViT) by fine-tuning a normally pre-trained model with the objective of encouraging foreground regions to have greater "relevance map" [8] values. By aligning the model's attention with the true foreground, the proposed method improves the models' general robustness, measured in terms of ImageNet-A/R/V2/Sketch/ObjNet and SI-Score.
strengths_and_weaknesses: ## Strengths

The greatest strength of the paper is that it seems to solve an important problem in a relatively straightforward and intuitive fashion. Some might say it is unrealistic and costly to use human-annotated foreground masks. I strongly disagree with that viewpoint. The problem of spurious correlation and attention misalignment is not solvable without extra human guidance; the dataset itself does not contain sufficient guidance as to which feature a model must utilise to solve the problem. If it did, you won't see the consistent issue with spurious correlations in various models and datasets. Nonetheless, many researchers shun away from using extra annotations (especially if they are expensive like segmentations) and data points, for the fear that they are seen as impractical. I do not think so. I believe using such extra annotations can in fact be much cheaper in practice than providing the needed human guidance through complex loss and regularisation terms and hyperparameters (which require expensive model re-training to run HP search for every dataset and architecture).

The method does seem to improve the robustness across the board (Tab 1 and 2). There do exist some inconsistencies here and there, but they are not major and are expected for such large-scale evaluations. The internal mechanisms are also verified quite well through ablative studies (Tab 3 and 4). The extra results in Appendix are also quite impressive. Hyperparameter selection is described well in Section D. The proposed method has used the same set of hyperparameters across experiments, while the baselines use hyperparameters tuned for each setup. The sensitivity tests in Section G provide a nice insight that the method already improves the robustness quite a bit with only 100 additional segmentation masks (100 classes x 1 sample/class). 

## Weaknesses

The main weakness is the choice of the explanation method - relevance map. This reduces the intuitiveness of the method quite a bit in my opinion. Based on my reading of Section B (yes, please put this important information in the main text!), the explanation is generated by 
- computing gradient of ViT output w.r.t. attention map
- pointwise multiplying the gradient map above with attention values
- taking pointwise ReLU and head-wise averaging.
This is complicated. It is not intuitive as to "what effect will regularising such a complicated output of a ViT have on its inner mechanisms". For example, it would have been much easier to grasp the effect of directly regularising the attention map with the ground-truth foreground masks. You then understand that the ViT will be performing attention-weighted pooling across tokens where the weights are better aligned with the actual foreground features. 

It would be great if the authors could explain in words or mathematical language how the regularisation of the relevance map would affect the model in question. 

About hyperparameter tuning - could the authors explain for which objective metric the parameter tuning is executed? I have probably missed it. I'm worried if the tuning is performed with respect to the robustness measures. If so, the robustness information has leaked to the HP tuning and the good performances here will less likely generalise to other data and models.

Other minor comments:
- What is the authors' opinion on applying the technique on CNNs? It is in principle possible to generate the relevance maps for them. Maybe CAM-like heatmaps are more suitable there though.
- What stops us from performing the technique on training from scratch? Is it purely a computational limitation (single V100) ? It would be nicer to assure an improvement in robustness for the scratch-training scenario. That will also render the "classification" loss (Eq 4) unnecessary and reduce complexity.

## Conclusion

I weigh the strengths far more than the weaknesses. This is a nice paper that addresses an important problem with an intuitive approach. I hope the authors answer the remaining questions (and make minor revisions) to juice out the last bits of possible improvements.
questions: See weaknesses above.
limitations: They are okay.
review_title: null

Meta-review: metareview: Initially, this paper received positive reviews. The rebuttal addresses the remaining concerns. All reviewers feel that the contributions of this work are sufficient to merit its acceptance. The area chair agrees with the reviewers and recommends it be acecpted at this conference. 
confidence: Certain
Decision: Accept