Optimizing Relevance Maps of Vision Transformers Improves Robustness
Authors: Hila Chefer
Idan Schwartz
Lior Wolf
Venue: NeurIPS 2022 Accept (2022)
Keywords: Vision Transformers
Explainability
Robustness

Abstract: It has been observed that visual classification models often rely mostly on spurious cues such as the image background, which hurts their robustness to distribution changes.  
To alleviate this shortcoming, we propose to monitor the model's relevancy signal and direct the model to base its prediction on the foreground object.
This is done as a finetuning step, involving relatively few samples consisting of pairs of images and their associated foreground masks. Specifically, we encourage the model's relevancy map (i) to assign lower relevance to background regions, (ii) to consider as much information as possible from the foreground, and (iii) we encourage the decisions to have high confidence. When applied to Vision Transformer (ViT) models, a marked improvement in robustness to domain-shifts is observed. Moreover, the foreground masks can be obtained automatically, from a self-supervised variant of the ViT model itself; therefore no additional supervision is required. Our code is available at: https://github.com/hila-chefer/RobustViT.

Optimizing Relevance Maps of Vision Transformers Improves Robustness
Hila Chefer School of Computer Science
Idan Schwartz Lior Wolf
Tel-Aviv University
Abstract
It has been observed that visual classification models often rely mostly on spu- rious cues such as the image background, which hurts their robustness to dis- tribution changes. To alleviate this shortcoming, we propose to monitor the model's relevancy signal and direct the model to base its prediction on the fore- ground object. This is done as finetuning step, involving relatively few samples consisting of pairs of images and their associated foreground masks. Specif- ically, we encourage the model's relevancy map (i) to assign lower relevance to background regions, (ii) to consider as much information as possible from the foreground, and (iii) we encourage the decisions have high confidence. When applied to Vision Transformer (ViT) models, a marked improvement robustness to domain-shifts is observed. Moreover, the foreground masks be obtained automatically, from a self-supervised variant of the ViT model it- self; therefore additional supervision is required. Our code is available at: https::/ htips:///ittubb hathu..cooohht.
Introduction
The reliance on simple image-level classification supervision, together with the sampling biases of object recognition datasets, leads vision models that exhibit unintuitive behavior, as depicted in Fig First, the models we tested (ViT [14), ViT AugReg [50], and DeiT (53] tend to give disproportional high weight to the background of the image in the decision-makinn process. Second, the tested models occasionally regard a sparse subset of the pixels in the foreground object for the classification, disregarding much of the object's data. As argued by Geirhos et [19], and stated in [26) "image classification datasets contain 'spurious cues' or 'shortcuts" For instance, cows tend co-occur with green pastures, and even though the background is inessential the identity of the object, models may predict 'cow", using primarily the green pasture background cue."
There is considerable evidence that context is useful cue 31 40]. However, many of the associated background elements and foreground shortcuts are only relevant to the specific data distribution. which leads to lack of robustness to distribution shifts (37 43). There many methods for overcoming domain shifts, including domain adaptation techniques 16 1 and methods that augment the training the training procedure (27] (38. In this work, however,
opt for direct approach, which monitors the relevancy score of the model for each image region, and manipulates the relevancy map be focused on the regions within the foreground mask.
The method is based on a finetuning procedure, which is applied to a pretrained Vision Transformer (ViT) model. A relatively small set of samples, for which the foreground is given, is employed during this phase. In most of our experiments, we use three samples for half the classes, following work that examined the effect of transfer learning half of the classes [60]. The ground-truth foreground mask is either human annotated [7 or estimated from a self-supervised ViT model [58]
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

ViT-base 14
AR-base 50
DeiT-base 53
Classification: Chestnut Confidence: 100%
Classification: Bullfrog Confidence: 99.1%
Classification: Dishcloth Confidence: 91.6%
Classification: Schooner Confidence: 97%
Classification: Zebra Confidence: 99.9%
Classification: Lipstick Confidence: 96.1%
Figure Examples of the salient issues with ViTs. Each pair depicts an input image and its corresponding relevance map. The first row demonstrates examples of background centered relevance, the second row shows examples of sparse foreground relevance. Both issues occur in all models- VIT [14, ViT AugReg [50] (AR), and DeiT' [53], even if the confidence of the model above 90%.
The finetuning procedure employs three loss terms. The first encourages the relevance map to assign lower values the background of the image. The second term aims to remedy the sparse relevance issue by making larger parts of the foreground part of the relevancy map. The third term is a regularizer that ensures that the classification accuracy of the original model remains unimpaired.
Unsurprisingly, applying this method leads to (modest) drop in accuracy on the original training dataset and datasets with very similar distributions. In extensive battery of experiments show that (i) the classification accuracy on datasets from shifted domains increases considerably. This includes real-world unbiased and adversarial datasets, well as synthetic ones that were created specifically to measure the robustness of the classification model, (ii) the resulting relevance maps demonstrate a significant improvement focusing on the foreground the image, i.e. the object, rather than on its background.
Related Work
Image classification datasets are becoming increasingly challenging, while same time models growing more complex [12 30
51 23 With the rapid advancements in object recognition, the
measuring stick being narrowed down to a single number, accuracy [52]. By relying solely on accuracy, classifiers introduce biases, since they utilize shortcuts select the right class
31 [9 The
shortcomings of models that rely on shortcuts have been demonstrated for domains other than vision, such natural language processing [15] and multi-modal learning [18]. Therefore, it is becoming increasingly important to evaluate classifiers based on additional measures, such as robustness to distribution shifts, to ensure that the accuracy of the model does not conceal problematic decision- making process.
Detecting shorteuts One way assess the salient behavior of model is to study Sufficient Input Subsets (SIS), i.e., the minimal number pixels necessary for a confident prediction 7 [) Finding an SIS for a class can imply that the classifier has overinterpreted its input, since it can make a confident accurate decision using a small, sparse subset of pixels, which does not seem meaningful to humans. We study the SIS with gradients approach for ViT models, and find that can be mislcading. Specifically, SIS can be regarded as an adversarial method that can lead to high-confidence classification of any label from sparse set of pixels, see Appendix A.
Explainability methods may used to determine the reasons for the decisions made by classifiers. As an example, we may find that models tend to overlook objects with relevance maps (see Fig. [ Gradients are dominant and useful signal for model interpretation |5.
34 By adding input Alternatives to gradient
signals to the gradients, relevance maps were refined propagation for explanation include attribution propagation theory-driven method based on
2
44
48
49
axioms, and permutation based on Shapley values [2] 35
36
33
For transformer architectures,

---

the combination of gradients and attention values has been shown to produce a viable interpretation of the model's prediction 88 9
Reducing reliance on shortcuts Our method optimizes the relevancy maps of the model as regularization term. Other works that investigated the of relevancy alleviate overfitting include Ross et al. [41, who introduced a regularization term for the input gradient, which reduces reliance on irrelevant cues, e.g., background pixels. Additional work in this vein has been conducted on medical data, studying how doctors classified a disease 14, 55]. Singh et al. 47 regularize feature representations of a category from its co-occurring context. Zhu et al. [6] enrich classifier representations by mimicking detection models. Importantly, unlike all the above methods, our method incorporates the foreground features and the classifier confidence, rather than considering only the background features. Furthermore, we apply our method not during training, but as short finetuning process that feasible for large models.
Reducing bias training with adversarial strategies [28 [57.. Other approaches suggest generating counterfactual samples to mitigate reliance on biased features, c.g., masking gender features and preventing a Additionally, Kolesnikov et captioning model from generating gender-related words 24 62 al. (29] examined background biases in object detectors.
related yet slightly different task that of debiasing models. One approach
Method
Our approach aims to direct vision models such that their decision will be based the features of the object rather than on other supportive background features. To achieve this, we employ additional supervision to distinguish between the foreground and background features. The method finetunes the model in a way that encourages the class relevance map, obtained through a relevance computation method, to roughly resemble the segmentation map. This way, the decision-making process is focused on the foreground. The relevance map employed by our method is calculated using recent advancement in explainability for transformer-basse based architectures 88] A brief introduction of the explainability method used is provided in Appendix B.
The method employs a small set of labeled segmentation maps for distinguishing between the foreground and background of the input image. Our first loss term discourages the model from considering mostly the background:
bbg MSE (R(1) o S))).),,
where i is the input image, R(i) is the relevance map produced for i, S is the inverse of the segmentation map for and @ is the Hadamard product. Put differently, Log extracts the relevance values assigned to the background using the provided segmentation, and encourages those values to be close to 0, which is the minimal possible relevance value.
Our second loss term encourages the model to consider as much information as possible from the foreground of the image:
Lg MSE (i) o S(i)
where S(i) is the foreground mask. This loss encourages the relevance of pixels inside the segmen- tation to be higher (1 the maximal achievable relevance value). The overall explainability loss constructed follows:
Lrelevance
>bg Lig + Arg Lig-
where Ang, Arg are hyperparameters, All our experiments apply the same choice of Ang 2, Ag = 0.3. Note that the coefficient Xrg is much smaller than Abg. The reason is two-fold: (i) we find that the issue of overinterpreting the background is more common than the issue of partial relevance of foreground pixels, and (ii) Lig implics a uniform relevance of for all foreground pixels, which may be detrimental, as we wish to allow the model to be able to focus specific features of the object.
Finally, in the absence of an additional regularization loss, the finetuning results in explanations that resemble the ground-truth segmentation, while the accuracy plummets due to the absence of encouragement to maintain high accuracy. Therefore, one must apply an additional loss term to ensure that the output distribution of the model remains similar to the original model. We opt to use maffidencebbossiee loss for this purpose, which is constructed as follows:
elaassffication = CE (Mi)), arg ma)(M())))
(4)
3

---

Same prediction
Corrected prediction
Incorrect prediction
Input
Original
Ours
Input
Original
Ours
Input
Original
Ours
VIT-BB
ViT-L Pred
Snowplow
Snowplow
Can- opener
Teddy- bear
Tripod
Strawberry
AR-B Pred
Jacamar
Jucamar
Indri
Lemur- catta
Coral- reef
Leather- back turtle
AR-L Pred
Sundial
Sundial
Hummiin- bird
Chickadee
Screwdr-
Screw
Pred DeiT-B Pred
Capuchin
Capuchin
Quilt
Chest
Eft
Bottlecap
Curly coat- Curly coat- ed retriever ed retriever
Bubble
Mountain-
Microwave Crock-
tent
Figure 2: Examples from the ImageNet validation set of cases where our method does not change the prediction. corrects the prediction, and ruins the prediction. Even in cases where method changes a correct prediction, there often a rationale behind the modified prediction. The "Pred" specifies the predictions before and after finetuning. The examples presented for the base, large models of VIT [4], ViT AugReg
(50) (AR), and the base model of DeiT' [53]
where M notates the vision model, and arg max(M(i)) is the class predicted by M for the input image i. Lelussiticationn calculates the cross-entropy loss between the output distribution of M and one-hot distribution where the predicted class is assigned probability of 1. In other words, this loss encourages the confidence the predicted class to increase.
The overall loss for the finetuning process is, therefore:
L >relevance Lrelevance + Adlassificntion Celassification
where Arelevance = 0.8, and Aelassification = 0.2 remain constant all our experiments.
Experiments
The main hypothesis of this work is that improving the salient maps of ViTs trained on ImageNet will result reduced overlitting, and better generalization to data from unseen distributions. We present a wide range of tests to confirm hypothesis. Importantly, all the models we experiment on have been finetuned on ImageNet-1k 1k, which also the dataset we employ in our finetuning process.
First, we evaluate the improvement in robustness, i.e., the ability to maintain high accuracy under distribution shifts. The datasets with shifted distributions are only used for evaluation and contain both real-world datasets and synthetic ones. Second, we conduct segmentation tests following 9 to assess the effect of our method on the level of agreement between the relevancy maps and the foreground segmentation maps. Third, following [60], our method employs samples from subset of

---

Net-AA
ObjectNet
Sl-rotation
Input
Original
Ours
Input
Original
Ours
Input
Original
Ours
ViT-B
Figure 3: Examples of cases where method corrects wrong predictions, alongside the original and modified (after finetuning) explainability maps. The "Pred" row specifies the predictions before and after finetuning. The original classifiers focus sparse irrelevant data (e.g. the presence of snow leads DeïT-B to predict that a bus is Snowplow, porcupine is classified by ViT-L as sea lion due to the presence of the ocean, tank is classified by AR-B tram due the presence of VIT-L Pred
Television
Mantis
Drill
Hand- blower
Golf
Lemon
Prod
Garbage- truck
Forklift
Ping-pong ball
Teddy
Sea Lion
Porcupine
AR-B
Pred
Tram
Tank
Maze
Orange
Swimmi- ng trunks
Sock
AR-L
Pred DeiT-B Pred
Kite
Vulture
Scale
Strainer
Lifeboat
Cello
CRECE
ERS
Snowplow School bus
Bathtub
Rumning-
Balloon
Ruler
shoe
tram cables in the image, etc.). The examples are presented for the base, large models of ViT [1] ViT AugReg 50 (AR), and DeiT [33
the classes during training, that differ from the set of classes not used during training (set B). Ideally, the method would have positive effect on samples from both sets and B. Furthermore, in Appendix sensitive the method is to the number of samples per class, and to the number of classes in set A.
check whether the training classes (set A of the labels)
evaluate how
Additional experiments Appendix L presents comparison with self-supervised models and demonstrates that our method is beneficial even in a self-supervised setting. To validate the effec- tiveness of our method in improving the representations by ViTs, Appendix M contains h experiments on datasets that contain non-ImageNet classes before and after applying our method.
Robustness benchmarks Several alternatives have been proposed to ImageNet 42 to examine robustness to distribution shifts. We conduct our experiments on ImageNet-v2 [39], ImageNet-A (26). ImageNet-R (25,, MnageNet-SSktchh (56), ObjectNet [4, and SI-Score [3] A description of the datasets can be found in Appendix C.
Baseline Methods We focus on methods that resemble ours, i.e. methods that strive to correct overfitting by manipulating the saliency maps of the model. Our baselines include GradMask [46), and Right for the Right Reasons 41 (RRR). Both GradMask and RRR were originally applied during training: this work, however, we focus on large vision models that require significant resources for training, which makes this approach computationally impossible for us. Therefore, we apply their accuracy and relevancy losses within a finetuning process, similar to ours, in order to test their success in a fair way, while adhering to computational limitations.
5

---

Both GradMask and RRR employ two loss functions. First, classic cross-entropy, loss with the ground-truth. labels to ensure correct labeling, and second, loss limiting the values of the gradients of irrelevant parts of the input. The latter resembles our background loss (Eq.[], with gradients as the relevance map. We refer the reader to Appendix D for the full deseription of the applied losses.
We note that while using the gradient of the output w.r.t. the input is common practice for interpreting CNNs, these gradients are less stable for transformer-basee models. For example, results presented in 32 demonstrate that for transformer-based models the classic Input Graddient method violates faithfulness. We found it difficult to grid-search hyperparameters to fit both accuracy and relevancy losses simultaneously for the baselines. Furthermore, we had to tune the hyperparameters for each model separately to obtain an improved relevance loss. Our method, on the other hand, uses the same hyperparameter choice (see Sec. 3) for all models, which makes it far more stable to use, thus allowing us to run experiments on large models as well (i.e. ViT-I., ViT AugReg-L.). We refer the reader to Appendix E for the full description of hyperparameters used in ours experiments.
Models and Training To demonstrate the effectiveness of our method, we experiment on three types of ViT-based models: vanilla ViT [14) ViT AugReg (AR) (50), and DeiT (54] which presents techniques for efficiently training ViTs. For each model type, we experiment with different model sizes, in order to learn whether our method improves robustness across training techniques and model sizes. All models use 224 resolution images with a patch size of 16 x 16. We use the implementation and pre-trained weights from [59]. The small, base models are finetuned on a single RTX 2080 Ti GPU. and the large models on single Tesla V100 GPU. All models are finetuned as described in Sec. 3 for 50 epochs, with a batch size of 8. We use 3 training images from 500 ImageNet classes for our finetuning (overall- 1500 samples), and another 414 images as a validation set. In most experiments, we simply select the first 500 classes from ImageNet-S to be used for training (set A). For results with multiple random seeds and multiple choices of the 500 classes to use for training, see Appendix F. The learning rate of each model is determined using a grid search between the values 7 and 5e- - 6. As a rule of thumb, we select the highest learning rate for which the validation accuracy does not decrease by more than 2 - 3% and Letassificatin does not increase. We find that our method is not sensitive to small shifts in the learning rate, thus this rule applies well to all models.
As mentioned in Sec. 3 we use segmentation maps to distinguish between the foreground and the background of an input image. Our experiments employ two options of obtaining segmentation maps for ImageNet training images. In the first option, we use human-annotated segmentation maps from the ImageNet-S dataset [17]. The ImageNet-S dataset contains 10 training samples with their segmentation maps for 919 of the ImageNet classes. We employ considerably smaller subset, as detailed above. The second option does not employ extra supervision in the form of manually labeled images; instead, it employs the Tokencut object localization method presented in (58) to produce foreground segmentation maps.
Results Tab. I presents the results of our method and the baseline methods applied to different ViT-based models. As can be seen, for real-world datasets, both adversarial datasets (INet-A) and datasets with random and controlled background, rotations, and view points (ObjectNet), our method significantly and consistently improves performance (average of 5 5.8%, 5.0% top-1 improvement on INet-A, ObjectNet, respectively). For datasets that contain art, sculptures. sketches etc. (INet-R, INet- Sketch) the increase in accuracy is less steep (2. 7%, 0.9% averaged top-1 improvement, respectively). This can be intuitively attributed to the fact that art and sketches often feature the object without a background, or with a uniform background. Additionally, as can be seen, although the baseline methods preserve accuracy on the datasets from the original ImageNet distribution (INet val set and INet-v2), they fall behind our method on real-world nuttof-diitribution datasets (INet-A and ObjectNet), indicating that the baselines are less successful in alleviating overfitting.
Following our method, there slight decrease in the performance of the models on data from the ImageNet distribution (INet val set and INet-v2), which can be attributed to the fact that we reduce some of the overfitting on the ImageNet data. Fig. Fig.eerrssents examples of all three possible prediction cases on the ImageNet validation set: cases where our method preserves the original classification, corrects the original prediction, and changes a correct prediction. As can be seen, while performance sometimes decreases with respect to the assigned label, in most cases the rationale of the class proposed by the modified model is clear, often indicating a more natural labeling option. In addition, in the cases where the prediction remains the same is corrected, our method produces improved relevance maps.

---

Table 1: Robustness evaluation for VIT 14 VIT AugReg [50] (AR), and DeiT [4 with our method, and the baseline methods GradMask 46 and Right for the Right Reason (RRR) [А]] "Annotated segmentation" indicates whether we used annotated segmentation [7] or unsupervised localization [58]. "Original" stands for the model without finetuning. The bottom rows indicate the average change caused by our method across all architectures (on some models the baselines could not be run successfully; therefore, we do not compute their average change).
Annotated
NNt val R@1 R@5
INet-A
INet-R
Sketch
INet-v2
ObjNet
Model
Method
segmentation
R
R@5
R@1
R
R@1 1 R@5
R@ 1 R@5
R@ R@5
Original
81.5 96.0
16.0
37.0
33.8 48.5
35,4 57.4
71.1
89.9
35.1 56.4
GradMask
81.8 96.1
17.5 39.8
34.5 49.4
35.8
57.8
71.4
90.5
36.7 58.2
VIT-B
RRR
81.9 96.2
18.9 41.9
34.8
49.7
35.8
57.8
71.4
90.5
38.1 60.0
Ours
80.3 95.4
24.1
48.0
36.3
51.4
36.2
58.5
70.0
89.4
42.2 65.1
Ours
80.4 95.4
23.0
45.7
35.4
50.0
35.8
69.8
89.4
40.8
64.0
Original
82.9 96.4
41.5
36.6
52.0
40.4
63.4
71.8
90.7
37.4 59.5
VIT-L
Ours
82.0 96.2
25.2
49.6
38.8
54.6
41.2
64.3
71.3
90.6
42.5 65.4
Ours
82.7 96.4
25.2
50.0
39.8
55.1
41.8
64.8
72.1
91.2
43.2
65.8
Original
81.4 96.1 13.0
33.9
31.2
47.1
32.8
54.2
69.9
90.1
34.3
55.8
GradMask
81.3 96.1
16.4
39.2
32.3
48.3
32.5
53.7
70.1 90.3
37.6 60.2
AR-S
RRR
81.5 96.1
13.7
35.1
31.6
47.4
32.9
54.2
70.3 90.1
35.1 56.7
Ours
79.8 95.7
18.2
40.6
33,9
50.2
33.5
55.4
69.6
90.0
38.7 61.1
Ours
80.3 95.8
19.1
42.2
33.8
49.7
33.8
55.5
69.6
90.1
39.3
61.7
Original
84.4 97.2
23.9
49.2
41.0
57.8
43.1
65.7
73.8
92.3
41.4
63.7
GradMask
84.5 97.3
25.1 51.4
41.5
58.1
43.1
65.7
74.0 92.6
42.7
64.8
AR-B
RRR
84.6 97.3
26.8
53.0
41.9
43.2
65.7
74.3 92.6
43.7
Ours
83.1 96.9
31.3
57.1
44.7
61.5
44.6
67.4
73.5
92.0
47.1
70.0
Ours
83.6 97.1
31.2
57.2
44.5
60.9
44.7
67.4
73.7
92.4
46.5
69.1
Original
85.6 97.8
34.7
61.0
48.8
64.9
51.8
73.6
75.8
93.4
46.5
68.3
AR-L
Ours
85.1 97.5
42.1
67.5
54.0
69.1
54.2
75.8
75.8
93.4
51.6
73.2
Ours
X
85.4 97.6
42.4
68.0
53.8
69.0
54.1
75.8
76.1
93.6
52.0
73.5
Original
X
78.1 93.7
8.3
23.5
28.2
41.9 28.8
46.7
66.5
86.6
28.3
47.3
GradMask
77.0 93.6
7.9
24.7
26.6
40.5 26.0 43.5
64.5 85.6
28.2
48.6
DeiT-S
RRR
78.1 94.1
9.0
26.9
26.9
40.6
26.9 44.4
66.0
86.7
29.3
49.9
Ours
/
78.6 94.5
10.1
29.0
29.3
43.6
29.1
47.8
67.3
87.3
31.6
53.0
Ours
78.6 94.4
11.0
30.3
29.9
44.4
29.4
48.0
67.1 87.4
31.6
52.9
Original
80.8 94.2
12.9
31.0
30.9
44.2
31.2
48.6
69.7
31.4
GradMask
81.1 95.3
15.1
36.9
31.0
45.5
31.2
49.1 69.7 88.7 33.5 53.1
DeiT-B
RRR
81.0 95.2
14.8
37.0
30.7
45.1
30.9
48.8
69.5 88.6
33.6 53.3
Ours
80.5 94.9
17.2 40.0
32.4
47.0
30.9
49.2
69.1 88.3 35.9 56.2
Ours
80.5 95.0
18.3 40.9
32.8 47.5
31.5 49,9
69.3 88.5
36.3 56.6
Avg.
Ours
-0.8 0.0
+5.8
+7.8
+2.7
+3.0
+0.9
+1.3
-0.3
+0.2
+5.0 +6.4
change
Ours
-0.5 0.0
+6.1 +8.2 +2.8 +2.9
+1.1 +1.4
-0.1 +0.4 +5.0 +6.3
Additionally, as shown in Tab. for the SI-Score dataset, which synthetic dataset designed for testing resilience for shifts in object locations, sizes, and rotations, there is very steep improvement in performance across all models, while, once again, the baselines fall behind on all models and sizes.
Evidently, in both Tab. 12 our method works just well and often better when using the unsupervised segmentation maps. This means that our method may be applied without requiring any manual supervision, except for the image label.
Fig. presents example cases in which our method is able to correct the prediction of the original model on images from various robustness datasets. As can be seen, the original models tend to overinterpret the background, and therefore produce false classifications based on it. For example, lemon is classified as a golf ball due the grass in the background (third example in the first row), tank is classified as a tram due to the tram cables at the top the image (first example in the third row), and so on. Additional examples can be found in Appendix G.

---

Table 2:
Robustness evaluation on the synthetic SI-Score dataset [3]. which tests changes in
object position, rotation, and size using our method and the baseline methods GradMask [46].
Right for the Right Reasons (RRR) The models tested are ViT [14, ViT AugReg [50] (AR),
and DeiT 154. "Annotated segmentation" indicates if we used annotated segmentation 17 or
unsupervised localization
58
"Original" stands for the model without finetuning.
Annotated
SI-location
SI-rotation
SI-size
Model
Method
segmentation
R@!
R@5
R@!
R@5
R@!
R@5
Original
33.3
52.2
39.1
58.3
55.6
76.2
GradMask
/ 34.6 (+1.3)
53.9 (+1.7)
40.7 (+1.6)
60.3 (+2.0)
$7.0 (+1.4)
77.5 (+1.3)
ViT-B
RRR
/
35.6 (42.3)
55.0 (+2.8)
41.9 (+2.8)
61.8 (+3.5)
58.0 (+2.4)
78.4 (+2.2)
Ours
/
38.6 (+5.3)
57.8 (+5.6)
46.2 (+7.1)
67.0 (+8.7)
61.0 (+5.4)
81.4 (+5.2)
Ours
X
38.4 (+5.1)
57.0 (+4.8)
44.8 (+5.7)
65.2 (+6.9)
60.2 (+4.6) 80.6 (+4.4)
Original
31.6
50.3
40.7
60.1
54.8
75.6
VIT-L
Ours
/
36,3 (+4.7)
56.2 (+5.9)
45.3 (44.6)
66.2 (+6.1)
58.6 (+3.8)
80.3 (+4.7)
Ours
X
36.7 (+5.1)
56.3 (+6.0)
45.3 (+4.6)
66.6 (+6.5)
59.1 (+4.3)
80.5 (+4.9)
Original
32.4
51.7
40.6
59.6
55.4
75.7
GradMask
34.3 (+1.9)
53.9 (+2.2)
43.3 (+2.7)
63.0 (+3.4)
58.0 (+2.6)
78.3 (+2.6)
AR-S
RRR
32.9 (+0.5)
52.3 (+0.6)
41.4 (+0.8)
60.6 (+1.0)
$6.0 (+0.6)
76.3 (+0.6)
Ours
36,8 (+4.4)
56.6 (+4.9)
47.6 (+7.0)
67.8 (+8.2)
61.3 (+5.9)
81.2 (45.5)
Ours
X
36.3 (43.9)
55.6 (+3.9)
46.6 (+6.0)
66.7 (+7.1)
60.7 (+5.3)
80.4 (+4.7)
Original
40.5
60.8
48.1
68.3
60.6
80.4
GradMask
41.5 (+1.0)
61.8 (+1.0)
49.3 (+1.2)
69.5 (+1.2)
61.4 (+0.8)
81.3 (+0.9)
AR-B
RRR
42.4 (+1.9
62.7 (+19)
50.4 (42.3)
70.7 (+2.4)
(+1.5)
82.0 (+1.6)
Ours
43.2 (42.7)
62.8 (+2.0)
54.0 (45.9)
74,6 (+6.3)
64.1
(+3.5)
83.9 (43.5)
Ours
X
44.3 (43.8)
64.0 (+3.2)
54.6 (+6.5)
74.7 (+6.4)
64.5 (+3.9)
84.6 (+4.2)
Original
43.8
64.2
52.4
72.5
62.3
82.2
AR-L
Ours
48.3 (44.5)
(+4.3)
57.0 (44.6)
77.2 (+4.7)
66.4 (+4.1)
86.0 (43.8)
Ours
47.4 (43.6)
67.4 (43.2)
58.0 (+5.6)
78.1 (+5.6)
66.5 (+4.2)
85.6 (43.4)
Original
X
30.7
50.4
36.7
54.3
51.6
72.0
GradMask
V
32.0 (+1.3)
50.7 (+0.3)
38.9 (+2.2)
56.7 (+2.4)
54.1 (+2.5)
74.0 (+2.0)
DeiT-S
RRR
32.0 (+1.3)
51.0 (+0.6)
38.5
56.3 (+2.0)
53.9 (+2.3)
73.8 (+18)
Ours
32,3 (41.6)
51.5 (+1.1)
40.6 (43.9)
59.4 (45.1)
55.8 (+4.2)
76.3 (44.3)
Ours
32.5 (+1.8)
51.4 (+1.0)
41.0 (44.3)
(45.3)
56.0 (+4.4)
76.1 (44.1)
Original
X
34,5
54.6
39.3
56.3
54.6
73.4
GradMask
34,1 (-0.4)
54,9 (+0.3)
39.1 (-0.2)
58.3 (+2.0)
55.2 (+0.6)
75.8 (+2.4)
DeiT-B
RRR
34.4 (-0.1)
55.2 (+0.6)
40.4
(42.222
55.3 (+0.7)
75.8 (+2.4)
Ours
36.6 (42.1)
57.0 (42.4)
42,9
(43.6)
61.5 (45.2)
58.0 (43.4)
78.2 (44.8)
Ours
37.8 (43.3)
58.1 (43.5)
(44.9)
62.7 (46.4)
59.3 (+4.7) 79.0 (45.6)
Table 3: Evaluation of segmentation performance from relevance maps on the ImageNet-ssggmentttio
dataset [2] for ViT [4), ViT AugReg (50) (AR), and DeiT [54] before and after finetuning with our
method. Metrics and dataset are taken from [9]
ViT-B
VIT-L
AR-S Orig Ours
AR-B
AR-L
DeiT-S
DeiT-B Orig Ours
Model
Orig Ours
Orig Ours
Orig Ours
Orig Ours
Orig Ours
Pixel acc.
76.3 82.1 73.4 82.5 76.7 83.3 76.6 81.2 65.2 78.9
78.7 80.8 79.0 81.3
mloU
58.3 65.8
54.4 66.4
57.7 67.7
57.1 64.6
43.6 61.0
60.7 64.0
61.6 64.7
mAP
85.3 87.5
82.7 86.9
84.2 87.7 84.4 86.8 78.6 85.4 85.0 86.4 85.7 86.8
Segmentation tests Since our motivation is to encourage the relevance to focus less on the background and more on much of the foreground possible, we test the resemblance of the resulting relevance maps to the segmentation maps following [9] As can be seen in Tab, 3 our method significantly and consistently improves segmentation metrics all models, indicating that our finetuning indeed achieves goal. Appendix contains perturbation tests that demonstrate that the improved relevancy maps after applying method still faithfully reflect the model's reasoning.
Comparing training classes the other classes To ensure that the effects of our finetuning generalize to classes that were not included in the training set, we test the increase in robustness

---

tritinn clamee
Aug. change accuracy
Aug. charge
Netvll (M66 Bibk Stech Nebva octher Size Sinne
Siite
MAd val PoLA
Neb-s Shathh
INcloa oueee. silec
Sloce
Dalasel
Dalaset
(a)
(b)
Figure 4: Evaluation of the average change produced by method the training classes and the classes that were not in the training set. Changes are averaged across the base models of ViT 14) ViT AugReg [50), and DeiT 533
top-1 average change, (b) top-5 average change.
Table 4: Ablation study for our method on VIT [1] and DeiT 53 base models. The table presents top-1 accuracy. We check the impact of each term our loss on robustness, as well as the choice of confidence boosting versus cross-entropy with the ground-truth label (labeled as w/ ground-truth).
Model
Method
Nee val
INet-A
INet-R
Sketch
INet-v2
ObjNet
SI-loc.
SI-rot.
SI-size
Original
81.5
16.0
33.8
35.4
71.1
35.1
33.3
39.1
55.6
Ours
80.3
24.1
36.3
36.2
70.0
42.2
38.6
46.2
61.0
w/o Letoediation Co
77.8
17.9
34.2
34.8
67.4
37.6
37.2
43.0
58.4
VIT-
w/o Log (Eq.
81.5
19.1
35.9
3644
71.4
39.7
34.9
42.3
58.4
w/o Egg (Eq.
80.2
24.1
34.3
35.2
69.6
41.8
39.2
45.6
60.7
w/ ground-truth
81.7
21.5
35.5
35.8
71.2
40.3
37.8
44.5
60.1
only Lnz
66.0
18.0
23.2
25.8
55.1
32.1
33.3
35.0
50.5
only Lg
77.1
8.9
31.1
33.6
65.4
31.4
25.8
33.1
50.3
Original
80.8
12.9
30,9
31,2
69.7
31,4
34.5
39.3
54.6
Ours
80.5
17.2
32.4
30.9
69.1
35.9
36.6
42,9
58.0
Letsennction CEo
81.0
33.3
32.1
69.8
36.1
58.3
DeTl'B
log (Eq,
81.0
30.5
30.2
69.6
33.0
32.7
39.7
54.5
fg (Eq.
80.3
17.9
32.6
31.0
69.0
35.8
37.3
43.0
58.2
ground-Iruth
80.7
17.2
31.9
31,3
69.2
35.6
36.7
42,8
57.6
only Lax
71.7
13.1
25.3
24.7
59.7
29.2
37.4
53.4
only Li
79.8
27.5
28.2
67.7
28.3
27.6
35.3
49.9
resulting from method separately on the training classes and non-training classes. Fig. presents average improvement across the base models of VIT [14), VIT AugReg (AR), and DeiT (53]
As be seen, both subsets of classes demonstrate very similar increase in accuracy for the robustness datasets, with the classes that belong to the training set demonstrating better performance on the datasets from the original ImageNet distribution (INet val, INet-v2), can be expected due to the fact that they were represented in the training set. The full results of this experiment are presented in Appendix I.
We conduct ablation study to test the impact of each loss term on the result of the finetuning process. For brevity, Tab, T presents the top-1 accuracy results for each version of our method: the complementary table for top-5 accuracy can be found Appendix J. Our ablation study is conducted on ViT-B and DeiT-B, as they demonstrate different advantages for each loss term.
Ablation Study
First, we study the impact removing each loss term (labeled "w/o Lelassification " "w/o Log "w/o Cig" in Tab, 44. As can be seen, validation accuracy is lost mostly due to our background loss (Eq, D when we remove it, the accuracy remains intact, and yet, as we hypothesized, is more effective than the foreground loss (Eq.2) P in increasing the robustness, since when removing it, the accuracy on the out-of-distribtion datasets drops significantly. Additionally, the DeiT ablation demonstrates that in some cases, the classification loss does not contribute to the increase in robustness (other than for original ImageNet validation accuracy. However, for
INet-A), nor necessary for preserving VIT, the classification loss is crucial for avoiding significant loss of accuracy. Next, explore the variants where only one of the relevance loss terms Log. Lig applied without the classification loss Celassification (labeled "only Log". ", "only Li"" in Tab. We note that by removing the classification

---

loss, the model is at risk of mode collapse since Leg encourages the relevance on the background to be O. The mode collapse, in this case. would be to zero-out the relevancy map for the entire image. In an analog manner, Lig encourages high relevance on the foreground and can cause mode collapse where all the image receives uniform relevancy of 1. When applied together (the "w/o Lelassificaatin (Eq. 4)" ablation in Tab. H the two losses balance each other. However, when only employing one loss without the other and not adding a regularization term, the finetuning would encourage a mode collapse leading to lower accuracy.
Additionally, we perform an ablation to test the choice of confidence- booceenn as the classification loss (Eq. 4). by replacing it with the classic cross cross-entropy entropy loss with the ground label. Our ablation study demonstrates the benefit of using confidence boosting over the ground truth for the classification loss (labeled "w/ ground-truth" in Tab. 4). While the ground-trun variant preserves the original accuracy better (more so for VIT than for De using confidence boosting often improves robustness more significantly over using the ground-truth labels (see for example INet-A for ViT).
Finally, we refer the reader to Appendix for additional experiments that explore different options to replace GAE S as the explainability algorithm to produce the relevance maps.
Discussion and limitations
A surge of works have explored the benefits of using large-scale datasets of unlabeled samples from the internet, and many techniques have been proposed to train vision models in an unsupervised self-supervised manner [5] 20] As result, it is increasingly important develop methods for boosting model robustness without requiring any labels. We note that our method is compatible with such frameworks. As can be seen from Tab. I 2 our method performs well, even when applied with foreground masks obtained in an unsupervised manner, using Tokencut. Additionally, all of our losses can be applied without knowledge of the ground-truth label, since our classification loss Lelassification uses the predicted label, and the relevance maps could be propagated w.r.t. the predicted class.
Recently, it has been discovered that the effect of augmentation basedd regularization is extremely uneven across classes [3]. While such a regularization improves performance on average, some classes benefit significantly from it, while other classes suffer from large drop in accuracy. This variation is also apparent in the effect of our method. Above we have established that the method helps both classes which are included in the training set of the finetuning and classes which not. However, in both sets there is considerable variability in the effect of individual classes. This makes sense, since some classes present more well-localized objects and some classes require more reliance on the context of the object. See Appendix K for these results.
Conclusions
Can segmentation information help image categorization? Intuitively, the answer has to be positive. However, despite some effort to manually segment images of classification datasets, the obtained improvement, if any, is soon overtaken by better image-level methods.
Here, we propose a generic way to improve classification accuracy that can be applied to virtually any image classifier. Applied to transformers, we present evidence to show that while the accuracy on the original dataset does not improve, there is an increase accuracy for out-of-distribution test sets. The method optimizes the relevancy maps directly based intuitive desiderata. It opens new way for improving accuracy using explainability methods, which are currently seldom used for improving downstream tasks other than seeding weakly supervised segmentation methods.
Acknowledgments
This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant ERC CoG 725974). The first author further supported by the Council for Higher Education in Israel. The contribution of the first author is part of PhD thesis research conducted at Tel Aviv University.
10

---

References
[1] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. stat, 1050:27, 2020.
[2] Sehastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Rober Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015.
[3] Randall Balestriero, Leon Bottou, and Yann LeCun. The effects of regularization and data augmentation are class dependent. arXiv preprint arXiv: 2022.
[4] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supe-supe vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[6] Brandon Carter, Siddhartha Jain, Jonas W Mueller, and David Gifford. Overinterpretation reveals image classification model pathologies. In Advances in Neural Information Processing Systems, 2021.
[7] Brandon Carter, Jonas Mueller, Siddhartha Jain, and David Gifford. What made you do this? understanding black-box decisions with sufficient input subsets. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 567-576. PMLR, 2019.
[8] Hila Chefer, Shir Gur, and I Lior Wolf. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 397-406, October 2021.
[9] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Computer Vision and Pattern Recognition Centiiinn(CVV (CVPR), 2021.
[10] Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, and Yueting Zhuang. Counter- factual samples synthesizing for robust visual question answering. In CVPR, 2020.
[11] Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Advances in Neural Information Processing Systems, pages 6970-6979, 2017.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li. K. Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[13] Josip Djolonga, Jessica Yung. Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' Amour, Dan Moldovan, Sylvan Gelly, Neil Houlsby, Xiaohua Zhai, and Mario Lucic. On robustness and transferability of convolutional neural networks. 2021 IEEE/CVF Conference on Computer Vision and Pattern 2021.
Recognition (CVPR), pages
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 19299, 2020.
[15] Shi Feng, Eric Wallace, Alvin Grissom II, Mohit lyyer, Pedro Rodriguez, and Jordan Boyd-Graber Pathologies of neural models make interpretations difficult. arXiv preprint arXiv:1804.00 1804.0788...7 2018.
[16) Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Davaiinadversarial training of neural networks. The journal of machine learning research, 17(1):209-2200 2030, 2016.

---

[17]
Shanghua Gao, Zhong- Yu Li, Ming- Hing-Hsuan Yang. Ming-Ming Cheng, Junwei Han, and Philip Torr. Large-scale unsupervised semantic segmentation. arXiv preprint arXiv:2106.03149, 2021.
[18] Itai Gat, Idan Schwartz, and Alex Schwing. Perceptual score: What data modalities does your model perceive? Advances in Neural Information Processing Systems, 34, 2021.
[19] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shorteut learning in deep neural networks. ArXiv, abs/2004.07780, 2020.
[20] Priya Goyal, Quentin Duval, Isaac Seessel, Mathilde Caron, Ishan Misra, Levent Sagun, Armand Joulin, and Piotr Bojanowski. Vision models are more robust and fair when pretrained on uncurated images without supervision. ArXiv, abs/2202.08366 083600 2022.
[21] Jiuxiang Gu, Zhenhua Wang, Jason Kuen, Lianyang Ma, Amir Shahroudy, Bing Shuai, Ting Liu, Xingxing Wang, Gang Wang, Jianfei Cai, et al. Recent advances in convolutional neural networks. Pattern Recognition, 77:354 377, 2018.
[22] Matthieu Guillaumin, Daniel Küttel, and Vittorio Ferrari. Imagenet auto-annotation with segmentation propagation. International Journal of Computer Vision, 110:328-348, 2014.
[23] K. He, X. Zhang. S. Ren, and Sun. Deep Residual Learning for Image Recognition. ArXiv e-prints, December 2015.
[24] Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. Women also snowboard: Overcoming bias in captioning models. In ECCV, 2018.
[25] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-rf-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8340-8349, October 2021.
[26] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15262-15222 June 2021.
[27] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32, 2019.
[28] Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not to learn: Training deep neural networks with biased data. In CVPR, 2018.
[29] Alexander Kolesnikov and Christoph H. Lampert. Improving weakly-supervised object local- ization by micro-annotation. ArXiv, fass/6605.0555 2016.
[30] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. In Proceedings of the Advances in Neural Information Processing Systems, pages 1097-1105, 2012.
[31] Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 10(1): 2019.
[32] Y. Liu, Haoliang Li, Yangyang Guo, Chen Kong, Jing Li, and Shiqi Wang. Rethinking attention- model explainability through faithfulness violation test. ArXiv, abs/2201. 14, 2022.
[33] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pages 4765-4774, 2017.
[34] Aravindh Mahendran and Andrea Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision, 120(3):233-255, 2016.
12

---

[35] Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus- Robert Müller Explaining nonlinear classification decisions with deep taylor decomposition. Puttern Recognition, 65:211-222, 2017.
[36] Woo-Jeoung Nam, Shir Gur, Jaesik Choi, Lior Wolf, and Seong- Whan Lee. Relative attributing propagation: Interpreting the comparative contributions of individual units in deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2501-2508, 2020.
[37] A Nguyen, J Yosinski, and Clune. Deep neural networks are easily fooled: high confidence predictions for unrecognizable images. arXiv preprint arXiv: 14/2. 1897, 2014.
[38] Oren Nuriel, Sagie Benaim, and Lior Wolf. Permuted adain: reducing the bias towards global statistics in image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9482-9491, 2021.
[39] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Leurning, pages 5389-5400. PMLR, 2019.
[40] Amir Rosenfeld, Richard Zemel, and John K Tsotsos. The elephant in the room. arXiv preprint arXiv: 03305, 2018.
[41] Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi- Velez. Right for the right reasons: Training differentiable models by constraining their explanations. ArXiv, abs/1 703.03717, 2017.
[42] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3)211-2222,2 2015.
[43] Rakshith Shetty, Bernt Schiele, and Mario Fritz. Not using the car to see the sidewalk- quantifying and controlling the effects of context in classification and segmentation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages $218-8226, 2019.
[44] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning- Vaarning-Vollmee Volume 70, pages 3155-3153. JMLR. org, 2017.
[45] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In In Workshop at International Conference on Learning Representations, Citeseer, 2014.
[46] Becks Simpson, Francis Dutil, Yoshua Bengio, and Joseph Paul Cohen. Gradmask: Reduce overfitting by regularizing saliency. ArXiv, abs/1904.07478. 2019.
[47] Krishna Kumar Singh, Dhruv Mahajan, Kristen Grauman, Yong Jae Lee, Matt Feiszli, and Deepti Ghadiyaram. Don't ion't judge an object by its context: learning to overcome contextual bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11070-11078, 2020.
[48] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smooth- grad: removing noise by adding noise. arXiv preprint arXiv:1706.03825 2017.
[49] Suraj Srinivas and François Fleuret. Full-gradient representation for neural network visualization.
In Advances in Neural Information Processing Systems, pages 4126-4135, 2019.
[50] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. ArXiv, 0tt/22106.102770 2021.
[51] C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-9, June 2015.
13

---

[52] Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 2011, pages 1521-1528. IEEE, 2011.
[53] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12887,, 2020.
[54] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv'e e J' egoo.. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
[55] Joseph D Viviano, Becks Simpson, Francis Dutil, Yoshua Bengio, and Joseph Paul Cohen. Saliency is possible red herring when diagnosing poor generalization. arXiv preprint 1xXiv:1910.00199, 2019.
[56) Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global represen- tations by penalizing local predictive power. In Advances Neural Information Processing Systems, volume 32, 2019.
[57] Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In ICCV, 2019.
[58] Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L. Crowley, and Dominique Vaufrey- daz. Self-supervised transformers for unsupervised object discovery using normalized cut. In Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, June 2022.
[59] Ross Wightman. Pytorch image models.
httpss///githbb..ccmmrrgggga
pytorch- models. 2019.
[60] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
[61] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2019.
[62] Ran Zmigrod, Sabrina J. Mielke, Hanna Wallach, and Ryan Cotterell. Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology. In ACL, 2019.
14

---

Submission checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Will be included in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In Appendix E,, and in the experiments section of the paper.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Reported Appendix F in the supplementary material. (d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] In the Models and Training section
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A] All used data and code is licensed for public use, given the appropriate credit (MIT license). The ObjectNet dataset is the only exception to this rule, while it is free to use for research and commercial use, may never be used in model training (we only use for evaluation), and individual images from it should be presented with I pixel red border (see Fig. 3
(c) Did you include any assets either in the supplemental material
URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curatings [N/A] The data assets used are curated in way that it does not contain sensitive or personal information.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] The data assets used are curated in a way that does not contain sensitive, personal, or offensive information.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "t2r6oVx4fI", "reviewer": "Reviewer_FGu4", "summary": "When models are trained without extra guidance, they often extract spurious features that are causally irrelevant to the target task. The paper proposes to regularise \"where the model looks at\" with human-annotated segmentation masks (or with automatically-generated foreground masks). More precisely, the proposed method regularises Vision Transformers (ViT) by fine-tuning a normally pre-trained model with the objective of encouraging foreground regions to have greater \"relevance map\" [8] values. By aligning the model's attention with the true foreground, the proposed method improves the models' general robustness, measured in terms of ImageNet-A/R/V2/Sketch/ObjNet and SI-Score.", "questions": "See weaknesses above.", "limitations": "They are okay.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe greatest strength of the paper is that it seems to solve an important problem in a relatively straightforward and intuitive fashion. Some might say it is unrealistic and costly to use human-annotated foreground masks. I strongly disagree with that viewpoint. The problem of spurious correlation and attention misalignment is not solvable without extra human guidance; the dataset itself does not contain sufficient guidance as to which feature a model must utilise to solve the problem. If it did, you won't see the consistent issue with spurious correlations in various models and datasets. Nonetheless, many researchers shun away from using extra annotations (especially if they are expensive like segmentations) and data points, for the fear that they are seen as impractical. I do not think so. I believe using such extra annotations can in fact be much cheaper in practice than providing the needed human guidance through complex loss and regularisation terms and hyperparameters (which require expensive model re-training to run HP search for every dataset and architecture).\n\nThe method does seem to improve the robustness across the board (Tab 1 and 2). There do exist some inconsistencies here and there, but they are not major and are expected for such large-scale evaluations. The internal mechanisms are also verified quite well through ablative studies (Tab 3 and 4). The extra results in Appendix are also quite impressive. Hyperparameter selection is described well in Section D. The proposed method has used the same set of hyperparameters across experiments, while the baselines use hyperparameters tuned for each setup. The sensitivity tests in Section G provide a nice insight that the method already improves the robustness quite a bit with only 100 additional segmentation masks (100 classes x 1 sample/class). \n\n## Weaknesses\n\nThe main weakness is the choice of the explanation method - relevance map. This reduces the intuitiveness of the method quite a bit in my opinion. Based on my reading of Section B (yes, please put this important information in the main text!), the explanation is generated by \n- computing gradient of ViT output w.r.t. attention map\n- pointwise multiplying the gradient map above with attention values\n- taking pointwise ReLU and head-wise averaging.\nThis is complicated. It is not intuitive as to \"what effect will regularising such a complicated output of a ViT have on its inner mechanisms\". For example, it would have been much easier to grasp the effect of directly regularising the attention map with the ground-truth foreground masks. You then understand that the ViT will be performing attention-weighted pooling across tokens where the weights are better aligned with the actual foreground features. \n\nIt would be great if the authors could explain in words or mathematical language how the regularisation of the relevance map would affect the model in question. \n\nAbout hyperparameter tuning - could the authors explain for which objective metric the parameter tuning is executed? I have probably missed it. I'm worried if the tuning is performed with respect to the robustness measures. If so, the robustness information has leaked to the HP tuning and the good performances here will less likely generalise to other data and models.\n\nOther minor comments:\n- What is the authors' opinion on applying the technique on CNNs? It is in principle possible to generate the relevance maps for them. Maybe CAM-like heatmaps are more suitable there though.\n- What stops us from performing the technique on training from scratch? Is it purely a computational limitation (single V100) ? It would be nicer to assure an improvement in robustness for the scratch-training scenario. That will also render the \"classification\" loss (Eq 4) unnecessary and reduce complexity.\n\n## Conclusion\n\nI weigh the strengths far more than the weaknesses. This is a nice paper that addresses an important problem with an intuitive approach. I hope the authors answer the remaining questions (and make minor revisions) to juice out the last bits of possible improvements.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "l4I5IgaCUxp", "reviewer": "Reviewer_KJ1F", "summary": "The paper focuses on improving the robustness of Vision Transformers by monitoring the relevancy map of models. Acted as a fine-tuning step, the proposed method contains three losses to suppress relevance on background regions, force the model to predict using foreground information, and learn from its own predictions. Experiments on several datasets show the effectiveness of the method.", "questions": "1. Provide a baseline experiment using DINO pretraining checkpoints.\n2. What if the out-of-distribution datasets have some semantic-different classes, would the K-nearest neighbor testing on these novel classes still show better performance than the baseline? As mentioned in the paper, only parts of the classes used during fine-tuning improve the performance of the other classes. Therefore, the method should also work well in unseen classes?\n", "limitations": "Yes, the authors have addressed the limitation. The potential negative societal impact is not mentioned, which is okay as the paper focuses on improving the robustness of general models.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe motivations in the paper were well established. It is a relatively simple idea, but the main claims are validated by experimental results and visualizations.\n\nWeakness:\n1. The paper should consider some self-supervised learning (SSL) methods using ViTs as a baseline, as SSL methods are believed to work well on out-of-distribution than supervised methods. Besides, DINO [1] also shows the property to identify the foregrounds, which may, to a certain extent, remedies the shortcoming of relying on image backgrounds to classify.  Providing the performance of SSL ViTs will make the paper more convincing. \n2. Adding the $L_{classification}$ on the fine-tuning phase, the method is limited to pre-trained classes. What if the out-of-distribution datasets have some semantic-different classes (iNaturalist or other natural datasets with different classes), would the K-nearest neighbor testing of these novel classes still show better performance? \n3. Also, the organization of the paper is not ideal. For instance, in Section 2, the description of validation datasets and related works are mixed together.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "ZTjsOyDf4fa", "reviewer": "Reviewer_y4aF", "summary": "This paper presents a method for robustifying Transformers-based image classifiers against different image distributions, assuming that better attention improves the generalization performance. The main idea is to force the relevance map (or an aggregation of attention maps in Transformers) to focus on foreground objects. The method generates a relevance map with [8] and gives a manual or unsupervised segementation map as a preferred relevance map. The advantage of the method is experimentally demonstrated.", "questions": "1. I would like to see some discussions and explanations on Weaknesses 1-3.\n1. For the unsupervised segmentation case, how many images are used for training with segmentation results? \n2. I'm also curious about the performance of TokenCut over the datasets used in the paper. ", "limitations": "I'm not very sure if this is a limitation, but the paper does not mention the distribution used for training the unsupervised segmentation model. At least segmentation may not work when the data distribution is completely different (e.g., X-ray images). I think this point is not discussed in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. The method is simple but actually improves the performance. \n2. A good set of comparisons and ablation are presented. \n\n## Weaknesses\n\n1. According to Eqs. (6) and (7), a relevance map is based on gradients. I guess the classifier (or the model) is trained through relevance maps, and if this is the case, more details on how the model is trained with these gradients. \n2. There are some methods that try to debias a model, like [Burns et al., \"Women also Snowboard: Overcoming Bias in Captioning Models,\" ECCV 2018], which also uses manual segmentation for debiasing the model. I think the proposed method shares the basic ideas with such methods, and they may be easily adapted to classification tasks. I would like to see how the proposed method differs from these methods. Experimentally comparing some of these methods (for the manual segmentation variant) or at least providing discussion on them could be beneficial if they are comparable. \n3. The paper could discuss the data distribution used for training TokenCut in the context of generalization of the method to severely different data distributions. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "U0e_CX4xBQ", "reviewer": "Reviewer_Tjbv", "summary": "This submission aims to improve the robustness of vision transformers (ViT) by leveraging interpretability methods during training. The proposed approach relies on a recent method for computing pixel-wise relevance maps for ViT models. The relevance map for a pre-trained ViT does not necessarily coincide with the area occupied by the foreground object. Accordingly, a pre-trained model is fine-tuned with cross-entropy loss and two regularisation terms aimed at improving the relevance map. These two terms respectively encourage positive and negative agreement of the relevance map with a foreground segmentation map. This method is evaluated on several ImageNet-adjacent robustness benchmarks, and results in either noticeable improvements (>1pts: ObjNet, INet-A, INet-R) or performance on par with the baseline (+/-1pt: INet, INet-v2, Sketch).\n\n", "questions": "I have a few questions about the experimental evaluation:\n\nI would have liked to see a baseline which fine-tunes the model only using the classification loss. The reported results labelled \"original\" appear to refer to performance for the pre-trained model without fine-tuning. Is this correct? As such, we don't quite have a proper comparison between methods.\n\nI am also very confused by the choice of hyperparameters for the competing methods (RRR, GradCAM) -- see Appendix. I don't understand why the weight for the classification loss was not kept static for all methods together with the learning rate. What happens if you conduct a grid search for the regularisation term weights by fixing the aforementioned hyperparameters. Could you also describe the \"difficulties\" you faced while choosing the appropriate hyperparam values?\n\nWhile it is interesting that the method is effective given very little data, it should however be mentioned that (based on Fig. 8 in the appendix) the number of classes (500) and number of samples per class (3) were selected based on accuracy. Increasing the number of classes and/or samples beyond the chosen values can have a slightly negative impact, especially visible for the ImageNet-A curve. \n\nThis is a little surprising to me and I wonder why that is. Was the number of fine-tuning steps adapted to a change in the number of samples/classes? Is then the improvement a function of then number of training steps, or does the diversity of fine-tuning data not matter all? This is not immediately clear based on Fig. 8. It thus seems like a relatively minor adaptation of the network weights is enough to get better relevance maps and with it improved robustness. On which layers does the fine-tuning have the largest impact?\n\nI also think that an opportunity was missed when it comes to examining the impact of different loss terms. I could not find an ablation study that considers different combinations of the regularisation terms that were considered (e.g. fg only, bg only).\n\nWhat is the difference between AugReg and the corresponding vanilla ViT in terms of training?\n\n", "limitations": "Negative societal impact of the work is not addressed ([N/A] in the check list). ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a nice simple idea and it appears to improve robustness without much effort. If I understand correctly, it relies on fine-tuning a model for 50 epochs with just 1500 images per epoch (500 classes, 3 images each -- half of the total classes). It thus requires very few annotated segmentation masks relative to the full dataset. However, the method also results in decent performance with automatic segmentations obtained by a recent method called TokenCut (CVPR '22). \n\nThe experiments make sense, particularly the choice of evaluation benchmarks as well as the selection of competing methods. I like that several architectural variants & differently pre-trained instances thereof were considered, as well as repeated runs with different random seeds (see Appendix). I do have some concerns/questions with the experiments however which I list in the next section of the review. I think an opportunity was missed to conduct a more detailed ablation, and I have concerns about the hyperparameter selection procedure for the competing methods. I am surprised that increasing the amount of fine-tuning data (beyond 3*500 images) can harm performance (see Fig. 8).\n\nIn terms of clarity, I think the description of the relevance map extraction procedure (mostly relegated to the appendix) could be improved. No need to reproduce the full description from the original paper which describes it, but some clarification is necessary, e.g. I am not sure how this is initialised to an identity map? Are there any relevant knobs and dials to tweak the results?\n\nOverall, I am leaning towards tentative acceptance because I like the method and the direction. The paper makes an important point (made elsewhere too), namely that focusing on a \"single measuring stick: [accuracy]\" can only get us so far. Instead -- and this is my read -- we have to look into encouraging better model behaviour through other means, e.g. in this case making sure the model attends to relevant parts of the object. Before reaching a final rating, I would appreciate responses to the questions I raise in the rebuttal regarding the experiments.\n\nOne unstated assumption that could perhaps be interrogated a bit more is that reliance on background is by default something to be discouraged. While this probably is the case for most classes, there are other classes with more ambiguous appearance where the background is presumably useful. A more fine-grained analysis of the impact on various classes would have been quite useful. This issue is alluded to in section 5 but without any corresponding qualitative or quantitative analysis as far as I can tell.\n\nThere is partially such an analysis in the appendix (section H), which compares performance on the 500 training classes. vs the 500 remaining classes. These are interesting results which are briefly discussed in the main paper (section 5). There is an accuracy difference between the two sets, but it is not consistent across models and datasets. Accuracy for the training classes is on average better, but in some cases it is the other way around. Are there any ideas as to why this is the case? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept